Fixing RAG retrieval context fragmentation

Short answer: context fragmentation happens when the information an answer needs is split across several chunks, and retrieval returns only some of them, so the model answers from half the story. Fix it by chunking along the document's structure (headings, sections, table rows) instead of fixed character counts, adding a small overlap, retrieving a small chunk but sending its larger parent section to the model, and measuring retrieval with recall@k on real questions before and after each change.

How to recognise it

• Answers that are right but incomplete ("the refund is allowed" without "within 7 days").
• A table's header in one chunk and its rows in another, so numbers lose their meaning.
• Pronouns with no referent: the chunk says "it must be filed by the 20th" and nothing says what "it" is.
• Good answers for short documents, poor ones for long policies and manuals.

Six fixes, in the order to try them

1. Chunk by structure. Split on headings, paragraphs and list items; keep each table (or each row with its header) together. Parse PDFs properly first: many fragmentation problems are really parsing problems.
2. Add overlap. A small overlap (for example 10 to 20 percent) keeps sentences that cross a boundary in both chunks. See a runnable chunker with overlap on the chunking glossary page.
3. Prepend context to each chunk. Add the document title and section path ("Refund policy > Eligibility") to the chunk text before embedding, so a chunk carries its own context.
4. Small-to-big (parent-document) retrieval. Embed and search small chunks for precision, then send the model the whole parent section they came from.
5. Retrieve neighbours. When chunk 12 matches, also include chunks 11 and 13 from the same document.
6. Hybrid search plus reranking. Keyword search finds exact terms (form numbers, product codes) that embeddings miss; a reranker then puts the truly relevant chunks first. See the hybrid search guide.

Measure, do not guess

Build a small set of real questions with the passages that answer them (a golden set). For each change, compute recall@k (did the right passages come back in the top k?) and MRR, then check answer faithfulness with a calibrated LLM judge. Change one thing at a time. A chunking change that helps one document type can hurt another.

When the problem is not retrieval

If the right chunks are retrieved but the answer is still wrong, the problem is in generation: the prompt does not tell the model to use only the context, the context is too long and the key passage is buried in the middle, or the model needs to cite its sources so you can check it.

Where this is taught

Module 8 (RAG Engineering) covers parsing real documents, chunking strategies and their trade-offs, embeddings and vector databases, hybrid search, reranking, grounded answers with citations and measuring retrieval with recall@k and MRR. Related: RAG explained · RAG portfolio projects.