RAG explained: how retrieval-augmented generation works

What retrieval-augmented generation (RAG) is, how a RAG pipeline works step by step, a worked example, how to evaluate it, and RAG vs fine-tuning.

Short answer: retrieval-augmented generation (RAG) is a pattern where a system first searches a knowledge source for passages relevant to a question, then gives those passages to a language model with the question, so the answer is grounded in that material. It lets a model use private or fresh information without retraining, and makes answers easier to check against sources.

Why RAG exists

A language model only knows what was in its training data, up to a cut-off date, and nothing about your company's documents. Asking it about your refund policy invites a confident, invented answer. RAG fixes this by looking the answer up first.

How a RAG pipeline works

Ahead of time (indexing):

Collect the documents: PDFs, help pages, tickets, database rows.
Chunk them into passages of a few hundred tokens.
Embed each chunk: turn it into a vector that represents its meaning.
Index the vectors in a vector database or search index, along with the original text and metadata.

At question time:

Retrieve: embed the question and fetch the most similar chunks (often combined with keyword search: hybrid search).
Rerank (optional): reorder the top results with a more accurate model.
Generate: send the question plus the best passages to the LLM, with instructions to answer only from them and cite them.

A worked example

A bank's support assistant gets the question "Can I prepay my home loan without a penalty?" The retriever searches 3,000 policy PDFs and returns the five most relevant passages about prepayment. The model answers using only those passages and cites the policy section, so a support agent can check it in seconds.

What goes wrong, and the fixes

Problem	Typical cause	Fix
The right passage is never found	Poor chunking, keyword-heavy questions	Better chunk sizes, hybrid search, query rewriting
The answer ignores the passages	Weak instructions, too much context	Clear grounding instructions, fewer and better passages, reranking
Wrong answers sound confident	No check on faithfulness	Citations, evaluation of faithfulness, "I don't know" allowed
Slow or expensive	Too many or too long passages	Smaller top-k, caching, a smaller model

How to evaluate RAG

Measure the two halves separately:

Retrieval: for a labelled set of questions, did the retriever return the passages that contain the answer (recall and precision at k)?
Generation: is the answer faithful to the retrieved passages, and does it actually answer the question?

Run the test set on every change, so you know whether a new chunk size or model made things better or worse.

RAG vs fine-tuning

They solve different problems. RAG gives the model knowledge that it can cite and that you can update by re-indexing. Fine-tuning changes the model's behaviour: its format, tone or skill at one narrow task. Facts that change belong in RAG; consistent behaviour belongs in fine-tuning. Many production systems use both.

Words used on this page

Retrieval-augmented generation · Chunking · Vector database · Hybrid search · Reranking