Decision guide

RAG vs fine-tuning vs prompting: which one should you use?

A decision guide to RAG, fine-tuning and prompt engineering: what each changes, cost and effort, when to choose each, and how to combine them.

1 min readFree, no sign-up

Short answer: start with good prompting. If the model lacks your facts, add retrieval (RAG). If it knows the facts but behaves wrongly (wrong format, tone or skill on a narrow task), consider fine-tuning. RAG changes what the model knows for a given question; fine-tuning changes how it behaves.

What each one changes

Prompting RAG Fine-tuning
Changes Instructions and examples in the request The facts supplied with each request The model's weights
Good for Format, tone, simple tasks Private, changing or citable knowledge Consistent style, narrow skills, shorter prompts
Updating Edit the prompt Re-index the documents Train again
Effort Lowest Medium: search, chunking, evaluation Highest: data, training, testing
Citations No Yes, from retrieved passages No

How to decide

  1. Is the answer in your documents or database? Use RAG. The model should read the source, not remember it.
  2. Does it fail on format or style even with clear instructions and examples? Try better prompts first, then fine-tune.
  3. Does the information change weekly? RAG: re-indexing is far cheaper than retraining.
  4. Do you need a specific narrow behaviour at high volume? Fine-tuning can shorten prompts and make output consistent.

Common mistakes

  • Fine-tuning to add facts. Models absorb facts from fine-tuning unreliably and cannot cite them. Use retrieval.
  • Skipping evaluation. Build 30 to 50 realistic test questions first; change one thing at a time and compare.
  • Building RAG on messy documents. Fix chunking and add keyword search too (hybrid search, chunking problems).

They combine well

Many production systems use all three: clear prompts, retrieval for facts, and sometimes a fine-tuned model for tone or a narrow task. Measure each step with an evaluation pipeline so you only keep what helps.

Next steps

Read the RAG guide, learn how embeddings power retrieval, and see what is LLM engineering for the wider picture.