Concept guide
Fine-tuning an LLM: when it helps, LoRA, and the alternatives
When fine-tuning a large language model is worth it, when prompting or RAG is better, how LoRA and QLoRA work, what data you need and how to evaluate the result.
Short answer: fine-tuning means training an existing model a little further on your own examples so it behaves the way you need: a fixed output format, a tone, or better accuracy on one narrow task. Try prompting and retrieval (RAG) first; fine-tune when they cannot give consistent results, or when a small fine-tuned model can replace a large, expensive one.
Prompting, RAG or fine-tuning?
| You need | Start with |
|---|---|
| The model to follow instructions or a format | Better prompts and structured output |
| Answers from your own or changing documents | RAG |
| A consistent behaviour or format that prompts cannot hold | Fine-tuning |
| A cheaper, faster model for one narrow task | Fine-tuning a small open model |
| New facts that change often | RAG, not fine-tuning |
How fine-tuning works
You prepare examples of inputs and the outputs you want, and continue training the model on them. The model's weights shift so that your kind of output becomes the most likely one.
- Full fine-tuning updates all the weights. It is powerful but needs a lot of GPU memory.
- LoRA (low-rank adaptation) freezes the model and trains small extra matrices instead. It needs far less memory, and the result is a small adapter file you can swap.
- QLoRA does the same on a model whose weights are stored in 4-bit, so even larger models fit on a single GPU.
What data you need
Quality matters more than quantity. A few hundred to a few thousand clean, consistent examples of exactly the task are often enough for a narrow job such as extracting fields from invoices. Remove contradictions, keep the format identical and hold some examples back for testing.
How to evaluate a fine-tuned model
Compare it with the baseline you are trying to beat (usually a well-prompted model) on the same held-out test set, using a metric that matches the task: exact match for extraction, schema validity for structured output, a rubric for style. If it does not beat the baseline, do not ship it.
Common mistakes
- Fine-tuning to add knowledge that changes: it goes stale and cannot cite sources. Use RAG.
- Too little or inconsistent data, so the model learns the noise.
- No baseline, so nobody knows whether it helped.
- Forgetting the cost of serving your own model.