AI engineering glossary · LLM training alignment
What is overtrained model?
Overtrained model: A model trained on far more tokens than compute-optimal scaling suggests, accepting higher training cost to obtain a smaller model that is cheaper to serve at similar quality. It is a common choice when many users will run the model.
Overtrained model: an example
A 8B-parameter model trained on 15 trillion tokens is about 1,900 tokens per parameter, far above the compute-optimal ratio.
Related terms
More LLM training alignment terms
- Paged optimisers
- Parameter-efficient fine-tuning
- PEFT library
- Pipeline parallelism
- Policy (reinforcement learning)
- Policy gradient
- Post-training
- Preference data
Where you learn it
Overtrained model is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.