AI engineering glossary · LLM training alignment
What is compute-optimal training?
Compute-optimal training: Choosing model size and training data amount to reach the lowest loss for a given compute budget, using fitted scaling laws. It optimises training cost only and ignores inference cost, which is why many production models are trained on more data than this rule suggests.
Compute-optimal training: an example
With a fixed GPU budget, a compute-optimal plan may pick a 13B model on 260B tokens over a 40B model on 30B tokens.
Related terms
More LLM training alignment terms
- Constitutional AI
- Continued pretraining
- Critical batch size
- Data curation
- Data mixture
- Debate (AI safety)
- DeepSpeed
- Direct preference optimisation
Where you learn it
Compute-optimal training is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.