AI engineering glossary · LLM training alignment
What is chinchilla scaling?
Chinchilla scaling (also called Chinchilla optimal): The finding from DeepMind's Chinchilla work that, for a fixed compute budget, model size and training tokens should grow together, roughly twenty tokens per parameter, rather than making models larger on limited data. Earlier large models were undertrained by this measure.
Chinchilla scaling: an example
By this rule a 7B-parameter model is compute-optimal at about 140 billion tokens.
Related terms
More LLM training alignment terms
- Chosen and rejected responses
- Clipped surrogate objective
- Common Crawl
- Compute-optimal training
- Constitutional AI
- Continued pretraining
- Critical batch size
- Data curation
Where you learn it
Chinchilla scaling is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.