AI engineering glossary · LLM training alignment
What is scaling laws?
Scaling laws (also called Neural scaling laws): Empirical rules showing that a language model's loss falls smoothly, following a power law, as model size, dataset size and training compute increase. They let teams predict the payoff of a larger run from small experiments.
Scaling laws: an example
Training several small models and plotting loss against compute allows forecasting the loss of a run 100 times larger.
Related terms
More LLM training alignment terms
- Self-instruct
- Self-rewarding language models
- Self-Taught Reasoner
- Sequence packing
- Sequence-level distillation
- Simple preference optimisation
- Soft labels
- Span corruption
Where you learn it
Scaling laws is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.