AI engineering glossary · LLM training alignment
What is maximal update parametrisation?
Maximal update parametrisation (also called muP, µP): A way of scaling initialisation and learning rates with model width so that good hyperparameters found on a small model transfer to a much larger one, cutting the cost of tuning big runs.
Maximal update parametrisation: an example
Tune the learning rate on a small width-256 model, then reuse the same value at width 4096 under this parametrisation.
Related terms
More LLM training alignment terms
- Megatron-LM
- Model collapse
- Model FLOPs utilisation
- Model merging
- Next-token prediction
- NormalFloat 4-bit
- Odds ratio preference optimisation
- Overtrained model
Where you learn it
Maximal update parametrisation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.