AI engineering glossary · Deep learning core
What is linear scaling rule?
Linear scaling rule: A rule of thumb from large-batch training: when the batch size is multiplied by k, multiply the learning rate by k too, usually with a warmup period at the start. It keeps training behaviour similar as hardware is scaled up.
Linear scaling rule: an example
Moving from batch size 256 at rate 0.1 to batch size 1,024 suggests trying a rate near 0.4.
Related terms
More Deep learning core terms
- Lion optimiser
- Log-sum-exp trick
- Logits
- Loss scaling
- Lottery ticket hypothesis
- LSTM
- Maxout
- Mixed precision training
Where you learn it
Linear scaling rule is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.