AI engineering glossary · Deep learning core
What is LAMB (LAMB optimiser)?
LAMB optimiser (also called LAMB): A layer-wise adaptive optimiser that rescales each layer's Adam-style update by the ratio of the layer's weight size to its update size. It allows very large batch sizes to be used without losing accuracy, as in fast BERT pretraining.
LAMB optimiser: an example
A layer with small weights takes proportionally smaller steps than a layer with large weights.
Related terms
More Deep learning core terms
- Layer
- Layer normalisation
- Leaky ReLU
- Learning rate finder
- LeCun initialisation
- Linear scaling rule
- Lion optimiser
- Log-sum-exp trick
Where you learn it
LAMB optimiser is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.