AI engineering glossary · Deep learning core
What is lion optimiser?
Lion optimiser (also called Lion): An optimiser found by automated program search that updates each weight using only the sign of a momentum-based direction, so every step has the same magnitude. It stores less state than Adam, saving memory, and has been reported to work well on some large models.
Lion optimiser: an example
Instead of scaling by gradient size, each weight moves by plus or minus the learning rate.
Related terms
More Deep learning core terms
- Log-sum-exp trick
- Logits
- Loss scaling
- Lottery ticket hypothesis
- LSTM
- Maxout
- Mixed precision training
- Model parallelism
Where you learn it
Lion optimiser is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.