AI engineering glossary · Deep learning core
What is adafactor?
Adafactor: A memory-efficient adaptive optimiser that stores compact row and column statistics of the squared gradients instead of one value per weight. It is popular for training large transformer models where Adam's extra memory is costly.
Adafactor: an example
For a 4,096 x 4,096 weight matrix it keeps two vectors of 4,096 numbers rather than a 16-million-entry array.
Related terms
More Deep learning core terms
- Artificial neural network
- Autoencoder
- Autograd
- Backpropagation
- Backpropagation through time
- Backward pass
- Batch normalisation
- Batch renormalisation
Where you learn it
Adafactor is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.