AI engineering glossary · Calculus optimisation
What is adaGrad?
AdaGrad (also called Adaptive gradient): An optimiser that scales each parameter's step by the inverse square root of the sum of all its past squared gradients. Rarely updated parameters get larger steps, which suits sparse features, but the accumulating sum can shrink steps too much over long training.
AdaGrad: an example
A word-embedding weight for a rare word gets large updates when it finally appears, while one for a common word takes small steps.
Related terms
More Calculus optimisation terms
- Adam
- AdamW
- Antiderivative
- Armijo condition
- Automatic differentiation
- Backtracking line search
- Batch gradient descent
- Bayesian optimisation
Where you learn it
AdaGrad is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.