AI engineering glossary · Calculus optimisation
What is adam?
Adam (also called Adaptive Moment Estimation): A popular optimiser that keeps exponential moving averages of the gradient (first moment) and of its square (second moment), corrects their start-up bias, and scales each parameter's step accordingly. It combines momentum with adaptive step sizes and works well with little tuning.
Adam: an example
Typical defaults are learning rate 0.001, beta1 = 0.9, beta2 = 0.999 and epsilon = 1e-8; in PyTorch: torch.optim.Adam(model.parameters(), lr=1e-3).
Related terms
More Calculus optimisation terms
- AdamW
- Antiderivative
- Armijo condition
- Automatic differentiation
- Backtracking line search
- Batch gradient descent
- Bayesian optimisation
- BFGS
Where you learn it
Adam is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.