AI engineering glossary · Calculus optimisation
What is adamW?
AdamW (also called Decoupled weight decay): A version of Adam that applies weight decay directly to the parameters, separately from the gradient-based update, instead of folding it into the gradient as an L2 penalty. This decoupling makes regularisation behave consistently and is the standard choice for training transformers.
AdamW: an example
torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01) shrinks each weight by a small fraction every step in addition to the Adam update.
Related terms
More Calculus optimisation terms
- Antiderivative
- Armijo condition
- Automatic differentiation
- Backtracking line search
- Batch gradient descent
- Bayesian optimisation
- BFGS
- Chain rule
Where you learn it
AdamW is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.