AI engineering glossary · Calculus optimisation
What is RMSProp?
RMSProp: An adaptive optimiser that divides each step by the square root of an exponentially decaying average of recent squared gradients. Unlike AdaGrad it forgets old history, so the effective learning rate does not vanish, making it suitable for non-stationary deep-learning objectives.
RMSProp: an example
With decay 0.9, running average s = 0.9 * s + 0.1 * g^2, and the update is lr * g / (sqrt(s) + epsilon).
Related terms
More Calculus optimisation terms
- Saddle point
- Secant line
- Second-order approximation
- Second-order condition
- Sharpness-aware minimisation
- Simulated annealing
- Stationary point
- Step decay
Where you learn it
RMSProp is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.