AI engineering glossary · Deep learning core
What is loss scaling?
Loss scaling (also called Gradient scaling): A companion to 16-bit training that multiplies the loss by a large constant before the backward pass so that small gradients do not underflow to zero, then divides the gradients back before updating weights.
Loss scaling: an example
Scaling the loss by 1,024 turns a gradient of 1e-8, which FP16 would round to zero, into a representable value.
Related terms
More Deep learning core terms
- Lottery ticket hypothesis
- LSTM
- Maxout
- Mixed precision training
- Model parallelism
- Multi-task learning
- Multilayer perceptron
- Negative log-likelihood
Where you learn it
Loss scaling is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.