AI engineering glossary · Deep learning core
What is flat minima?
Flat minima (also called Wide minima): Minima in the loss landscape surrounded by a broad low-loss region, as opposed to sharp narrow valleys. They are widely believed, though debated, to generalise better because small shifts in weights change predictions little.
Flat minima: an example
A solution whose loss stays near 0.2 when weights are perturbed by 5 percent is flatter than one that jumps to 2.0.
Related terms
More Deep learning core terms
- Focal loss
- Forward pass
- Fully connected layer
- Gated linear unit
- Gating mechanism
- GELU
- Gradient checkpointing
- Gradient norm
Where you learn it
Flat minima is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.