AI engineering glossary · LLM training alignment
What is loss spike?
Loss spike (also called Training instability): A sudden jump in training loss during large-model training, often caused by unstable optimisation, bad data batches or numerical problems. Teams mitigate by lowering learning rate, clipping gradients, skipping batches or resuming from an earlier checkpoint.
Loss spike: an example
At step 40,000 the loss jumps from 2.1 to 6.5; the team restarts from the step-39,000 checkpoint and skips that data.
Related terms
More LLM training alignment terms
- Masked language modelling
- Maximal update parametrisation
- Megatron-LM
- Model collapse
- Model FLOPs utilisation
- Model merging
- Next-token prediction
- NormalFloat 4-bit
Where you learn it
Loss spike is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.