AI engineering glossary · LLM training alignment
What is warmup-stable-decay schedule?
Warmup-stable-decay schedule (also called WSD schedule): A learning-rate schedule with three phases: a short warmup, a long stretch at a constant rate, and a final decay to a small value. Since the constant phase has no fixed end, training can be extended or branched flexibly.
Warmup-stable-decay schedule: an example
Learning rate rises to 3e-4 over 2,000 steps, stays there for most of the run, then falls to near zero over the last 10 percent.
Related terms
More LLM training alignment terms
- Weak-to-strong generalisation
- ZeRO
- 3D parallelism
- 8-bit optimiser
- Adapter
- Advantage function
- Alignment
- Alignment tax
Where you learn it
Warmup-stable-decay schedule is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.