AI engineering glossary · LLM training alignment
What is paged optimisers?
Paged optimisers: A memory technique that stores optimiser state in pageable memory so it can move between GPU and CPU RAM when GPU memory spikes, preventing out-of-memory crashes during training with long sequences.
Paged optimisers: an example
A QLoRA run with occasional long batches keeps training instead of crashing because Adam's state pages to CPU memory.
Related terms
More LLM training alignment terms
- Parameter-efficient fine-tuning
- PEFT library
- Pipeline parallelism
- Policy (reinforcement learning)
- Policy gradient
- Post-training
- Preference data
- Prefix tuning
Where you learn it
Paged optimisers is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.