AI engineering glossary · LLM training alignment
What is zeRO?
ZeRO (also called ZeRO optimiser): Zero Redundancy Optimizer, a memory-saving method for data-parallel training that partitions optimiser state, then gradients, then parameters across GPUs instead of copying them on each. Stages 1, 2 and 3 partition progressively more.
ZeRO: an example
With 8 GPUs, ZeRO stage 1 stores one-eighth of optimiser state on each GPU instead of a full copy.
Related terms
More LLM training alignment terms
Where you learn it
ZeRO is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.