AI engineering glossary · LLM training alignment
What is critical batch size?
Critical batch size (also called Gradient noise scale): The batch size beyond which making batches larger yields sharply diminishing gains in learning per example. Below it, doubling the batch nearly halves the steps needed; above it, extra compute is largely wasted.
Critical batch size: an example
If the critical size is about 4 million tokens, using 16 million tokens per batch costs more compute for little speed-up.
Related terms
More LLM training alignment terms
- Data curation
- Data mixture
- Debate (AI safety)
- DeepSpeed
- Direct preference optimisation
- Domain-adaptive pretraining
- DoRA
- Double quantisation
Where you learn it
Critical batch size is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.