AI engineering glossary · Inference serving optimisation
What is all-reduce?
All-reduce: A communication operation where every GPU contributes a tensor, the tensors are combined (usually summed) and every GPU receives the combined result. Tensor-parallel inference performs it after layers, so its speed limits multi-GPU scaling.
All-reduce: an example
Four GPUs each hold a partial output of a layer and use it to obtain the full sum before the next layer.
Related terms
More Inference serving optimisation terms
- Arithmetic intensity
- Asymmetric quantisation
- Autoscaling
- AWQ
- Batch inference
- Block table
- Cache-aware routing
- Calibration dataset
Where you learn it
All-reduce is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.