AI engineering glossary · LLM training alignment
What is pipeline parallelism?
Pipeline parallelism: Placing consecutive groups of layers on different GPUs and passing micro-batches through them like an assembly line. It reduces per-GPU memory, but idle time, called pipeline bubbles, must be managed by using many micro-batches.
Pipeline parallelism: an example
Layers 1-16 sit on GPU 0 and layers 17-32 on GPU 1, and micro-batches flow through GPU 0 then GPU 1.
Related terms
More LLM training alignment terms
- Policy (reinforcement learning)
- Policy gradient
- Post-training
- Preference data
- Prefix tuning
- Pretraining
- Process reward model
- Prompt tuning
Where you learn it
Pipeline parallelism is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.