AI engineering glossary · LLM training alignment
What is tensor parallelism?
Tensor parallelism: Splitting individual weight matrices across GPUs so each computes part of every layer's matrix multiplication, then combining results. It lets layers too large for one GPU run, but needs fast interconnects because it communicates at every layer.
Tensor parallelism: an example
A 16384 x 16384 matrix is split column-wise across 4 GPUs, each holding 16384 x 4096.
Related terms
More LLM training alignment terms
- Test-time compute scaling
- Trust region policy optimisation
- UltraFeedback
- Value function
- Warmup-stable-decay schedule
- Weak-to-strong generalisation
- ZeRO
- 3D parallelism
Where you learn it
Tensor parallelism is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.