AI engineering glossary · LLM training alignment
What is 3D parallelism?
3D parallelism: Combining data, tensor and pipeline parallelism at once to train models too big for any single strategy. Each axis maps to a different network layer of the cluster, with tensor parallelism inside fast nodes.
3D parallelism: an example
A 512-GPU job might use tensor parallel 8, pipeline parallel 4 and data parallel 16, since 8 x 4 x 16 = 512.
Related terms
More LLM training alignment terms
Where you learn it
3D parallelism is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.