AI engineering glossary · Deep learning core
What is model parallelism?
Model parallelism: A way of training a network too large for one device by splitting the model itself, such as its layers or weight matrices, across several devices that pass activations between them. It contrasts with data parallelism, which copies the whole model.
Model parallelism: an example
Layers 1-12 of a 24-layer network sit on GPU 0 and layers 13-24 on GPU 1.
Related terms
More Deep learning core terms
- Multi-task learning
- Multilayer perceptron
- Negative log-likelihood
- Neural network pruning
- Neural tangent kernel
- Neuron
- One-cycle policy
- Optimiser
Where you learn it
Model parallelism is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.