AI engineering glossary · Deep learning core
What is data parallelism?
Data parallelism: A way to train faster by copying the model onto several devices, giving each a different slice of the batch, and averaging their gradients before every update. It scales training with more GPUs as long as the model fits on one.
Data parallelism: an example
Eight GPUs each processing 32 examples behave like one step of batch size 256.
Related terms
More Deep learning core terms
Where you learn it
Data parallelism is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.