AI engineering glossary · LLM training alignment
What is FSDP (fully sharded data parallel)?
Fully sharded data parallel (also called FSDP): A PyTorch training strategy that shards model parameters, gradients and optimiser state across GPUs and gathers each layer's weights just in time for its computation, allowing much larger models than plain data parallelism.
Fully sharded data parallel: an example
On 8 GPUs, each holds one-eighth of the weights and temporarily gathers a layer's full weights during its forward pass.
Related terms
More LLM training alignment terms
- Generalised advantage estimation
- Goodhart's law
- Group relative policy optimisation
- Helpful, honest and harmless
- Hugging Face TRL
- Identity preference optimisation
- InstructGPT
- Instruction backtranslation
Where you learn it
Fully sharded data parallel is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.