AI engineering glossary · LLM training alignment
What is deepSpeed?
DeepSpeed: An open-source library from Microsoft for large-scale deep-learning training and inference. It provides ZeRO memory partitioning, CPU offloading, mixed precision and parallelism helpers so very large models fit on limited GPU clusters.
DeepSpeed: an example
Enabling a DeepSpeed ZeRO config lets a 13B model fine-tune on GPUs that could not hold it under plain data parallelism.
Related terms
More LLM training alignment terms
- Direct preference optimisation
- Domain-adaptive pretraining
- DoRA
- Double quantisation
- Emergent abilities
- Evol-Instruct
- Exposure bias
- FLAN
Where you learn it
DeepSpeed is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.