AI engineering glossary · LLM training alignment
What is Megatron-LM?
Megatron-LM: NVIDIA's open-source framework for training very large transformer models efficiently across many GPUs, using tensor, pipeline and sequence parallelism, which has influenced many later training libraries and tools. Its techniques have influenced many later large-model training libraries and tools.
Megatron-LM: an example
A team uses Megatron-LM to split a large transformer across many GPU nodes with tensor and pipeline parallelism.
Related terms
More LLM training alignment terms
- Model collapse
- Model FLOPs utilisation
- Model merging
- Next-token prediction
- NormalFloat 4-bit
- Odds ratio preference optimisation
- Overtrained model
- Paged optimisers
Where you learn it
Megatron-LM is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.