AI engineering glossary · LLM training alignment
What is FP8 training?
FP8 training: Training with 8-bit floating-point numbers for some matrix operations, which roughly doubles arithmetic throughput and halves memory versus 16-bit on supporting GPUs, while keeping sensitive values such as master weights in higher precision.
FP8 training: an example
Matrix multiplications run in FP8 while weights are kept in higher precision to preserve accuracy.
Related terms
More LLM training alignment terms
- Full fine-tuning
- Fully sharded data parallel
- Generalised advantage estimation
- Goodhart's law
- Group relative policy optimisation
- Helpful, honest and harmless
- Hugging Face TRL
- Identity preference optimisation
Where you learn it
FP8 training is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.