AI engineering glossary · LLM training alignment
What is FLOP (FLOPs)?
FLOPs (also called FLOP): Floating-point operations, a count of arithmetic operations used to measure compute. Training cost is often quoted as total FLOPs; a common estimate for a transformer is about 6 x parameters x tokens.
FLOPs: an example
Training a 7-billion-parameter model on 1 trillion tokens costs roughly 6 x 7e9 x 1e12 = 4.2e22 FLOPs.
Related terms
More LLM training alignment terms
- Foundation model
- FP8 training
- Full fine-tuning
- Fully sharded data parallel
- Generalised advantage estimation
- Goodhart's law
- Group relative policy optimisation
- Helpful, honest and harmless
Where you learn it
FLOPs is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.