AI engineering glossary · Inference serving optimisation
What is W8A8 (activation quantisation)?
Activation quantisation (also called W8A8): Lowering the precision of the intermediate values flowing between layers, not just the stored weights. It allows the matrix multiplications themselves to run in fast integer or FP8 units, but activations contain occasional very large values, so it is harder to do without hurting quality.
Activation quantisation: an example
With 8-bit weights and 8-bit activations, a matrix multiplication can use the GPU's INT8 tensor cores instead of FP16 ones.
Related terms
More Inference serving optimisation terms
- All-reduce
- Arithmetic intensity
- Asymmetric quantisation
- Autoscaling
- AWQ
- Batch inference
- Block table
- Cache-aware routing
Where you learn it
Activation quantisation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.