AI engineering glossary · Inference serving optimisation
What is W4A16 (weight-only quantisation)?
Weight-only quantisation (also called W4A16): A scheme that compresses only the stored weights, for example to 4 bits, while activations stay in 16-bit. Because text generation is limited by how fast weights can be read from memory, this cuts memory traffic and speeds decoding without needing integer matrix hardware.
Weight-only quantisation: an example
An AWQ 4-bit checkpoint keeps activations in FP16; weights are unpacked to FP16 inside the kernel just before each multiplication.
Related terms
More Inference serving optimisation terms
- Zero-point
- Acceptance rate
- Activation quantisation
- All-reduce
- Arithmetic intensity
- Asymmetric quantisation
- Autoscaling
- AWQ
Where you learn it
Weight-only quantisation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.