AI engineering glossary · Inference serving optimisation
What is quantisation?
Quantisation (also called Quantization, Model quantisation): Storing a model's numbers in fewer bits, for example 8-bit or 4-bit integers instead of 16-bit floats. The weights take less memory and move faster through the chip, so the model runs on cheaper hardware, usually with a small loss in output quality that must be measured.
Quantisation: an example
A 7-billion-parameter model needs about 14 GB in 16-bit form but roughly 3.5 GB with 4-bit weights, so it fits on a 8 GB consumer graphics card.
Quantisation in Python
A short example you can run (Python 3.11, NumPy). The output below is what it really prints.
import numpy as np
w = np.array([0.42, -1.37, 0.05, 2.10, -0.88], dtype=np.float32)
scale = np.abs(w).max() / 127
q = np.round(w / scale).astype(np.int8) # 1 byte per weight instead of 4
print("int8:", q)
print("restored:", np.round(q * scale, 3))
print("max error:", round(float(np.abs(q * scale - w).max()), 4))Output
int8: [ 25 -83 3 127 -53] restored: [ 0.413 -1.372 0.05 2.1 -0.876] max error: 0.0066
Related terms
- Post-training quantisation
- INT8
- INT4
- Dequantisation
More Inference serving optimisation terms
- Quantisation-aware training
- Queueing delay
- RadixAttention
- Request scheduler
- Reserved capacity
- Roofline model
- Scale factor
- Semantic caching
Where you learn it
Quantisation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.