AI engineering glossary · Inference serving optimisation
What is GPTQ?
GPTQ: A post-training method that quantises a model's weights layer by layer to 3 or 4 bits, using second-order information about the layer's error to adjust the remaining weights after each rounding step. It keeps accuracy well and needs only a small calibration set.
GPTQ: an example
A 13B model is compressed to 4-bit with GPTQ in under an hour on one GPU and then served from a GPU with 12 GB of memory.
Related terms
More Inference serving optimisation terms
- GPU hours
- GPU kernel
- GPU utilisation
- Group-wise quantisation
- High bandwidth memory
- Inference engine
- INT4
- INT8
Where you learn it
GPTQ is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.