AI engineering glossary · Inference serving optimisation
What is PTQ (post-training quantisation)?
Post-training quantisation (also called PTQ): Converting an already trained model to lower precision without retraining it. A small calibration dataset is often run through the model to pick good scaling values. It is quick and cheap, but at very low bit widths accuracy can drop more than with methods that train with quantisation in mind.
Post-training quantisation: an example
A team takes a finished chatbot model, runs 512 sample prompts to calibrate, and exports an 8-bit version in an afternoon.
Related terms
More Inference serving optimisation terms
- Preemption
- Prefill-decode disaggregation
- Prefix caching
- Prompt lookup decoding
- Pruning
- Quantisation
- Quantisation-aware training
- Queueing delay
Where you learn it
Post-training quantisation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.