AI engineering glossary · Inference serving optimisation
What is per-tensor quantisation?
Per-tensor quantisation: Using a single scale factor for an entire tensor. It is simple and fast, but one large value can force a coarse scale that crushes the many small values, so accuracy is usually lower than with finer groupings.
Per-tensor quantisation: an example
One outlier of 60 in a tensor of mostly values below 1 makes an INT8 scale of about 0.47, rounding most small weights to zero.
Related terms
More Inference serving optimisation terms
- Percentile latency
- Post-training quantisation
- Preemption
- Prefill-decode disaggregation
- Prefix caching
- Prompt lookup decoding
- Pruning
- Quantisation
Where you learn it
Per-tensor quantisation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.