AI engineering glossary · Inference serving optimisation
What is smoothQuant?
SmoothQuant: A method that makes activation quantisation easier by shifting difficulty from activations to weights: it divides outlier-heavy activation channels by a factor and multiplies the matching weights by the same factor, leaving the output unchanged while flattening the ranges.
SmoothQuant: an example
Applying it lets a large language model run all its matrix multiplications in INT8 with almost no accuracy loss.
Related terms
More Inference serving optimisation terms
- Sparsity
- Speculative decoding
- Spot instance
- Static batching
- Streaming multiprocessor
- Structured pruning
- Symmetric quantisation
- Tail latency
Where you learn it
SmoothQuant is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.