AI engineering glossary · Inference serving optimisation
What is group-wise quantisation?
Group-wise quantisation (also called Block-wise quantisation): Splitting weights into small blocks, commonly 32 to 128 values, each with its own scale (and sometimes zero-point). The finer granularity keeps 4-bit models accurate, at the cost of storing extra scale values.
Group-wise quantisation: an example
With group size 128 and 16-bit scales, 4-bit weights use about 4.125 bits each once scales are counted.
Related terms
More Inference serving optimisation terms
- High bandwidth memory
- Inference engine
- INT4
- INT8
- Inter-token latency
- Interconnect bandwidth
- Iteration-level scheduling
- Kernel fusion
Where you learn it
Group-wise quantisation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.