AI engineering glossary · Inference serving optimisation
What is INT8?
INT8: An 8-bit whole-number format with 256 possible values. Model weights or activations are mapped into it using a scale factor, halving memory compared with 16-bit floats and enabling fast integer arithmetic on many GPUs, CPUs and phone chips.
INT8: an example
A weight of 0.37 with scale 0.01 is stored as the integer 37 and recovered as 37 x 0.01 when needed.
Related terms
More Inference serving optimisation terms
- Inter-token latency
- Interconnect bandwidth
- Iteration-level scheduling
- Kernel fusion
- KV cache eviction
- KV cache quantisation
- llama.cpp
- LLM.int8()
Where you learn it
INT8 is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.