AI engineering glossary · Inference serving optimisation
What is INT4?
INT4: A 4-bit whole-number format with only 16 possible values. It cuts weight memory to a quarter of 16-bit storage, but needs finer grouping of scales and careful algorithms to keep model accuracy acceptable.
INT4: an example
Llama-style 7B checkpoints in 4-bit form are commonly around 4 GB, small enough to run on a laptop.
Related terms
More Inference serving optimisation terms
- INT8
- Inter-token latency
- Interconnect bandwidth
- Iteration-level scheduling
- Kernel fusion
- KV cache eviction
- KV cache quantisation
- llama.cpp
Where you learn it
INT4 is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.