AI engineering glossary · Inference serving optimisation
What is LLM.int8()?
LLM.int8(): A method for 8-bit matrix multiplication in large language models that splits out the few outlier dimensions and computes them in 16-bit, while everything else uses INT8. It halves memory with little quality change, but the mixed computation can be slower.
LLM.int8(): an example
Loading a model with the bitsandbytes library's 8-bit option applies it, letting a 30B model fit on one large GPU.
Related terms
More Inference serving optimisation terms
- Load balancing
- Local inference
- Low-rank factorisation
- Medusa decoding
- Memory fragmentation
- Memory-bound
- Model cascade
- Model compression
Where you learn it
LLM.int8() is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.