AI engineering glossary · Inference serving optimisation
What is GGUF?
GGUF: A single-file binary format for storing quantised language models, used by llama.cpp and tools built on it. It packs weights, tokeniser and metadata together and supports many quantisation levels, so models can be run easily on laptops and CPUs.
GGUF: an example
A file named model-Q4_K_M.gguf holds a 4-bit-ish version of a model that runs on a MacBook without a dedicated GPU.
Related terms
- llama.cpp
- Quantisation
- Edge inference
- CPU inference
More Inference serving optimisation terms
- Goodput
- GPTQ
- GPU hours
- GPU kernel
- GPU utilisation
- Group-wise quantisation
- High bandwidth memory
- Inference engine
Where you learn it
GGUF is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.