AI engineering glossary · Inference serving optimisation
What is llama.cpp?
llama.cpp: An open-source C/C++ inference engine for running language models efficiently on ordinary hardware such as laptops, phones and CPUs, with optional GPU offload. It popularised aggressive quantisation and the GGUF file format for local use.
llama.cpp: an example
A developer runs a 4-bit 8B model through llama.cpp on a laptop with no discrete GPU to prototype offline.
Related terms
- GGUF
- Quantisation
- CPU inference
- Edge inference
More Inference serving optimisation terms
- LLM.int8()
- Load balancing
- Local inference
- Low-rank factorisation
- Medusa decoding
- Memory fragmentation
- Memory-bound
- Model cascade
Where you learn it
llama.cpp is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.