AI engineering glossary · Inference serving optimisation
What is ollama?
Ollama: A tool for downloading and running language models locally with a simple command line and local API, built on efficient engines such as llama.cpp. It handles model files, quantised variants and a background server so developers can test models without setting up infrastructure.
Ollama: an example
Typing ollama run with a model name pulls the weights and opens a chat on your own machine.
Related terms
More Inference serving optimisation terms
- Online inference
- ONNX
- ONNX Runtime
- OpenAI-compatible API
- Outlier features
- PagedAttention
- Per-channel quantisation
- Per-tensor quantisation
Where you learn it
Ollama is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.