AI engineering glossary · Inference serving optimisation
What is inference engine?
Inference engine (also called Serving engine): The software that loads a trained model and executes it efficiently to produce predictions, handling memory, batching and hardware kernels. Examples include vLLM, TensorRT-LLM, ONNX Runtime and llama.cpp, and the choice affects speed and cost.
Inference engine: an example
A team benchmarks two engines on the same model and picks the one giving lower cost per million tokens.
Related terms
More Inference serving optimisation terms
- INT4
- INT8
- Inter-token latency
- Interconnect bandwidth
- Iteration-level scheduling
- Kernel fusion
- KV cache eviction
- KV cache quantisation
Where you learn it
Inference engine is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.