AI engineering glossary · Inference serving optimisation
What is online inference?
Online inference (also called Real-time inference): Running a model in response to individual live requests, where each caller waits for the answer. It demands low and predictable latency, so batches are small and hardware is kept ready, which usually raises cost per prediction.
Online inference: an example
A payment app must approve or block a transaction within 100 ms.
Related terms
More Inference serving optimisation terms
- ONNX
- ONNX Runtime
- OpenAI-compatible API
- Outlier features
- PagedAttention
- Per-channel quantisation
- Per-tensor quantisation
- Percentile latency
Where you learn it
Online inference is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.