AI engineering glossary · Inference serving optimisation
What is ONNX Runtime?
ONNX Runtime: A cross-platform engine from Microsoft for running models saved in the ONNX format. It applies graph optimisations and can use different hardware back ends, such as CPUs, GPUs and mobile chips, through pluggable execution providers.
ONNX Runtime: an example
A company runs a scikit-learn and a PyTorch model through one ONNX Runtime service on CPU.
Related terms
- ONNX
- TensorRT
- Inference engine
- Execution provider
More Inference serving optimisation terms
- OpenAI-compatible API
- Outlier features
- PagedAttention
- Per-channel quantisation
- Per-tensor quantisation
- Percentile latency
- Post-training quantisation
- Preemption
Where you learn it
ONNX Runtime is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.