AI engineering glossary · Inference serving optimisation
What is SLO (service level objective)?
Service level objective (also called SLO): A measurable performance target that a service commits to, such as 99 percent of requests receiving a first token within 800 ms. It guides capacity planning and is used to judge whether an inference setup is good enough.
Service level objective: an example
The team sets an objective of p95 time to first token below 1 second and a monthly availability of 99.9 percent.
Related terms
More Inference serving optimisation terms
- SGLang
- Shared memory
- Small language model
- SmoothQuant
- Sparsity
- Speculative decoding
- Spot instance
- Static batching
Where you learn it
Service level objective is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.