AI engineering glossary · Inference serving optimisation
What is queueing delay?
Queueing delay: The time a request waits before processing starts because the server is busy. It grows sharply as load approaches capacity and is often the largest part of user-visible latency at peak times.
Queueing delay: an example
At 90 percent utilisation, requests wait behind others for several seconds even though each needs only 200 ms of compute.
Related terms
More Inference serving optimisation terms
- RadixAttention
- Request scheduler
- Reserved capacity
- Roofline model
- Scale factor
- Semantic caching
- Semi-structured sparsity
- Service level objective
Where you learn it
Queueing delay is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.