AI engineering glossary · Inference serving optimisation
What is tail latency?
Tail latency (also called p99 latency): The response time experienced by the slowest requests, usually reported as the 95th or 99th percentile. Averages hide it, but it drives user frustration and service-level agreements, and heavy load or long prompts inflate it.
Tail latency: an example
A service with a median of 0.8 s and a p99 of 6 s means 1 request in 100 waits at least six seconds.
Related terms
More Inference serving optimisation terms
- Target model
- Tensor core
- TensorRT
- TensorRT-LLM
- Time per output token
- Time to first token
- Tokens per second
- torch.compile
Where you learn it
Tail latency is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.