AI engineering glossary · Inference serving optimisation
What is percentile latency?
Percentile latency: A latency figure that says what share of requests finished within a given time, such as p50 for the median or p95 and p99 for slower cases. Percentiles describe real user experience better than an average.
Percentile latency: an example
If p95 is 1.2 s, then 95 percent of requests completed in 1.2 seconds or less.
Related terms
More Inference serving optimisation terms
- Post-training quantisation
- Preemption
- Prefill-decode disaggregation
- Prefix caching
- Prompt lookup decoding
- Pruning
- Quantisation
- Quantisation-aware training
Where you learn it
Percentile latency is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.