AI engineering glossary · Inference serving optimisation
What is ITL (inter-token latency)?
Inter-token latency (also called ITL): The gap between consecutive generated tokens as seen by the client. Unlike the average per-token time, it exposes stalls and jitter, for example when a long prefill from another user briefly blocks the batch.
Inter-token latency: an example
Most gaps are 30 ms, but one arrives after 400 ms because a large prompt was scheduled at that step.
Related terms
More Inference serving optimisation terms
- Interconnect bandwidth
- Iteration-level scheduling
- Kernel fusion
- KV cache eviction
- KV cache quantisation
- llama.cpp
- LLM.int8()
- Load balancing
Where you learn it
Inter-token latency is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.