AI engineering glossary · Inference serving optimisation
What is TPOT (time per output token)?
Time per output token (also called TPOT): The average time taken to produce each generated token after the first. Its inverse is the streaming speed a user sees, and it is governed largely by how fast weights and the KV cache can be read from memory.
Time per output token: an example
At 25 ms per token, a reply streams at 40 tokens per second, faster than most people read.
Related terms
More Inference serving optimisation terms
- Time to first token
- Tokens per second
- torch.compile
- Total cost of ownership
- Tree attention
- Triton Inference Server
- Unstructured pruning
- Verification step
Where you learn it
Time per output token is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.