AI engineering glossary · Inference serving optimisation
What is TPS (tokens per second)?
Tokens per second (also called TPS): A measure of generation speed: how many tokens a model produces each second, for one user (streaming speed) or across all users on a server (total throughput). The two numbers differ and answer different questions.
Tokens per second: an example
A server producing 2,400 tokens per second across 60 users gives each about 40 tokens per second.
Related terms
More Inference serving optimisation terms
- torch.compile
- Total cost of ownership
- Tree attention
- Triton Inference Server
- Unstructured pruning
- Verification step
- vLLM
- Warp
Where you learn it
Tokens per second is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.