AI engineering glossary · Inference serving optimisation
What is TTFT (time to first token)?
Time to first token (also called TTFT): The delay between sending a request and receiving the first generated token. It covers queueing and processing the whole prompt, and is the main figure for how quickly a chat interface feels responsive.
Time to first token: an example
A user sends a 2,000-token prompt and the first word appears after 0.6 seconds.
Related terms
More Inference serving optimisation terms
- Tokens per second
- torch.compile
- Total cost of ownership
- Tree attention
- Triton Inference Server
- Unstructured pruning
- Verification step
- vLLM
Where you learn it
Time to first token is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.