AI engineering glossary · Transformers LLM architecture
What is maximum output tokens?
Maximum output tokens (also called Max tokens, max_new_tokens): A limit on how many new tokens the model may generate in one response. Generation ends when it is reached, even mid-sentence. It caps cost and latency, and input plus output must fit within the context window.
Maximum output tokens: an example
Setting it to 256 ensures a summary is cut off after at most about 190 words.
Related terms
More Transformers LLM architecture terms
- Mechanistic interpretability
- Min-p sampling
- Mixtral
- Mixture of depths
- Mixture of experts
- Model dimension
- Multi-head attention
- Multi-head latent attention
Where you learn it
Maximum output tokens is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.