AI engineering glossary · Transformers LLM architecture
What is context window?
Context window (also called Context length, Context size): The maximum number of tokens a model can consider at once, covering the prompt, any documents, the conversation so far and the reply. Beyond it, earlier text must be dropped or summarised. Longer windows allow bigger documents but cost more memory and compute.
Context window: an example
A model with a 128,000-token window can take a roughly 300-page book in one prompt, while a 4,096-token window cannot.
Context window in Python
A short example you can run (Python 3.11). The output below is what it really prints.
def fit_to_window(messages, max_tokens, count=lambda s: len(s.split())):
kept, used = [], 0
for m in reversed(messages): # keep the newest messages first
n = count(m)
if used + n > max_tokens:
break
kept.insert(0, m); used += n
return kept, used
chat = ["hello there", "tell me about RAG in detail please", "and what about reranking", "give an example"]
print(fit_to_window(chat, max_tokens=10))Output
(['and what about reranking', 'give an example'], 7)
Related terms
More Transformers LLM architecture terms
- Contrastive decoding
- Cross-attention
- Decoder
- Decoder-only model
- Decoding
- Dense model
- Diverse beam search
- ELECTRA
Where you learn it
Context window is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.