AI engineering glossary · Transformers LLM architecture
What is long-context model?
Long-context model: A language model designed or extended to accept very long inputs, often hundreds of thousands of tokens, through position-scaling methods, efficient attention kernels and training on long sequences. It lets whole codebases or books be placed in a single prompt.
Long-context model: an example
A long-context model reads a 400-page annual report in one prompt to answer a question about a footnote.
Related terms
More Transformers LLM architecture terms
- Longformer
- Mamba
- Maximum output tokens
- Mechanistic interpretability
- Min-p sampling
- Mixtral
- Mixture of depths
- Mixture of experts
Where you learn it
Long-context model is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.