AI engineering glossary · Transformers LLM architecture
What is longformer?
Longformer: An encoder Transformer for long documents that combines sliding-window local attention with a few tokens that attend globally, so cost grows linearly with length. It allows inputs of thousands of tokens where standard BERT is limited to 512.
Longformer: an example
A legal-tech team classifies 4,000-token contracts with a Longformer instead of splitting them into 512-token pieces.
Related terms
More Transformers LLM architecture terms
- Mamba
- Maximum output tokens
- Mechanistic interpretability
- Min-p sampling
- Mixtral
- Mixture of depths
- Mixture of experts
- Model dimension
Where you learn it
Longformer is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.