AI engineering glossary · Transformers LLM architecture
What is bigBird?
BigBird: A sparse-attention Transformer that combines a sliding window, a few global tokens and random connections, so cost grows linearly with length while keeping theoretical properties of full attention. It was designed for long documents such as articles and genomic sequences.
BigBird: an example
BigBird lets an encoder process a 4,096-token research paper in one pass.
Related terms
More Transformers LLM architecture terms
- Capacity factor
- Causal masking
- Constrained decoding
- Context window
- Contrastive decoding
- Cross-attention
- Decoder
- Decoder-only model
Where you learn it
BigBird is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.