AI engineering glossary · Transformers LLM architecture
What is Transformer-XL?
Transformer-XL: A Transformer variant that processes long text in segments while letting each segment attend to cached hidden states from the previous one, and uses relative positions to make this consistent. It extends the effective context beyond a single fixed segment.
Transformer-XL: an example
A 512-token segment attends to its own tokens plus the cached states of the prior 512-token segment.
Related terms
More Transformers LLM architecture terms
- Typical sampling
- Universal Transformer
- YaRN
- Activation patching
- Active parameters
- Additive attention
- ALiBi
- Attention head
Where you learn it
Transformer-XL is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.