AI engineering glossary · Transformers LLM architecture
What is Universal Transformer?
Universal Transformer: A Transformer variant that applies the same block repeatedly, sharing its weights across depth, and can adaptively decide how many times to apply it to each position. It combines parallel attention with recurrence in depth.
Universal Transformer: an example
Instead of 6 different blocks, a Universal Transformer applies one block up to 6 times, stopping earlier for easy tokens.
Related terms
More Transformers LLM architecture terms
- YaRN
- Activation patching
- Active parameters
- Additive attention
- ALiBi
- Attention head
- Attention mechanism
- Attention pattern
Where you learn it
Universal Transformer is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.