AI engineering glossary · Transformers LLM architecture
What is transformer block?
Transformer block (also called Transformer layer): The repeating unit of a Transformer: an attention sublayer followed by a position-wise feed-forward sublayer, each wrapped with normalisation and a residual connection. The model is built by stacking many identical blocks, with each block refining the token representations.
Transformer block: an example
A 7B-parameter Llama-style model stacks 32 such blocks, each with attention followed by an MLP.
Related terms
More Transformers LLM architecture terms
- Transformer MLP block
- Transformer-XL
- Typical sampling
- Universal Transformer
- YaRN
- Activation patching
- Active parameters
- Additive attention
Where you learn it
Transformer block is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.