AI engineering glossary · Transformers LLM architecture
What is Transformer MLP block?
Transformer MLP block (also called Position-wise feed-forward network, FFN block): The feed-forward sublayer in each Transformer block, applied independently to every token: a linear layer that expands the dimension (often 4x), a nonlinearity, and a linear layer that projects back. It holds most of a block's parameters and is thought to store much factual knowledge.
Transformer MLP block: an example
With d_model = 768, the MLP expands to 3,072 units and projects back to 768.
Related terms
More Transformers LLM architecture terms
- Transformer-XL
- Typical sampling
- Universal Transformer
- YaRN
- Activation patching
- Active parameters
- Additive attention
- ALiBi
Where you learn it
Transformer MLP block is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.