AI engineering glossary · Transformers LLM architecture
What is residual stream?
Residual stream: The main vector for each token that flows through the layers of a Transformer, to which every attention and MLP sublayer adds its output. Viewing the model this way, layers read from and write to a shared channel, which is a common framing in interpretability work.
Residual stream: an example
Layer 5's attention adds a small vector to the residual stream, which layer 6 then reads.
Related terms
More Transformers LLM architecture terms
- Ring attention
- RoBERTa
- Rotary position embedding
- RWKV
- Scaled dot-product attention
- Self-attention
- Sequence length
- Sequence-to-sequence model
Where you learn it
Residual stream is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.