AI engineering glossary · Transformers LLM architecture
What is activation patching?
Activation patching (also called Causal tracing): An interpretability experiment where an internal activation from one run of a model is copied into another run, and the change in output shows which component carries the relevant information. It helps locate the heads or layers responsible for a behaviour.
Activation patching: an example
Copying one attention head's output from a prompt about Delhi into a prompt about Mumbai flips the predicted answer, revealing that head's role.
Related terms
More Transformers LLM architecture terms
- Active parameters
- Additive attention
- ALiBi
- Attention head
- Attention mechanism
- Attention pattern
- Attention sink
- Attention weights
Where you learn it
Activation patching is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.