AI engineering glossary · Transformers LLM architecture
What is induction head?
Induction head: A pair of cooperating attention heads found in Transformers that complete repeated patterns: on seeing token A, they look for an earlier occurrence of A and predict the token that followed it. They are thought to underlie part of in-context learning.
Induction head: an example
In 'Asha Rao ... Asha', an induction head attends to the earlier 'Asha' and predicts 'Rao'.
Related terms
More Transformers LLM architecture terms
- Instruct model
- KV cache
- Large language model
- Learned positional embedding
- Length penalty
- Linear attention
- Llama
- LM head
Where you learn it
Induction head is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.