AI engineering glossary · Transformers LLM architecture
What is logit lens?
Logit lens: An interpretability technique that applies the model's final output layer to intermediate layers' residual stream, to see what token the model would predict if it stopped early. It shows how the prediction is refined across layers.
Logit lens: an example
Decoding layer 12 of a 32-layer model shows 'Paris' as top token for 'The Eiffel Tower is in', sharpening by layer 20.
Related terms
More Transformers LLM architecture terms
- Long-context model
- Longformer
- Mamba
- Maximum output tokens
- Mechanistic interpretability
- Min-p sampling
- Mixtral
- Mixture of depths
Where you learn it
Logit lens is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.