AI engineering glossary · Transformers LLM architecture
What is LM head?
LM head (also called Output projection, Unembedding): The final linear layer of a language model that maps the last hidden vector of each position to one score, called a logit, for every token in the vocabulary. Softmax on these logits gives next-token probabilities. Its weights are sometimes tied to the input embeddings.
LM head: an example
A model with vocabulary of 32,000 and d_model 4,096 has an LM head of shape 4,096 x 32,000.
Related terms
More Transformers LLM architecture terms
- Load-balancing loss
- Log probabilities
- Logit lens
- Long-context model
- Longformer
- Mamba
- Maximum output tokens
- Mechanistic interpretability
Where you learn it
LM head is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.