AI engineering glossary · Transformers LLM architecture
What is hidden states?
Hidden states (also called Hidden representations): The vectors that each layer of a Transformer produces for every token position, carrying its evolving representation. The last layer's hidden states feed the output head, and intermediate ones are often extracted as features or inspected for analysis.
Hidden states: an example
A 12-layer BERT gives 13 sets of hidden states (embeddings plus 12 layers), each of size tokens x 768.
Related terms
More Transformers LLM architecture terms
- Induction head
- Instruct model
- KV cache
- Large language model
- Learned positional embedding
- Length penalty
- Linear attention
- Llama
Where you learn it
Hidden states is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.