AI engineering glossary · Transformers LLM architecture
What is model dimension?
Model dimension (also called d_model, Hidden size, Embedding dimension): The width of the vectors that flow through a Transformer, often written d_model. Token embeddings, attention outputs and the residual stream all have this size. Along with layer count it largely determines a model's parameter count.
Model dimension: an example
GPT-2 small uses d_model = 768; a 7B Llama-style model uses 4,096.
Related terms
More Transformers LLM architecture terms
- Multi-head attention
- Multi-head latent attention
- Multi-query attention
- Multi-token prediction
- Open-weight model
- Padding mask
- Parameter count
- Position interpolation
Where you learn it
Model dimension is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.