AI engineering glossary · Transformers LLM architecture
What is scaled dot-product attention?
Scaled dot-product attention (also called Dot-product attention): The core attention formula of the Transformer: multiply queries by transposed keys to get similarity scores, divide by the square root of the key dimension, apply softmax to get weights, then multiply by values. The scaling keeps scores from growing large and making softmax saturate.
Scaled dot-product attention: an example
attention(Q,K,V) = softmax(Q K^T / sqrt(d_k)) V. With d_k = 64 the scores are divided by 8 before softmax.
Related terms
More Transformers LLM architecture terms
- Self-attention
- Sequence length
- Sequence-to-sequence model
- Shared expert
- Sinusoidal positional encoding
- Sliding window attention
- Sparse attention
- Sparse autoencoder
Where you learn it
Scaled dot-product attention is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.