AI engineering glossary · Transformers LLM architecture
What is positional encoding?
Positional encoding (also called Position encoding, Positional embedding): Extra information added to or mixed into token representations so the model knows the order of tokens. Attention by itself treats its input as an unordered set, so without position information 'dog bites man' and 'man bites dog' would look the same.
Positional encoding: an example
A token embedding of size 768 has a position vector of size 768 added to it before the first Transformer block.
Related terms
- Sinusoidal positional encoding
- Learned positional embedding
- Rotary position embedding
- Relative position encoding
More Transformers LLM architecture terms
- Post-norm
- Pre-norm
- Prefill and decode
- Prefix language model
- QK-norm
- Quadratic attention cost
- Query, key and value
- Reformer
Where you learn it
Positional encoding is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.