AI engineering glossary · Transformers LLM architecture
What is sinusoidal positional encoding?
Sinusoidal positional encoding: A fixed, non-learned position scheme from the original Transformer that uses sine and cosine waves of different frequencies at each embedding dimension. Each position gets a unique pattern, and offsets between positions correspond to simple transformations of these patterns.
Sinusoidal positional encoding: an example
PE(pos, 2i) = sin(pos / 10000^(2i/d_model)) and PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model)).
Related terms
More Transformers LLM architecture terms
- Sliding window attention
- Sparse attention
- Sparse autoencoder
- State space model
- Switch Transformer
- T5
- Temperature
- Test-time compute
Where you learn it
Sinusoidal positional encoding is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.