AI engineering glossary · Transformers LLM architecture
What is learned positional embedding?
Learned positional embedding (also called Absolute position embedding): A position scheme where each position index has its own trainable vector, looked up like a word embedding and added to the token embedding. It is simple and works well within the trained length, but the model has no vectors for positions beyond the maximum it saw in training.
Learned positional embedding: an example
GPT-2 uses a table of 1,024 learned position vectors, so it cannot process inputs longer than 1,024 tokens.
Related terms
More Transformers LLM architecture terms
- Length penalty
- Linear attention
- Llama
- LM head
- Load-balancing loss
- Log probabilities
- Logit lens
- Long-context model
Where you learn it
Learned positional embedding is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.