AI engineering glossary · Transformers LLM architecture
What is ALiBi?
ALiBi (also called Attention with Linear Biases): Attention with Linear Biases: a position method that adds no position vectors, but subtracts a penalty from attention scores that grows linearly with the distance between query and key, with a different slope per head. It helps models handle inputs longer than seen in training.
ALiBi: an example
A key 10 tokens back gets a penalty of 10 * m added to its score, where m is that head's slope.
Related terms
More Transformers LLM architecture terms
- Attention head
- Attention mechanism
- Attention pattern
- Attention sink
- Attention weights
- Autoregressive generation
- Autoregressive model
- BART
Where you learn it
ALiBi is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.