AI engineering glossary · Transformers LLM architecture
What is prefix language model?
Prefix language model (also called Prefix LM): A Transformer setup in which the input prefix, such as an instruction or document, is read with full bidirectional attention while the continuation is generated with causal attention. It sits between encoder-decoder and decoder-only designs and uses a single stack of layers.
Prefix language model: an example
In a prefix LM, the 300-token question is attended to in both directions, and the answer tokens are then generated left to right.
Related terms
More Transformers LLM architecture terms
- QK-norm
- Quadratic attention cost
- Query, key and value
- Reformer
- Relative position encoding
- Residual stream
- Ring attention
- RoBERTa
Where you learn it
Prefix language model is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.