AI engineering glossary · Transformers LLM architecture
What is encoder-only model?
Encoder-only model: A Transformer that uses only the encoder stack with bidirectional attention. It does not generate text but produces rich representations for understanding tasks such as classification, entity tagging, and semantic search. BERT and its relatives are typical examples.
Encoder-only model: an example
A fine-tuned encoder-only model labels support tickets as billing, delivery or refund.
Related terms
More Transformers LLM architecture terms
- Expert
- Expert parallelism
- Fill-in-the-middle
- FlashAttention
- Frequency and presence penalty
- GPT
- Greedy decoding
- Grouped-query attention
Where you learn it
Encoder-only model is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.