AI engineering glossary · Transformers LLM architecture
What is encoder?
Encoder: The part of a Transformer that reads the whole input with bidirectional attention and turns it into a set of context-aware vectors, one per token. Those vectors can be used for classification, search embeddings or as input to a decoder.
Encoder: an example
A BERT encoder turns a 20-token customer review into 20 vectors of size 768.
Related terms
More Transformers LLM architecture terms
- Encoder-decoder model
- Encoder-only model
- Expert
- Expert parallelism
- Fill-in-the-middle
- FlashAttention
- Frequency and presence penalty
- GPT
Where you learn it
Encoder is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.