AI engineering glossary · Transformers LLM architecture
What is ELECTRA?
ELECTRA (also called Replaced token detection): An encoder pretraining method in which a small generator replaces some tokens with plausible fakes and the main model learns to detect which tokens were replaced. Because it learns from every token, not just masked ones, it trains more efficiently than masked language modelling.
ELECTRA: an example
In 'The chef cooked the meal', the generator swaps 'cooked' for 'ate' and the model must flag that position as replaced.
Related terms
More Transformers LLM architecture terms
- Encoder
- Encoder-decoder model
- Encoder-only model
- Expert
- Expert parallelism
- Fill-in-the-middle
- FlashAttention
- Frequency and presence penalty
Where you learn it
ELECTRA is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.