AI engineering glossary · Transformers LLM architecture
What is BERT?
BERT: Bidirectional Encoder Representations from Transformers: an encoder-only model pretrained by masking random words and predicting them, plus a sentence-pair task. Its representations are fine-tuned for many language-understanding tasks and it popularised the pretrain-then-fine-tune approach.
BERT: an example
A BERT model is fine-tuned on 5,000 labelled Hindi-English sentiment examples to classify tweets.
Related terms
More Transformers LLM architecture terms
- Bidirectional attention
- BigBird
- Capacity factor
- Causal masking
- Constrained decoding
- Context window
- Contrastive decoding
- Cross-attention
Where you learn it
BERT is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.