AI engineering glossary · Transformers LLM architecture
What is roBERTa?
RoBERTa: A robustly optimised retraining of BERT that trains longer on more data with larger batches, drops the next-sentence prediction task and masks tokens dynamically, changing the hidden words each time a sequence is seen. It improved results without changing the architecture.
RoBERTa: an example
A team swaps a BERT checkpoint for RoBERTa in its ticket-classification pipeline and sees a small accuracy gain with no code changes.
Related terms
More Transformers LLM architecture terms
- Rotary position embedding
- RWKV
- Scaled dot-product attention
- Self-attention
- Sequence length
- Sequence-to-sequence model
- Shared expert
- Sinusoidal positional encoding
Where you learn it
RoBERTa is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.