AI engineering glossary · LLM training alignment
What is doRA?
DoRA: Weight-Decomposed Low-Rank Adaptation, a LoRA variant that splits a pretrained weight into a magnitude and a direction, applies the low-rank update to the direction and trains the magnitude separately, often improving quality at similar cost.
DoRA: an example
Swapping LoRA for DoRA in a fine-tuning config aims at accuracy closer to full fine-tuning with a similar number of trainable parameters.
Related terms
More LLM training alignment terms
- Double quantisation
- Emergent abilities
- Evol-Instruct
- Exposure bias
- FLAN
- FLOPs
- Foundation model
- FP8 training
Where you learn it
DoRA is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.