AI engineering glossary · Transformers LLM architecture
What is load-balancing loss?
Load-balancing loss (also called Auxiliary loss): An extra training term added to the loss in mixture-of-experts models that pushes the router to spread tokens evenly across experts. Without it, the router tends to favour a few experts, leaving others unused and wasting capacity.
Load-balancing loss: an example
If 90 percent of tokens go to 2 of 8 experts, the auxiliary loss becomes large and nudges the router to distribute more evenly.
Related terms
More Transformers LLM architecture terms
- Log probabilities
- Logit lens
- Long-context model
- Longformer
- Mamba
- Maximum output tokens
- Mechanistic interpretability
- Min-p sampling
Where you learn it
Load-balancing loss is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.