AI engineering glossary · Transformers LLM architecture
What is active parameters?
Active parameters (also called Activated parameters): The number of parameters actually used to process a single token. In dense models it equals the total; in mixture-of-experts models it is much smaller, and it drives compute per token while total parameters drive memory.
Active parameters: an example
A model with 47B total parameters that uses 13B per token needs memory for 47B but computes like a 13B model.
Related terms
More Transformers LLM architecture terms
- Additive attention
- ALiBi
- Attention head
- Attention mechanism
- Attention pattern
- Attention sink
- Attention weights
- Autoregressive generation
Where you learn it
Active parameters is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.