AI engineering glossary · Transformers LLM architecture
What is dense model?
Dense model: A model in which every parameter is used to process every token, in contrast with sparse designs such as mixture of experts that use only a subset per token. Dense models are simpler to train and serve, but their compute cost grows in step with their size.
Dense model: an example
A 70B dense model uses all 70 billion parameters for each generated token.
Related terms
More Transformers LLM architecture terms
- Diverse beam search
- ELECTRA
- Encoder
- Encoder-decoder model
- Encoder-only model
- Expert
- Expert parallelism
- Fill-in-the-middle
Where you learn it
Dense model is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.