AI engineering glossary · Transformers LLM architecture
What is parameter count?
Parameter count (also called Model size): The total number of learned numbers (weights and biases) in a model, commonly stated in millions or billions, such as 7B. It roughly indicates model capacity and sets memory needs: at 16-bit precision each parameter takes 2 bytes.
Parameter count: an example
A 7-billion-parameter model in 16-bit precision needs about 14 GB just to store its weights.
Related terms
More Transformers LLM architecture terms
- Position interpolation
- Positional encoding
- Post-norm
- Pre-norm
- Prefill and decode
- Prefix language model
- QK-norm
- Quadratic attention cost
Where you learn it
Parameter count is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.