AI engineering glossary · Transformers LLM architecture
What is mamba?
Mamba: A state space model architecture whose parameters depend on the current input, so it can choose what to remember or forget, and which uses a hardware-aware scan. It offers linear scaling with length and competes with Transformers on some language tasks.
Mamba: an example
A Mamba model generates text with constant memory per step rather than a KV cache that grows with every token.
Related terms
More Transformers LLM architecture terms
- Maximum output tokens
- Mechanistic interpretability
- Min-p sampling
- Mixtral
- Mixture of depths
- Mixture of experts
- Model dimension
- Multi-head attention
Where you learn it
Mamba is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.