AI engineering glossary · Transformers LLM architecture
What is MLA (multi-head latent attention)?
Multi-head latent attention (also called MLA): An attention design used in the DeepSeek models that compresses keys and values into a small shared latent vector for caching, and expands them back when needed. It sharply reduces KV cache size while aiming to retain multi-head quality.
Multi-head latent attention: an example
Instead of caching full keys and values for every head, only a compact latent per token is stored.
Related terms
More Transformers LLM architecture terms
- Multi-query attention
- Multi-token prediction
- Open-weight model
- Padding mask
- Parameter count
- Position interpolation
- Positional encoding
- Post-norm
Where you learn it
Multi-head latent attention is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.