AI engineering glossary · Inference serving optimisation
What is semi-structured sparsity?
Semi-structured sparsity (also called 2:4 sparsity): A pattern where in every small group of weights a fixed number must be zero, such as 2 out of every 4 values. Because the pattern is regular, recent NVIDIA GPUs can skip the zeros in hardware and roughly double matrix-multiplication throughput.
Semi-structured sparsity: an example
A 2:4 sparse layer stores two non-zero weights per block of four plus small index bits, halving weight storage.
Related terms
More Inference serving optimisation terms
- Service level objective
- SGLang
- Shared memory
- Small language model
- SmoothQuant
- Sparsity
- Speculative decoding
- Spot instance
Where you learn it
Semi-structured sparsity is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.