AI engineering glossary · Inference serving optimisation
What is structured pruning?
Structured pruning: Removing whole groups of parameters, such as entire channels, attention heads or layers, so the resulting model is genuinely smaller and runs faster on ordinary hardware. It is less flexible than removing single weights but gives real speedups.
Structured pruning: an example
Removing 8 of 32 attention heads from every layer produces a smaller dense model that needs no special sparse kernels.
Related terms
More Inference serving optimisation terms
- Symmetric quantisation
- Tail latency
- Target model
- Tensor core
- TensorRT
- TensorRT-LLM
- Time per output token
- Time to first token
Where you learn it
Structured pruning is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.