AI engineering glossary · Inference serving optimisation
What is roofline model?
Roofline model: A chart-based way to reason about performance: achievable speed is capped either by peak compute or by memory bandwidth multiplied by arithmetic intensity, whichever is lower. It shows whether a kernel should be optimised for data movement or for arithmetic.
Roofline model: an example
A kernel with intensity 10 FLOPs per byte on a 2 TB/s chip is capped at 20 TFLOPs, however high the chip's peak compute is.
Related terms
More Inference serving optimisation terms
- Scale factor
- Semantic caching
- Semi-structured sparsity
- Service level objective
- SGLang
- Shared memory
- Small language model
- SmoothQuant
Where you learn it
Roofline model is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.