AI engineering glossary · Inference serving optimisation
What is GPU utilisation?
GPU utilisation: The share of time a GPU reports being busy. It is a coarse indicator: a GPU can show 100 percent utilisation while its arithmetic units are mostly idle waiting for memory, so it does not by itself show efficient use.
GPU utilisation: an example
nvidia-smi shows 95 percent utilisation while the model reaches only 30 percent of the chip's peak arithmetic rate.
Related terms
More Inference serving optimisation terms
- Group-wise quantisation
- High bandwidth memory
- Inference engine
- INT4
- INT8
- Inter-token latency
- Interconnect bandwidth
- Iteration-level scheduling
Where you learn it
GPU utilisation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.