AI engineering glossary · Inference serving optimisation
What is CUDA graphs?
CUDA graphs: A CUDA feature that records a whole sequence of GPU operations once and replays it with a single launch call. It removes per-kernel launch overhead, which matters in token-by-token decoding where each step contains many tiny kernels.
CUDA graphs: an example
Capturing the decode step as a graph cuts CPU launch overhead and lowers per-token time.
Related terms
More Inference serving optimisation terms
Where you learn it
CUDA graphs is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.