AI engineering glossary · Inference serving optimisation
What is GPU kernel?
GPU kernel (also called CUDA kernel): A function that runs on the GPU, executed in parallel by thousands of threads. A model's forward pass is a sequence of kernels; launching many tiny ones wastes time, which motivates combining them.
GPU kernel: an example
A softmax and a matrix multiplication are two separate kernels unless fused.
Related terms
More Inference serving optimisation terms
- GPU utilisation
- Group-wise quantisation
- High bandwidth memory
- Inference engine
- INT4
- INT8
- Inter-token latency
- Interconnect bandwidth
Where you learn it
GPU kernel is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.