AI engineering glossary · Inference serving optimisation
What is kernel fusion?
Kernel fusion (also called Operator fusion): Combining several consecutive GPU operations into one kernel, so intermediate results stay in fast on-chip memory instead of being written to and re-read from main GPU memory. It cuts memory traffic and launch overhead.
Kernel fusion: an example
Fusing a bias add, activation and dropout into one kernel avoids two round trips through GPU memory.
Related terms
More Inference serving optimisation terms
- KV cache eviction
- KV cache quantisation
- llama.cpp
- LLM.int8()
- Load balancing
- Local inference
- Low-rank factorisation
- Medusa decoding
Where you learn it
Kernel fusion is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.