AI engineering glossary · Inference serving optimisation
What is CPU offloading?
CPU offloading: Keeping part of a model's weights or cache in ordinary computer memory and moving it to the GPU only when needed. It lets a model larger than GPU memory run, but transfers are slow, so speed drops sharply.
CPU offloading: an example
A 30B model on a 16 GB GPU keeps some layers in RAM and streams them in per token, running at a few tokens per second.
Related terms
More Inference serving optimisation terms
Where you learn it
CPU offloading is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.