AI engineering glossary · Inference serving optimisation
What is memory-bound?
Memory-bound (also called Bandwidth-bound): A workload whose speed is limited by how quickly data can be moved from memory to the compute units, not by arithmetic. Generating one token at a time is typically like this, because every step must read all model weights.
Memory-bound: an example
A 70B FP16 model read at 2 TB/s cannot generate a single sequence faster than about 14 tokens per second.
Related terms
More Inference serving optimisation terms
Where you learn it
Memory-bound is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.