AI engineering glossary · Inference serving optimisation
What is compute-bound?
Compute-bound: A workload whose speed is limited by how many arithmetic operations the chip can perform, not by data movement. Processing a long prompt in one pass and large-batch matrix multiplication usually fall in this category.
Compute-bound: an example
Prefilling 4,000 prompt tokens keeps the GPU's arithmetic units near full use.
Related terms
More Inference serving optimisation terms
- Continuous batching
- Cost per token
- CPU offloading
- CUDA graphs
- Draft model
- Dynamic batching
- EAGLE
- Edge inference
Where you learn it
Compute-bound is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.