AI engineering glossary · Inference serving optimisation
What is arithmetic intensity?
Arithmetic intensity (also called Operational intensity): The number of arithmetic operations performed per byte moved from memory. Low values mean a task is limited by memory bandwidth and high values mean it is limited by compute; batching raises it because weights are read once for many tokens.
Arithmetic intensity: an example
Batch size 1 decoding does about 1 multiply-add per weight read; batch size 64 does about 64 with the same reads.
Related terms
More Inference serving optimisation terms
- Asymmetric quantisation
- Autoscaling
- AWQ
- Batch inference
- Block table
- Cache-aware routing
- Calibration dataset
- Chunked prefill
Where you learn it
Arithmetic intensity is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.