AI engineering glossary · Inference serving optimisation
What is batch inference?
Batch inference (also called Offline inference): Running a model over a large stored set of inputs in one scheduled job, with no user waiting for each answer. Because latency does not matter, it can use big batches, cheap spot hardware and off-peak hours to minimise cost.
Batch inference: an example
A bank scores 10 million customer records overnight in one batch job.
Related terms
More Inference serving optimisation terms
- Block table
- Cache-aware routing
- Calibration dataset
- Chunked prefill
- Cold start
- Compute-bound
- Continuous batching
- Cost per token
Where you learn it
Batch inference is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.