AI engineering glossary · Inference serving optimisation
What is dynamic batching?
Dynamic batching: Collecting requests that arrive within a short time window and running them as one batch, up to a size limit. It trades a few milliseconds of waiting for much better hardware utilisation, and is a standard feature of model servers such as Triton.
Dynamic batching: an example
A server waits up to 5 ms to gather up to 16 image-classification requests before launching one GPU call.
Related terms
More Inference serving optimisation terms
Where you learn it
Dynamic batching is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.