AI engineering glossary · Inference serving optimisation
What is continuous batching?
Continuous batching (also called In-flight batching): A scheduling method for language-model servers where finished requests leave the running batch immediately and waiting requests join at the next generation step, instead of waiting for the whole batch to finish. It keeps the GPU busy and greatly raises throughput when output lengths vary.
Continuous batching: an example
If one request in a batch of 32 ends after 20 tokens and another needs 500, a new request takes the freed slot after token 20 rather than after token 500.
Related terms
More Inference serving optimisation terms
Where you learn it
Continuous batching is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.