AI engineering glossary · Inference serving optimisation
What is static batching?
Static batching: Grouping several requests and running them together until every one has finished. It is easy to implement, but short requests must wait for the longest one and the GPU sits partly idle, so it wastes capacity for text generation.
Static batching: an example
Eight prompts are batched; seven finish in 30 tokens but the eighth needs 400, so seven slots idle for most of the run.
Related terms
More Inference serving optimisation terms
- Streaming multiprocessor
- Structured pruning
- Symmetric quantisation
- Tail latency
- Target model
- Tensor core
- TensorRT
- TensorRT-LLM
Where you learn it
Static batching is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.