AI engineering glossary · Inference serving optimisation
What is request scheduler?
Request scheduler: The component of a serving engine that decides which waiting requests run next, how many tokens are processed per step and when to pause or drop work. Its policy affects fairness, latency and total throughput.
Request scheduler: an example
A scheduler admits requests first-come-first-served but lets short prompts jump ahead if the GPU has spare room.
Related terms
More Inference serving optimisation terms
- Reserved capacity
- Roofline model
- Scale factor
- Semantic caching
- Semi-structured sparsity
- Service level objective
- SGLang
- Shared memory
Where you learn it
Request scheduler is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.