AI engineering glossary · Inference serving optimisation
What is iteration-level scheduling?
Iteration-level scheduling: Making scheduling decisions at every single token-generation step rather than once per request. The scheduler can add, remove or pause sequences between steps, which is the idea behind continuous batching in systems such as Orca.
Iteration-level scheduling: an example
After each decode step the scheduler checks memory and either admits a queued request or pauses a low-priority one.
Related terms
More Inference serving optimisation terms
- Kernel fusion
- KV cache eviction
- KV cache quantisation
- llama.cpp
- LLM.int8()
- Load balancing
- Local inference
- Low-rank factorisation
Where you learn it
Iteration-level scheduling is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.