AI engineering glossary · Inference serving optimisation
What is load balancing?
Load balancing: Spreading incoming requests over several identical servers so no one is overloaded and failed servers are bypassed. For LLM serving, smarter policies consider queue length and cache contents, not just simple rotation.
Load balancing: an example
A router sends a new chat turn to the replica that already holds that conversation's cached prefix.
Related terms
More Inference serving optimisation terms
- Local inference
- Low-rank factorisation
- Medusa decoding
- Memory fragmentation
- Memory-bound
- Model cascade
- Model compression
- Model serving
Where you learn it
Load balancing is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.