AI engineering glossary · Inference serving optimisation
What is autoscaling?
Autoscaling: Automatically adding or removing serving replicas as traffic changes, based on signals such as request rate, queue length or GPU use. It saves money off-peak but must handle slow start-up, since loading a large model onto a GPU can take minutes.
Autoscaling: an example
Replicas scale from 2 to 6 during the evening exam-results rush and back down at night.
Related terms
More Inference serving optimisation terms
- AWQ
- Batch inference
- Block table
- Cache-aware routing
- Calibration dataset
- Chunked prefill
- Cold start
- Compute-bound
Where you learn it
Autoscaling is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.