AI engineering glossary · Inference serving optimisation
What is cold start?
Cold start: The delay when a new serving instance must be created and its model loaded before it can answer. For large models, downloading weights and moving them into GPU memory can take from seconds to many minutes.
Cold start: an example
A scaled-to-zero endpoint takes about 90 seconds to answer the first request after idling.
Related terms
- Autoscaling
- Model serving
- Model loading
- Time to first token
More Inference serving optimisation terms
- Compute-bound
- Continuous batching
- Cost per token
- CPU offloading
- CUDA graphs
- Draft model
- Dynamic batching
- EAGLE
Where you learn it
Cold start is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.