AI engineering glossary · Inference serving optimisation
What is spot instance?
Spot instance (also called Preemptible VM): A cloud machine rented at a steep discount on the condition that the provider can take it back at short notice. It suits fault-tolerant or batch inference jobs but is risky for latency-sensitive endpoints.
Spot instance: an example
A nightly job that embeds 5 million documents runs on spot GPUs and resumes from a checkpoint if one is reclaimed.
Related terms
More Inference serving optimisation terms
- Static batching
- Streaming multiprocessor
- Structured pruning
- Symmetric quantisation
- Tail latency
- Target model
- Tensor core
- TensorRT
Where you learn it
Spot instance is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.