AI engineering glossary · Inference serving optimisation
What is vLLM?
vLLM: An open-source engine for serving large language models with high throughput. It combines PagedAttention for memory management, continuous batching and many optimised kernels, and exposes an OpenAI-compatible HTTP API, making it a common choice for self-hosted LLM serving.
vLLM: an example
A startup serves an open 8B model behind vLLM and points its existing OpenAI client at the new base URL.
Related terms
More Inference serving optimisation terms
- Warp
- Weight-only quantisation
- Zero-point
- Acceptance rate
- Activation quantisation
- All-reduce
- Arithmetic intensity
- Asymmetric quantisation
Where you learn it
vLLM is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.