AI engineering glossary · Inference serving optimisation
What is SGLang?
SGLang: An open-source framework and serving runtime for language models that pairs a programming interface for multi-step prompting with an efficient backend featuring prefix-sharing caches and structured output support. It targets complex workflows such as agents and multi-call pipelines.
SGLang: an example
An application that calls a model five times with a shared context runs faster because the runtime reuses the shared cache.
Related terms
More Inference serving optimisation terms
- Shared memory
- Small language model
- SmoothQuant
- Sparsity
- Speculative decoding
- Spot instance
- Static batching
- Streaming multiprocessor
Where you learn it
SGLang is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.