AI engineering glossary · Inference serving optimisation
What is Triton Inference Server?
Triton Inference Server (also called NVIDIA Triton): An open-source server from NVIDIA that hosts models from several frameworks behind HTTP and gRPC endpoints. It offers dynamic batching, running multiple model copies and chaining models in pipelines, so one deployment can serve a whole application.
Triton Inference Server: an example
One Triton instance serves an ONNX classifier and a TensorRT-LLM chat model on the same GPU node.
Related terms
More Inference serving optimisation terms
- Unstructured pruning
- Verification step
- vLLM
- Warp
- Weight-only quantisation
- Zero-point
- Acceptance rate
- Activation quantisation
Where you learn it
Triton Inference Server is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.