AI engineering glossary · Inference serving optimisation
What is TensorRT-LLM?
TensorRT-LLM: NVIDIA's library for running large language models efficiently on its GPUs. It supplies optimised attention kernels, quantisation, in-flight batching and multi-GPU parallelism, and is often deployed behind Triton Inference Server.
TensorRT-LLM: an example
An enterprise compiles a Llama-family model into a TensorRT-LLM engine for its A100 servers.
Related terms
More Inference serving optimisation terms
- Time per output token
- Time to first token
- Tokens per second
- torch.compile
- Total cost of ownership
- Tree attention
- Triton Inference Server
- Unstructured pruning
Where you learn it
TensorRT-LLM is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.