AI engineering glossary · Inference serving optimisation
What is tensorRT?
TensorRT: NVIDIA's toolkit for optimising trained networks for its GPUs. It fuses layers, chooses fast kernels, and supports lower precisions such as INT8 and FP8, producing an engine file tuned for one GPU type.
TensorRT: an example
Compiling an image classifier with TensorRT and INT8 cuts its latency on a T4 GPU compared with running it directly in PyTorch.
Related terms
More Inference serving optimisation terms
- TensorRT-LLM
- Time per output token
- Time to first token
- Tokens per second
- torch.compile
- Total cost of ownership
- Tree attention
- Triton Inference Server
Where you learn it
TensorRT is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.