AI engineering glossary · Inference serving optimisation
What is torch.compile?
torch.compile: A PyTorch feature that traces a model's operations and compiles them into optimised code, fusing operations and generating efficient kernels. It can speed up inference with a single line, though the first call is slow while compiling.
torch.compile: an example
Wrapping a model with torch.compile(model) gives faster repeated calls after a warm-up run.
Related terms
More Inference serving optimisation terms
- Total cost of ownership
- Tree attention
- Triton Inference Server
- Unstructured pruning
- Verification step
- vLLM
- Warp
- Weight-only quantisation
Where you learn it
torch.compile is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.