AI engineering glossary · Inference serving optimisation
What is tensor core?
Tensor core: A specialised unit inside NVIDIA GPUs that multiplies small blocks of numbers in low precision far faster than general-purpose cores. Deep learning speed depends heavily on using these units, which support formats like FP16, BF16, INT8 and FP8.
Tensor core: an example
Matrix multiplications in FP16 run on tensor cores at many times the speed of standard FP32 cores.
Related terms
More Inference serving optimisation terms
- TensorRT
- TensorRT-LLM
- Time per output token
- Time to first token
- Tokens per second
- torch.compile
- Total cost of ownership
- Tree attention
Where you learn it
Tensor core is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.