AI engineering glossary · Inference serving optimisation
What is SM (streaming multiprocessor)?
Streaming multiprocessor (also called SM): One of the many parallel processing blocks inside a GPU, holding arithmetic cores, registers and fast local memory. Work is split into thread blocks that are assigned to these blocks, so a chip's count of them gives a rough sense of its parallel capacity.
Streaming multiprocessor: an example
An A100 GPU has 108 of them, each running many thread blocks at once.
Related terms
More Inference serving optimisation terms
- Structured pruning
- Symmetric quantisation
- Tail latency
- Target model
- Tensor core
- TensorRT
- TensorRT-LLM
- Time per output token
Where you learn it
Streaming multiprocessor is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.