AI engineering glossary · Inference serving optimisation
What is SRAM (shared memory)?
Shared memory (also called SRAM): A small, fast, programmable memory inside each GPU processing block that threads in the same block can use to exchange data. Efficient kernels load tiles of data into it once and reuse them many times instead of re-reading slow main memory.
Shared memory: an example
A matrix-multiplication kernel loads a 64 x 64 tile into it and uses each value 64 times.
Related terms
More Inference serving optimisation terms
- Small language model
- SmoothQuant
- Sparsity
- Speculative decoding
- Spot instance
- Static batching
- Streaming multiprocessor
- Structured pruning
Where you learn it
Shared memory is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.