AI engineering glossary · Inference serving optimisation
What is HBM (high bandwidth memory)?
High bandwidth memory (also called HBM): A type of stacked memory placed right next to a GPU chip, offering much higher data rates than ordinary graphics memory. Its size and speed largely determine which models fit and how fast they can generate tokens.
High bandwidth memory: an example
A data-centre GPU with 80 GB of it can hold the FP16 weights of a 30B model but not a 70B one.
Related terms
More Inference serving optimisation terms
- Inference engine
- INT4
- INT8
- Inter-token latency
- Interconnect bandwidth
- Iteration-level scheduling
- Kernel fusion
- KV cache eviction
Where you learn it
High bandwidth memory is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.