AI engineering glossary · Inference serving optimisation
What is interconnect bandwidth?
Interconnect bandwidth: The rate at which data can be moved between GPUs or between servers. When a model is spread over several devices, this rate can become the bottleneck that limits how much extra hardware helps.
Interconnect bandwidth: an example
NVLink offers many times the bandwidth of PCIe, so splitting one model across NVLink-connected GPUs scales better.
Related terms
More Inference serving optimisation terms
- Iteration-level scheduling
- Kernel fusion
- KV cache eviction
- KV cache quantisation
- llama.cpp
- LLM.int8()
- Load balancing
- Local inference
Where you learn it
Interconnect bandwidth is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.