AI engineering glossary · Inference serving optimisation
What is memory fragmentation?
Memory fragmentation (also called Internal fragmentation): Wasted memory that exists in small gaps or reserved-but-unused space, so that a large allocation fails even though total free memory looks sufficient. In LLM serving it appears when each request reserves space for its maximum possible length.
Memory fragmentation: an example
Reserving 4,096 slots for a request that ends at 300 leaves over 90 percent of its reservation unused.
Related terms
More Inference serving optimisation terms
Where you learn it
Memory fragmentation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.