AI engineering glossary · Inference serving optimisation
What is local inference?
Local inference (also called On-device inference): Running a model on your own laptop, phone or server instead of calling a hosted API. It keeps data on your device, works offline and has no per-token fee, but you are limited by your hardware and must manage the model yourself.
Local inference: an example
A hospital in Pune runs a small model on an in-house server so patient notes never leave its network.
Related terms
More Inference serving optimisation terms
- Low-rank factorisation
- Medusa decoding
- Memory fragmentation
- Memory-bound
- Model cascade
- Model compression
- Model serving
- NCCL
Where you learn it
Local inference is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.