AI engineering glossary · Inference serving optimisation
What is goodput?
Goodput: The rate of requests completed while still meeting stated latency targets, rather than the raw count of all requests. A server can show high throughput yet low goodput if most responses arrive too late to be useful.
Goodput: an example
An engine finishes 100 requests per second but only 60 meet the 500 ms first-token target, giving a goodput of 60 per second.
Related terms
More Inference serving optimisation terms
- GPTQ
- GPU hours
- GPU kernel
- GPU utilisation
- Group-wise quantisation
- High bandwidth memory
- Inference engine
- INT4
Where you learn it
Goodput is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.