AI engineering glossary · MLOps, LLMOps cloud
What is LLM gateway?
LLM gateway: A service placed between applications and one or more model providers that offers one consistent API and central features such as authentication, rate limits, logging, cost tracking, caching, retries and fallback to other models.
LLM gateway: an example
All apps call the company's gateway, which forwards to two providers and records tokens used by each team.
Related terms
More MLOps, LLMOps cloud terms
Where you learn it
LLM gateway is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.