What Is an LLM Gateway? A Guide for Production Teams

An LLM gateway is a proxy that sits between your application and every model provider you call. It gives you one API, then handles routing, retries, fallbacks, rate limits, caching, spend tracking and logging on each request. The cost is one more hop, and one more place where behavior can change without an error.

What is an LLM gateway?

Without a gateway, every service that calls a model holds its own provider keys, its own retry logic, and its own idea of what a request costs. That works with one provider and one team. It stops working when you add a second provider, a second team, or a finance department that wants a single bill.

An LLM gateway centralizes that plumbing. Your code sends a request to one endpoint, usually in the OpenAI-compatible chat completions format, and names a model. The gateway decides where the request actually goes, what happens if that provider fails, and what gets recorded afterward.

Most gateways are reverse proxies, not SDKs. That distinction matters because a proxy can enforce policy for every caller in the company, including callers you did not write. A client library can only enforce policy for code that imports it.

The pattern is now common enough that cloud vendors publish reference builds. AWS, for example, documents a multi-provider generative AI gateway that runs the open-source LiteLLM proxy on ECS or EKS behind a load balancer, with Redis for caching and a database for virtual API keys.

How an LLM gateway handles a request

A single call passes through roughly seven stages. Not every gateway implements all of them, and the order varies, but this is the common shape:

1. Authenticate the caller. The application presents a virtual key issued by the gateway, never a raw provider key. The gateway maps it to a team, a budget and a set of allowed models.
2. Apply policy. Input guardrails run here: PII redaction, blocked topics, maximum prompt size, and per-key rate limits.
3. Check the cache. An exact-match cache returns a stored response for an identical prompt. A semantic cache returns one for a prompt that is merely similar.
4. Route. The gateway picks a provider and model. The rule can be static (always this model), weighted (70/30 across two deployments), or learned (a classifier decides whether the query needs the expensive model).
5. Call the provider, with retries and fallback. On a timeout, a 429 or a 5xx, the gateway retries, then falls back to the next model on the list.
6. Normalize the response. Provider-specific fields are translated back into one format, including streamed tokens and tool calls.
7. Account and log. Token counts, cost, latency, the model that actually answered, and any error are written out for dashboards and billing.

Stages 3, 4 and 5 are where a gateway earns its keep. They are also where it can quietly return something other than what you asked for.

Core features, and what each one costs you

Every gateway feature trades something. The table lists the common ones with the price attached.

The pattern in the right-hand column is consistent. Each feature moves a decision out of your application code and into configuration, where it is easier to change and harder to test.

LLM gateway vs AI gateway vs API gateway

The three terms overlap, and vendors use them loosely. The useful distinction is what each one understands about the traffic passing through it.

"LLM API gateway" and "LLM gateway" usually describe the same thing; the longer name tends to emphasize that it fronts provider APIs rather than models you host. "AI gateway" is the broader label, and it is increasingly used for products that also govern agent tool calls. Several traditional API gateway vendors now sell an AI gateway built on their existing proxy, which is why the product names cross columns. Treat the labels as marketing and compare the feature lists.

Can an LLM gateway cut model costs with routing?

Learned routing is one of the few gateway features with peer-reviewed research behind it. The idea is to send easy queries to a cheap model and hard ones to an expensive model, and let a trained classifier decide which is which.

The clearest evidence is RouteLLM, published at ICLR 2025 by researchers at UC Berkeley, Anyscale and Canva. Routing between GPT-4 and Mixtral-8x7B, their best routers cut cost relative to using GPT-4 alone by 3.66x on MT Bench while keeping 95% of GPT-4's score. The saving shrank on other benchmarks: 1.41x on MMLU at 92% of GPT-4's score, and 1.49x on GSM8K at 87%.

Two caveats travel with those numbers. First, the benchmarks are academic question sets, and even the multi-turn one, MT Bench, stops at two turns and uses no tools. Second, the paper reports that routers trained only on chat preference data did poorly on MMLU, because most of those questions were outside the training distribution. A router learns your traffic, or it learns someone else's.

For a conversational agent, the practical reading is narrower. Routing can save real money on classification, summarization and other isolated calls. Routing individual turns of a live conversation between two different models is a riskier proposition, because the agent's personality and tool behavior can change mid-call.

When you need an LLM gateway, and when you don't

A gateway is worth the extra hop when at least one of these is true:

• More than one provider. You call two or more model vendors, or plan to, and want to swap without a code change.
• More than one team. Several services or teams share provider accounts, and you need per-team keys, budgets and usage reports.
• Uptime matters more than consistency. A provider outage costs you more than an occasional answer from a different model.
• Compliance needs one choke point. Redaction, logging and data-residency rules are easier to prove when every request passes through one place.

It is probably not worth it when you call one model from one service. In that case your SDK's built-in retry logic does most of what a gateway would, with one less network hop and one less system to operate.

The cost of the hop is real but usually small next to model inference. Vendors publish their own overhead figures, and none of the explainer pages we reviewed for this topic cite an independent measurement of them. Measure it yourself, at p95 and p99 rather than on average, and under streaming.

Open-source vs managed gateways

Gateways split into three deployment models, and the choice decides who operates the failure modes described below.

• Self-hosted open source. LiteLLM proxy, the Kong and Apache APISIX AI plugins, and several newer projects. You run it, you own its uptime, and you see every log line.
• Managed service. Cloudflare AI Gateway, OpenRouter, and hosted tiers of open-source projects. Someone else operates it, and you see what they choose to expose.
• Cloud reference build. The AWS guidance mentioned above, and similar patterns on other clouds. Self-hosted, but assembled from managed parts.

The distinction matters most for third-party aggregators, which resell access to many vendors' models through one bill. With those, you are trusting the operator to call the model you named.

Six ways an LLM gateway fails silently

The explainers ranking for this topic describe what a gateway does and rarely how it fails. The dangerous failures share one property: they return HTTP 200, so uptime checks and error-rate alerts never fire.

1. The model you named is not the model that answered

In a direct test of this, researchers at UMass Boston and Arizona State audited 10 commercial LLM API gateways in a paper accepted at the ACM Internet Measurement Conference 2026. They built behavioral fingerprints for 24 models from the vendors' own APIs, then checked which model each gateway actually served.

When requests went straight to the vendor's own API, 97.09% of gpt-5 responses were identified as gpt-5. On one gateway, only 13.09% were. For each of the four models reported in the paper's main table, the official APIs scored above 95%, while several gateways fell below 60%. The authors reviewed each gateway's documentation and pricing pages and found no disclosure of the behavior.

The caveats matter. The gateways are anonymized, the results are a point-in-time snapshot, and all 10 are third-party commercial aggregators. A gateway you host yourself is not substituting models behind your back, although your own fallback rules can produce the same effect.

2. Long conversations lose context

The same study ran a 25-turn conversation that planted a fact early, updated it midway, and asked for it at the end. Through the official gpt-4o API, all five runs recalled it at turns 24 and 25. Several gateways failed those final checkpoints, and one showed low recall and low cache use together, which the authors say could indicate silent truncation or model switching.

For any multi-turn agent, this is the failure that looks most like a model problem. The model is fine. It never received the full conversation.

3. The bill does not match the tokens

The audit also compared each gateway's charges against its own published prices and reported token counts for gpt-4o. Most matched exactly. One charged 7.6% more than expected, and another charged 62.8% more on similar token usage. If you meter spend through a third-party gateway, reconcile it against the provider's own usage data at least once.

4. Retries make an outage worse

Retries are the default fix for transient errors, and they are also how a brief provider hiccup turns into a sustained one. When many clients retry on the same fixed interval, they synchronize and hit the provider's rate limit together on every window. A July 2026 preprint cataloguing gateway failure modes documents exactly this thundering-herd pattern in an LLM gateway client.

Some errors should not be retried at all. Anthropic's API error reference notes that a 429 caused by a usage tier's monthly spend cap carries no retry-after header and "keeps failing until access resumes." A gateway that retries it, or falls back to the same account on another model, burns latency for nothing. OpenAI's error guide describes a different trap: a slow_down 429 that can arrive even when traffic is inside the requests-per-minute and tokens-per-minute limits. Configure retries per error type, with jitter, and never treat every 429 the same way.

5. Streaming and tool calls break in translation

Translating each provider's streaming format into one common format is hard, and tool calls are where it breaks. The same preprint documents a streaming bug in which an index collision merged independent tool calls into malformed JSON, while every request returned success. Because each provider streams tool calls differently, a fix verified on one provider says nothing about the next one in your fallback chain.

The model produced a valid tool call. The gateway delivered a broken one. Infrastructure dashboards show nothing wrong.

6. The fallback model behaves like a different agent

A fallback is a model swap you did not schedule. Your prompt was tuned against the primary model, and the fallback may follow it differently, call tools in a different order, or handle a refusal case another way.

Cekura's benchmark data shows how large that difference can be with nothing else changed. In a controlled Cekura experiment on a Telnyx voice agent, the prompt, tools, voice, speech-to-text, text-to-speech and evaluators stayed fixed, and only the LLM changed. Kimi K2.6 reached 88.1% pass³, the share of the 59 evaluators passing on all three repeats, against 76.3% for GPT-4.1, and cut P95 turn latency from 5.00 seconds to 3.22 seconds. On the red team, safety and privacy category, the direction reversed: GPT-4.1 held 100.0% and Kimi K2.6 dropped to 80.0%.

That study covers one platform, one pair of models and 177 calls per model, run in July 2026 on an earlier evaluation suite, so it is evidence of variance, not a ranking. The lesson for gateway design is the reversal. A fallback can be better on average and worse in exactly the category you care about most, and an aggregate score will hide it.

Why the gateway matters more for voice agents

A text chatbot that takes an extra second to answer is slower. A voice agent that takes an extra second leaves dead air, and the caller starts talking over it. Every gateway decision lands inside that turn budget.

Three gateway behaviors hit voice hardest. A retry triggered by a timeout adds the whole timeout before the second attempt even starts. A fallback to a slower model shifts the whole latency distribution, which shows up in the tail long before it moves the average; the guide to what P99 latency means for voice AI agents walks through why. And a gateway that buffers a streamed response before forwarding it destroys time to first token (TTFT), even when total generation time looks fine.

The fix is to set gateway timeouts from the voice turn budget, not from gateway defaults, and to decide in advance what the agent says while a fallback is in flight. Our guide to handling LLM stalls and timeouts in voice agents covers filler responses and timeout ladders in detail.

How to monitor an LLM gateway

Standard gateway dashboards track request count, error rate, latency and spend. Those catch the loud failures. The silent ones need three more signals.

• Requested model vs responding model. The OpenTelemetry GenAI semantic conventions define both gen_ai.request.model and gen_ai.response.model, and specify that the response value must be the exact name of the model actually used. Record both on every span, and alert when they diverge more often than your fallback rate explains. The conventions are still marked Development, so pin the version you instrument against.
• Fallback rate by route. A fallback rate that climbs from 1% to 15% is an incident, even if every request succeeded.
• Error class, not just error count. The same conventions define error.type. Separate spend-cap 429s, rate-limit 429s, timeouts and 5xx errors, because each one needs a different retry policy.

Infrastructure signals still cannot tell you whether the answer was right. That needs evaluation of the conversation itself. Cekura monitors production calls for voice and chat agents by scoring each transcript on the metrics you define, including a predefined latency metric per turn. Tag each call with the provider or gateway route that served it, using Cekura's metadata field, and you can segment quality and latency by route to see whether fallback traffic performs worse than primary traffic.

How to test an LLM gateway before production

Most teams test the primary route and assume the fallback works. The fallback is the route that runs during an incident, when you can least afford a surprise. A workable test plan covers five cases:

1. Primary vs fallback on the same scenarios. Run an identical scenario set against each model in the fallback chain and compare results side by side.
2. Forced failover mid-conversation. Make the primary fail on turn 6 of a 12-turn conversation and check that the fallback keeps context, persona and tool state.
3. Error-class handling. Inject a spend-cap 429, a rate-limit 429, a timeout and a 503, and confirm each follows its intended retry path.
4. Streaming with tools. Run tool-calling scenarios with streaming on, against every provider in the chain.
5. Long-conversation recall. Plant a fact early, update it, and ask for it near the end, mirroring the audit's 25-turn design.

Cekura runs the first of those comparisons directly. Cekura executes the same scenarios against two agent configurations, for example one pointed at the primary model and one at the fallback, and the dashboard's Compare view shows the results side by side on your metrics. The same approach applies when a provider ships a new model version; see how to test new model versions against real production calls.

When a production call exposes a gateway failure, Cekura can turn that call into regression scenarios, so the next routing or fallback change is tested against the case that already broke once. Cekura tests, monitors and improves voice and chat agents, and a gateway is one more layer that changes what those agents do. If you run a fallback chain today and have never tested the fallback, book a Cekura demo and run your first comparison against it.

Frequently asked questions

What is the difference between an LLM gateway and an AI gateway?

An LLM gateway fronts language model APIs and meters tokens, cost and latency per model. An AI gateway is the broader term, and it usually adds governance for agent tool calls, MCP servers, embeddings and image models. Many products use both labels. Judge them by features, not name.

Does an LLM gateway add latency?

Yes, every gateway adds a network hop and some processing, and guardrails or caching lookups add more. Vendors publish their own overhead figures, but independent measurements are scarce. Measure p95 and p99 with streaming on, in your own region, because averages hide the tail that voice callers hear.

Is an LLM gateway the same as an LLM router?

No. Routing is one feature of a gateway. A router decides which model receives a request, sometimes with a trained classifier. A gateway also handles authentication, retries, fallbacks, rate limits, caching, spend tracking and logging around that decision.

Can a third-party LLM gateway serve a different model than the one I requested?

It can, and a 2026 audit of 10 commercial gateways found evidence that some did. On the weakest gateway measured for gpt-5, only 13.09% of gpt-5 responses matched gpt-5's fingerprint, against 97.09% on the official API. Results were anonymized and point-in-time. Log the responding model and compare it with the requested one.

Is an LLM gateway the same as an LLM proxy?

Mostly, but not exactly. A plain proxy forwards requests and may add authentication or logging. An LLM gateway is a proxy that also understands model traffic: it meters tokens, runs model-aware fallbacks, caches prompts and enforces per-key budgets. Most gateways are built as reverse proxies, so the terms overlap.

Should a voice agent use an LLM gateway?

Often yes, because provider outages during a live call are costly. Set timeouts from the voice turn budget rather than gateway defaults, make sure tokens stream through without buffering, and test every model in the fallback chain on the same conversation scenarios before it ever takes a real call.