LLM Gateways Explained:
Model Routing, Failover, Caching, Security & Costs

ENGINEERING GUIDE

What is an LLM gateway?

An LLM gateway is a central policy and routing boundary between an application and one or more large-language-model APIs. It can authenticate application callers, isolate provider credentials, enforce budgets and rate limits, choose a model route, apply controlled retries and record redacted request telemetry. A gateway does not automatically improve model quality, prevent prompt injection or guarantee uninterrupted streaming.

If you run an AI agent that uses Claude for complex reasoning, a smaller model for document classification and another model in a restricted region, the gateway can make the decision policy visible and auditable. The application still owns task-specific acceptance tests and downstream action safety. Read the AI model selection guide for quality evaluation and production deployment for durable work and rollback.

ENGINEERING GUIDE

Production LLM gateway architecture: app, policy, router and providers

1. Authenticate: establish trusted tenant/application identity and map it to approved models and budget.
2. Validate input: bound schema, payload size, sensitive-data class and tool/JSON-mode requirements.
3. Apply admission policy: check RPM, TPM, in-flight request concurrency, cost cap and region requirements.
4. Route: choose a pinned provider/model based on measurable capability, task policy and current health.
5. Execute: send provider-compatible requests, interpret errors and account for actual attempts; nested retries must be coordinated.
6. Validate output: verify expected schema, model-specific refusal semantics, tool calls and end-user acceptance.
7. Observe: emit correlation IDs, usage, provider error classes, tail latency and redacted logs under explicit retention policy.

ENGINEERING GUIDE

Do you need an LLM gateway or a thin SDK wrapper?

An AI gateway introduces an extra hop and control plane. It is justified when it reduces duplicated policy code or operational risk, not merely because a demo makes it easy to list five providers. Model selection can remain in the application for small deployments.

ENGINEERING GUIDE

LLM routing strategies: reliability, quality, latency and cost

Routing can be static (task-to-model mapping), weighted (split traffic across similar deployments), capability-aware (JSON schema, context size, tools, vision, approved region) or adaptive (route health, quotas, cost and verified task outcomes). Failover is a separate choice from initial routing.

A common trap is treating the phrase “OpenAI-compatible API” as proof that any two models are interchangeable. Tool-call object shapes, content types, error semantics, function calling, structured outputs, reasoning fields, streaming and safety behavior can differ. Build a model capability registry instead of blindly forwarding every flag.

{"route_policy":"support-answering-v3","allowed_regions":["eu"],"required_capabilities":["json_schema"],"max_attempts_total":2,"model_primary":"providerA:pinned-model-version","fallback":["providerB:evaluated-model-version"],"tenant_budget_usd":100,"log_prompt_content":false}

ENGINEERING GUIDE

Retries, provider failover, circuit breakers and streaming boundaries

Rate limits are often measured in requests per minute and tokens per minute, and providers can enforce multiple or smaller-window constraints. A gateway should not compensate for a real 429 by blasting the same provider with immediate parallel retries. Respect provider-specific Retry-After behavior, use bounded backoff with jitter, and enforce a total request attempt/time/cost budget. OpenAI documents rate limits and backoff patterns in official guidance.

A circuit breaker temporarily stops sending traffic to a failing destination and probes recovery under a defined policy. It reduces cascading failure but can also concentrate traffic on a backup that lacks quota. Consider budget caps, health checks, regional restrictions and manual fallback override. And critically: fallback can increase HTTP availability while decreasing accepted-task quality.

SXF ORIGINAL ENGINEERING LAB

Original SXF evidence: deterministic HTTP gateway fault-injection study

We created and ran a real local HTTP server with two mock upstream destinations. A Python client sent actual HTTP POST requests over the loopback interface and validated returned JSON. For each of three independent policies, we evaluated the same 80 logical tasks. The primary mock returned 50 immediately valid responses, ten rate limits that recover on the second attempt, ten persistent server errors and ten persistent rate limits. The backup returned valid responses for 15 of 20 fallback calls, and five HTTP 200 responses with an invalid schema.

The fallback policy rejected five malformed HTTP 200 responses instead of reporting them as successful completions. The modelled cost used $0.002 per attempted primary HTTP request and $0.003 per attempted backup request, including errors. These are invented lab coefficients transparently disclosed as assumptions—not vendor prices.

Reproduce: run python sxf_gateway_http_lab.py from your terminal. No API keys, internet connection, paid model calls or third-party dependencies are required. The source checks expected results with assertions; editing the fault schedule changes the scenario and requires new conclusions.

ENGINEERING GUIDE

LLM caching: prompt-prefix reuse vs storing an answer

OpenAI and Anthropic each document provider-side prompt caching; it is not equivalent to persisting and replaying a finished response. Cloudflare documents AI Gateway caching as a different gateway feature. Do not claim that the gateway's cache is automatically compatible with every provider's prompt-caching semantics.

Safe response cache key components include tenant/policy identity, model and version, instruction version, tool schema version, relevant retrieval corpus version and request-normalization rules. Cache bypass or tight expiration is often the right default for personal data, regulated records, live facts, tool calls and high-consequence decisions. Never turn on cross-tenant answer reuse just to improve a hit-rate dashboard.

ENGINEERING GUIDE

API key security, data privacy and zero-trust boundaries

A well-designed gateway can avoid distributing raw vendor API credentials across client apps. Its own admin API and virtual keys become sensitive control points. Apply short-lived or rotated credentials, allowlists for provider egress, tenant-scoped policies and explicit secret ownership. Never accept a tenant label or spending policy solely from an LLM prompt.

A unified gateway is also a concentrated failure and trust boundary. Deploy it with service authentication, documented restore procedures and operational visibility. Detailed agent permission and prompt injection issues belong in the AI Agent Security Guide.

ENGINEERING GUIDE

RPM, TPM, concurrency, rate-limit recovery and budget enforcement

Request rate (RPM), token throughput (TPM), concurrent in-flight calls and per-tenant spend are related but not interchangeable. A request might be cheap but produce many output tokens. Concurrency limits manage saturation while token budgets depend on provider accounting of input/output and, where available, cached tokens.

Distributed gateways need coordinated counters and time windows or providers will see aggregate load that exceeds supposedly enforced limits. On a rate-limit event, avoid multiple retry layers stacking: SDK, gateway and application retries can multiply attempts.

DOCUMENTATION-BASED COMPARISON

LiteLLM vs Portkey vs Cloudflare AI Gateway vs Kong: what to evaluate

The following comparison summarizes publicly documented product approaches, not a vendor ranking or a claim that every feature is available on every plan. Vendor documentation and product details are live systems and should be checked before procurement. We did not benchmark these products.

Do not declare a speed, price or reliability winner from these feature lists. Different service tiers, regions, connection reuse, model versions and token budgets make generic rankings invalid. Evaluate each candidate's official deployment documentation for the exact plan you intend to run.

INTERACTIVE ENTERPRISE TOOL

Interactive SXF AI Gateway Decision Matrix

Choose your application's constraints to generate an evaluation plan. No vendor is automatically recommended or scored. All computation happens in your browser; no credentials or personal prompts are requested.

ENGINEERING GUIDE

LLM gateway cost: model attempts, caching, gateway overhead and accepted outcomes

For procurement, separate provider inference spend, gateway service fees/runtime, telemetry and storage, cache infrastructure, and human/incident costs. Retrying a failed call may be billable depending on when it failed and provider terms. A cached response may skip generation but create privacy and freshness obligations. Count every attempt, not only the visible answer.

See SXF Model Pricing, AI Agent Cost and ROI, and AI Model Cost Calculator for the adjacent inference and business cost layers.

ENGINEERING GUIDE

Observability and SLOs: measure the accepted task, not the HTTP status

Link end-user task IDs to provider request attempts and redact sensitive prompt bodies unless approved. OpenTelemetry GenAI conventions can support trace attributes, but keep instrumentation version-aware and do not blindly collect private prompt or completion text. For full span and incident analysis read AI Agent Observability.

ENGINEERING GUIDE

A ten-step production rollout checklist

1. Define a narrow business workflow and independently verified output contract.
2. Inventory provider models and pin versions, tool, streaming, context and structured-output capabilities.
3. Choose an owner for gateway secrets and tenant authorization.
4. Set RPM/TPM/concurrency/spend and denial behavior per tenant and provider.
5. Implement total retry budgets, jitter, circuit breaker, fallback compatibility tests and recovery drills.
6. Decide caching per information class, with private data defaulting to no response-cache reuse.
7. Publish redacted logs, attempt-level traces and cost-per-accepted-outcome dashboards.
8. Replay fault cases 429/5xx/timeouts, invalid 200, partial stream and unsupported tools.
9. Canary a pinned version with abort rules, rollback and a defined treatment of in-flight calls.
10. Record outcomes, source documentation, owner approvals and review the configuration after provider changes.

ENGINEERING GUIDE

Ten LLM gateway mistakes that hurt reliability and cost

1. Calling HTTP 200 a successful business outcome.
2. Assuming OpenAI-compatible interfaces imply interchangeable model behavior.
3. Stacking client, gateway and provider retries without a global attempt budget.
4. Retrying a potentially irreversible tool action without idempotency and verification.
5. Switching models after part of a streamed answer has already reached the user.
6. Routing a sensitive tenant's traffic into a nonapproved region during fallback.
7. Reusing cached answers across tenants, users or permission versions.
8. Counting token price but ignoring extra attempts, reviews, observability and operations.
9. Letting the model grant its own provider credentials, permissions or budgets.
10. Publishing a vendor ranking based on incompatible tiers, models or unexplained test fixtures.

ENGINEERING GUIDE

Frequently asked questions about LLM and AI gateways

EVIDENCE AND REFERENCES

Primary sources, SXF experimental method and editorial independence

Evidence rule: the comparison table reflects official documented features, not independently benchmarked vendor performance. Our original local HTTP experiment is reproducible and explicitly separates actual loopback requests from mock upstream behavior and synthetic cost coefficients. Provider limits, plans and compatibility need verification for your exact production versions.

INTERNAL TOPICAL CLUSTER

Build the complete SXF enterprise AI infrastructure stack