AI Gateway Overview
What Apinizer AI Gateway Gives You
Point any OpenAI SDK (or compatible client) at one base_url and reach every LLM provider you've connected — no per-provider integration code.
Route requests to OpenAI, Anthropic, Google Vertex/Gemini, AWS Bedrock, and OpenAI-compatible self-hosted engines from the same interface. Non-OpenAI responses are normalized back to the OpenAI format automatically.
PII masking, prompt-injection guards, semantic caching, token quotas, and per-model cost tracking are available out of the box — no separate tooling to wire up.
Usage and cost reports broken down by person, team, project, model, and deployment type (cloud vs. on-premise) give you a single source of truth for AI spend.
True streaming is supported end-to-end: responses are relayed to the client chunk-by-chunk as they arrive from the provider, with no artificial buffering.
Each AI proxy has its own Streaming setting on the AI Proxy Routing screen. When it's off, the gateway always processes requests as a single, non-streaming response — even if the client sets stream: true in the request body, the gateway overrides it to false (and drops stream_options) before forwarding to the provider. This keeps token counts and cost accurate in the Traffic report regardless of what an individual client sends. When Streaming is on, the request is unaffected and relayed via server-sent events as described above.
New to LLM terms like tokens, embeddings, or RAG? See AI Fundamentals for a primer — this page focuses on what Apinizer AI Gateway does with them.
Supported LLM Providers
- Cloud: OpenAI, Anthropic (Claude), Azure OpenAI, Google Vertex AI (Gemini), AWS Bedrock
- Self-hosted: vLLM, Ollama, Hugging Face TGI
- Custom: any OpenAI-compatible API endpoint
Each provider is configured once as a connection, with encrypted credentials and a deployment type. See LLM Providers and Connections for the full list and setup steps.
Request Flow
Every request to the gateway flows through the same pipeline, whether it targets OpenAI or a self-hosted model:
- Intake — the request is validated and parsed as OpenAI-format JSON
- Guardrails (optional) — PII masking, prompt-injection checks, content filters
- Semantic cache (optional) — a similar recent request short-circuits the call and returns a cached response
- Rate limiting — token and USD quotas are checked for the request's scope
- Routing — a provider connection is selected based on the requested model and your routing/failover configuration
- Inference — the request is translated to the provider's native format and forwarded
- Response — the provider's response is normalized back to OpenAI format and streamed to the client along with usage metrics
Deployment Type
Every provider connection is tagged Cloud or On-Premise, so usage and cost reports can distinguish provider-hosted traffic from traffic served by infrastructure you run yourself. See LLM Providers and Connections for details.
AI Gateway vs. Classic API Proxy
An AI Gateway is the same ApiProxy entity in Apinizer — it goes through the same create/deploy/undeploy flow, the same environment management, the same proxy list, and the same endpoints as a classic API proxy. The difference is how the request is routed to the backend:
- Classic API proxy → the
routingobject (address list, circuit breaker, mTLS, proxy server, NTLM, etc.) - AI Gateway → the
aiRoutingobject (LLM provider connection, model, primary pool, conditional routing, failover chain) — see Routing and Failover
The AI Gateway runtime never reads the classic routing object, so settings tied to it don't apply to an AI Gateway.
The following 12 settings endpoints (PATCH .../apiProxies/{apiProxyName}/settings/<X>/) now return HTTP 400 when called on an AI Gateway. Previously they returned HTTP 200 but had no effect at all (a silent no-op) — that's now an explicit rejection:
circuit-breaker · proxy-server · mtls · ntlm · connection · error-handling · custom-message · grpc · websocket · addresses · routing-status · metadata
On an AI Gateway, settings like mTLS, proxy server, and circuit breaker live under aiRouting instead — the correct endpoint is PUT .../apiProxies/{apiProxyName}/ai-routing/. The metadata endpoint's 400 check is conditional: it only triggers when the request body sets fixSoapApiPortType (a field meaningful only for classic routing) — a plain metadata update keeps working on an AI Gateway.
11 type-agnostic settings endpoints — CORS, cache, idempotency, XML/JSON error templates, forwarded-ip-header, spec-access-type, client-route, keys, maintenance mode, trace, and traffic log — continue to work on AI Gateways unchanged.
See API Reference: API Proxy Settings for full details.
Try It: OpenAI SDK Quickstart
Use Apinizer AI Gateway with the Python OpenAI SDK or any compatible client — no client-side code changes beyond base_url and the API key:
from openai import OpenAI
client = OpenAI(
api_key="your-apinizer-credential-key",
base_url="https://your-apinizer-gateway.com/api/ai/v1"
)
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello"}],
stream=True
)
for chunk in response:
print(chunk.choices[0].delta.content, end="")
The client interface is always OpenAI-format — there is no separate Anthropic Messages or Gemini inbound format to learn. Apinizer translates each request into the target provider's native format and normalizes the response back to the OpenAI canonical format, so the same client code works against OpenAI, Anthropic, Google Vertex/Gemini, AWS Bedrock, or any OpenAI-compatible endpoint.
For a full step-by-step walkthrough — connecting a provider and sending your first request — see the AI Gateway Quickstart.
Model Discovery: GET /v1/models
The gateway serves an OpenAI-compatible model-listing endpoint at GET /v1/models, so client.models.list() in the OpenAI Python SDK — and OpenAI-compatible tooling built on it, such as LangChain and LiteLLM — can discover which models are available instead of having the model name hardcoded:
models = client.models.list()
for model in models:
print(model.id)
The response body:
{
"object": "list",
"data": [
{ "id": "gpt-4o", "object": "model", "created": 0, "owned_by": "openai" },
{ "id": "claude-3-5-sonnet", "object": "model", "created": 0, "owned_by": "anthropic" }
]
}
created is always 0Apinizer's model catalog carries no creation timestamp for a model — OpenAI SDKs parse the field but don't attach meaning to it. Extra catalog detail such as context window, pricing, and capabilities is deliberately left out of this response as well; the discovery endpoint isn't a surface for exposing that.
The List Is This Proxy's Serving Surface, Not the Provider Catalog
The response is not the LLM provider's full catalog — it's the union of models this specific AI proxy's aiRouting configuration can actually serve (the primary model, plus any primary pool, conditional routing, and failover chain entries), after three elimination checks run in order:
- A candidate whose provider connection is not deployed or disabled is dropped.
- A candidate whose model id isn't in that provider connection's own allow-list (when one is configured) is dropped.
- A candidate past its catalog sunset date is dropped — a model that would already fail with a 400 at request time isn't advertised here. A model merely flagged deprecated, with no sunset date reached, still appears.
An empty result is not an error: {"object":"list","data":[]} with HTTP 200 is a valid response when nothing survives the checks above.
This endpoint only lists; OpenAI's single-model retrieval endpoint (GET /v1/models/{model_id}, client.models.retrieve(...) in the SDK) is out of scope and does not match here.
Authentication and the Normal Policy Chain Still Apply
Discovery is not a .well-known-style authentication-exempt path — the proxy's normal policy chain, including any auth policy and its block anonymous requests setting, runs on this request exactly as it does on any other, so an unauthenticated discovery call against a proxy with that setting on still gets a 401.
A discovery request never reaches an LLM provider, so no token or cost usage is recorded for it. The call still shows up in the Traffic report, but its usage and cost columns stay empty. The endpoint accepts GET only — any other method returns 405.
Next Steps
Connect a provider and send your first request
Configure provider connections and deployment types
Bundled models, pricing, and cost tracking
Failover chains, cost- and latency-aware routing
Inject fixed system instructions into every request
Let clients reference a reusable, parameterized prompt by name
Set up quota rules and monitoring
Configure pricing and multi-currency display
View usage and cost breakdowns
Configure inbound/outbound Agent2Agent communication
Add DLP, loop, off-topic, and oversized guards
Use audio (STT/TTS) and image generation endpoints
Inspect the request chain in a timeline and re-execute it