AI Gateway Overview
What Apinizer AI Gateway Gives You
Point any OpenAI SDK (or compatible client) at one base_url and reach every LLM provider you've connected — no per-provider integration code.
Route requests to OpenAI, Anthropic, Google Vertex/Gemini, AWS Bedrock, and OpenAI-compatible self-hosted engines from the same interface. Non-OpenAI responses are normalized back to the OpenAI format automatically.
PII masking, prompt-injection guards, semantic caching, token quotas, and per-model cost tracking are available out of the box — no separate tooling to wire up.
Usage and cost reports broken down by person, team, project, model, and deployment type (cloud vs. on-premise) give you a single source of truth for AI spend.
True streaming is supported end-to-end: responses are relayed to the client chunk-by-chunk as they arrive from the provider, with no artificial buffering.
Each AI proxy has its own Streaming setting on the AI Proxy Routing screen. When it's off, the gateway always processes requests as a single, non-streaming response — even if the client sets stream: true in the request body, the gateway overrides it to false (and drops stream_options) before forwarding to the provider. This keeps token counts and cost accurate in the Traffic report regardless of what an individual client sends. When Streaming is on, the request is unaffected and relayed via server-sent events as described above.
New to LLM terms like tokens, embeddings, or RAG? See AI Fundamentals for a primer — this page focuses on what Apinizer AI Gateway does with them.
Supported LLM Providers
- Cloud: OpenAI, Anthropic (Claude), Azure OpenAI, Google Vertex AI (Gemini), AWS Bedrock
- Self-hosted: vLLM, Ollama, Hugging Face TGI
- Custom: any OpenAI-compatible API endpoint
Each provider is configured once as a connection, with encrypted credentials and a deployment type. See LLM Providers and Connections for the full list and setup steps.
Request Flow
Every request to the gateway flows through the same pipeline, whether it targets OpenAI or a self-hosted model:
- Intake — the request is validated and parsed as OpenAI-format JSON
- Guardrails (optional) — PII masking, prompt-injection checks, content filters
- Semantic cache (optional) — a similar recent request short-circuits the call and returns a cached response
- Rate limiting — token and USD quotas are checked for the request's scope
- Routing — a provider connection is selected based on the requested model and your routing/failover configuration
- Inference — the request is translated to the provider's native format and forwarded
- Response — the provider's response is normalized back to OpenAI format and streamed to the client along with usage metrics
Deployment Type
Every provider connection is tagged Cloud or On-Premise, so usage and cost reports can distinguish provider-hosted traffic from traffic served by infrastructure you run yourself. See LLM Providers and Connections for details.
AI Gateway vs. Classic API Proxy
An AI Gateway is the same ApiProxy entity in Apinizer — it goes through the same create/deploy/undeploy flow, the same environment management, the same proxy list, and the same endpoints as a classic API proxy. The difference is how the request is routed to the backend:
- Classic API proxy → the
routingobject (address list, circuit breaker, mTLS, proxy server, NTLM, etc.) - AI Gateway → the
aiRoutingobject (LLM provider connection, model, primary pool, conditional routing, failover chain) — see Routing and Failover
The AI Gateway runtime never reads the classic routing object, so settings tied to it don't apply to an AI Gateway.
The following 12 settings endpoints (PATCH .../apiProxies/{apiProxyName}/settings/<X>/) now return HTTP 400 when called on an AI Gateway. Previously they returned HTTP 200 but had no effect at all (a silent no-op) — that's now an explicit rejection:
circuit-breaker · proxy-server · mtls · ntlm · connection · error-handling · custom-message · grpc · websocket · addresses · routing-status · metadata
On an AI Gateway, settings like mTLS, proxy server, and circuit breaker live under aiRouting instead — the correct endpoint is PUT .../apiProxies/{apiProxyName}/ai-routing/. The metadata endpoint's 400 check is conditional: it only triggers when the request body sets fixSoapApiPortType (a field meaningful only for classic routing) — a plain metadata update keeps working on an AI Gateway.
11 type-agnostic settings endpoints — CORS, cache, idempotency, XML/JSON error templates, forwarded-ip-header, spec-access-type, client-route, keys, maintenance mode, trace, and traffic log — continue to work on AI Gateways unchanged.
See API Reference: API Proxy Settings for full details.
Try It: OpenAI SDK Quickstart
Use Apinizer AI Gateway with the Python OpenAI SDK or any compatible client — no client-side code changes beyond base_url and the API key:
from openai import OpenAI
client = OpenAI(
api_key="your-apinizer-credential-key",
base_url="https://your-apinizer-gateway.com/api/ai/v1"
)
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello"}],
stream=True
)
for chunk in response:
print(chunk.choices[0].delta.content, end="")
The client interface is always OpenAI-format — there is no separate Anthropic Messages or Gemini inbound format to learn. Apinizer translates each request into the target provider's native format and normalizes the response back to the OpenAI canonical format, so the same client code works against OpenAI, Anthropic, Google Vertex/Gemini, AWS Bedrock, or any OpenAI-compatible endpoint.
For a full step-by-step walkthrough — connecting a provider and sending your first request — see the AI Gateway Quickstart.
Next Steps
Connect a provider and send your first request
Configure provider connections and deployment types
Bundled models, pricing, and cost tracking
Failover chains, cost- and latency-aware routing
Inject fixed system instructions into every request
Let clients reference a reusable, parameterized prompt by name
Set up quota rules and monitoring
Configure pricing and multi-currency display
View usage and cost breakdowns
Configure inbound/outbound Agent2Agent communication
Add DLP, loop, off-topic, and oversized guards
Use audio (STT/TTS) and image generation endpoints
Inspect the request chain in a timeline and re-execute it