The Story of Apinizer AI Gateway: We Did Not Build a Second Box, We Redrew a Boundary
A differentiated data path, an undivided control plane.
The outcome
Apinizer AI Gateway is live today. More than 17 LLM providers behind a single OpenAI-compatible endpoint; token and cost budgets, a prompt firewall, PII redaction, MCP and A2A governance — all of it on top of the same gateway that already runs your REST APIs. No separate console, no separate identity system, no separate on-call rotation.
But this article is not a feature list. What I actually want to explain is something else: why did we build this product inside the existing gateway instead of as a second box? And how did a warning we wrote to ourselves fifteen months ago shape that decision?
The path the market recommended
When LLMs entered enterprise systems, the market's advice was clear: you have a gateway for your APIs, now buy a separate AI Gateway for your AI traffic. A second console, a second deployment pipeline, a second vendor, a second on-call rotation.
We did not take that path when we built this product. But we did not rush to that decision either — because fifteen months ago, before we had even started designing the AI module, we had written ourselves a warning that pointed in exactly the opposite direction.
Our own warning: April 2025
In April 2025 we published an article titled Gateway Zoo: API, Event, Kafka and AI Gateways from the Perspective of Conway's Law. It started from Conway's Law: "Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations."
The concern in that article was this: the scope of API Gateways was being continuously expanded with concepts such as Event Gateway, Kafka Gateway, AI Gateway and Agent Gateway. That expansion carried the risk of recreating the trap the ESBs (Enterprise Service Bus) of the past fell into — growing on the claim of being the "one tool that solves everything," accumulating leaked business logic, becoming bloated over time and eventually turning into an obstacle to change.
Our conclusion was clear: in large-scale, long-lived systems, distinct functions should be designed as separate components connected through well-defined interfaces — everything should not be piled into a single runtime. Back then, Agent Gateway was still a theoretical category. If we had been making a product decision, that article would have pushed us toward "build a separate AI Gateway."
When we actually sat down to design the AI module, the work started with exactly that discipline: taking our own warning seriously. The question was not "where do we put AI"; it was "how do we build this properly without falling into gateway sprawl."
LLMs arrived, and the question sharpened
In enterprise software, managing APIs through a single door had long been the API Gateway's job: authentication, quota, logging, routing, security policies. Then applications started talking not just to a REST endpoint, but to OpenAI, to Anthropic, to Azure, to Bedrock, to self-hosted models. Multiple models at once, streaming responses, token-based cost, prompt injection risk, the need for answers grounded in company documents.
Three scenes summed up why a classic API Gateway was not enough on its own:
Switching providers breaks operations. A team starts with GPT, looks at a self-hosted model when costs rise, then wants to move to another cloud when regulation arrives. Each time, client code, key management and error handling get rewritten.
The real bill is not the request count, it is tokens. At the end of the month "API calls look low" but the invoice has ballooned — long prompts, bloated context and silent retries burned the tokens. In classic APIs, TPS is usually enough; with LLMs, visibility shifts to tokens, models and cost.
Security cannot be left to application code. A company procedure gets pasted into a chatbot's system prompt, PII goes to the model unredacted, prompt injection never crosses anyone's mind. The model makes things up on a clause it does not know; everyone assumes "it works."
All three scenes led to the same conclusion: LLM traffic is not a "normal API." A control layer was needed. But our own April warning was still on the table — so should that control layer have been a separate product?
Where we drew the boundary
At this point we questioned an assumption: the question "separate product or integrated" was on the wrong axis. The real question was this — what needs to differentiate, and what needs to stay unified?
The ESB trap was functions becoming entangled at the runtime and at the responsibility boundary: a single runtime, leaked business logic, a monolith that cannot change. But the risk that emerged with LLM traffic was the exact opposite: every AI tool bringing its own user management, its own log format, its own key store — the fragmentation of governance. One of these is called bloat, the other shadow IT.
These two do not sit on the same axis. What needed to be separated was the technical character of the data path — streaming, TTFT, token economics and long-lived open connections simply cannot be compared to the millisecond-level request/response cycle of classic REST, and we could not ignore that. But what needed to stay unified was the control plane: identity, policy, deployment, audit, analytics. When those get divided, the platform team ends up dealing with a second identity system, a second audit trail and a second on-call rotation — and that cost comes not from any technical necessity of LLM traffic, but from packaging it as a separate console, a separate identity management and a separate audit trail.
There is a distinction here that should not be missed: it is true that the data path needs to differentiate, but who takes on that differentiation is a separate question. The customer does not need to stand up a second runtime or operate a second infrastructure for it — we handle that inside the product, behind a single platform. The team looking from the outside sees one gateway and one console; the streaming-specific connection handling, backpressure and timeout strategies run invisibly inside that single product.
That is where the decision came from: in the gateway, AI became not a new product but a new proxy type — a first-class citizen alongside REST, SOAP and gRPC. It runs in the same runtime, is authenticated by the same Identity Manager (OAuth2/OIDC/JWT/mTLS), logs into the same Analytics Engine, and goes through the same deployment pipeline. The platform team learns not a new product, but a new tab.
The April warning was not wrong; what we had not clarified back then was which axis the separation belongs on. The ESB trap and the shadow IT trap are two ends of the same Conway tension — one wants the runtime kept undivided, the other wants governance kept undivided. Satisfying both at once was the decision itself.
What actually differentiated in the data path
We did not leave this as a claim. We did not ignore how LLM traffic differs from REST; we solved it item by item inside the product — without the customer having to stand up a separate system:
It does not leak PII while streaming. What happens if an identity number is split in two exactly at a chunk boundary? A naive redactor scans each packet on its own, sees no "complete pattern" in either, and both halves leak. We solved it with a sliding window — a half-finished pattern carries over into the next chunk, without buffering the entire response. Speed is preserved, and so is privacy.
A budget is not a report, it is a gate. Instead of learning about token spend at the end of the month, we work with pre-reservation: when a request arrives, the estimated cost is reserved up front and reconciled against the actual once the stream ends. A four-level hierarchy — credential, organization, proxy, project — with 50/80/90/100% threshold alerts, and options to block, downgrade to a cheaper model or fail over on overrun.
The cache does not bypass redaction. A semantic cache lowers cost, but set up wrong it creates a dangerous hole: if the raw response is written to the cache before redaction is applied, redaction never runs on a cache HIT. The rule is clear: what gets cached is always the safe form that would go to the user, never the raw body.
The provider circuit breaker and failover never disturb the client. If a provider degrades, the next one in the chain takes over; the client keeps talking in the same OpenAI-compatible format. Self-hosted models sit in the same pool as SaaS models, governed by the same budgets and guards.
The agent loop is not unbounded. Tool-call feedback loops are capped at a defined number of turns; there is no runaway chaining. When a tool result comes back, it passes through the entire policy chain again — because the risk is not only in what the agent sends, but also in what it reads from the outside world and carries into the model.
None of this is "behave like REST." Each was designed specifically for the characteristics unique to LLM traffic — token economics, the non-atomic nature of streaming, the bidirectional data flow of agents. A differentiated data path; an undivided control plane.
Product tour: from the why to the solution
The product we are launching today is not a feature list that happened to come together — every piece answers one of the scenes above.
- Against provider lock-in: many LLM providers behind a single OpenAI-compatible endpoint, routing by cost, latency or request, proactive failover — the provider changes without the application code changing.
- Against uncontrolled cost: a four-level, real-time token/USD budget — the gate closes before the call goes out.
- Against the "boundary" defined in the prompt: topic, injection, context integrity, DLP and loop guards — five orthogonal signals, applied centrally before the model ever sees the request. With inline/async/shadow modes, so a new rule can be measured first and enforced afterwards.
- Against the provider key sitting in the application layer: virtual keys — the real key never reaches the application, and rotation is not a deployment.
- Against the company document pasted into a chatbot: RAG as a product-level capability — Knowledge Base, document ingestion, multi-tenant isolation; a managed policy, not copy-paste into a prompt.
- Against "I cannot see why it slowed down or where the money went": latency visibility split into TTFT/TPOT, a rich AI field set on every record, and a trace that can replay any call as a DAG.
- Against the bidirectional risk surface of agents: in MCP and A2A, both what is sent and what is read — the tool result — pass through the full policy chain.
The launch: not a product, a discipline
The AI Gateway we are launching today is proof that we did not violate the "do not fall into gateway sprawl" warning we wrote to ourselves fifteen months ago: the data path genuinely must differentiate — streaming, token economics and agent loops make that mandatory — but you do not need to stand up a second runtime for that differentiation, because we take it on for you. The control plane, on the other hand, does not need to be divided at all.
Agent Gateway, a possibility back in April, became real fifteen months later. We do not know today what the next traffic type will be, or which protocol will show up at the door. But we now have a question to ask every time a new category appears in front of us: what genuinely needs to differentiate here, and what needs to stay unified? That was Apinizer AI Gateway's answer — the data path separated from REST, the control plane stayed whole. The product we are launching today is not a closing statement; it is the first concrete proof of taking that question seriously.
Apinizer AI Gateway — product page and technical details: apinizer.com/products/ai-gateway
For those wondering which release brought what is described here, and what else changed across the platform: 2026 release notes
For those who want to read the conceptual framing from the start, the first part of the series: The Way We Manage APIs Is How We Should Manage LLMs