LLM Providers and Connections
Overview
An LLM Provider Connection is a secure, encrypted link to an external LLM service (OpenAI, Anthropic, Azure, a self-hosted engine, etc.). Each connection holds the provider type, API credentials, and deployment metadata, and becomes available to any AI Gateway for routing once saved.
Provider Categories
OpenAI, Anthropic, Azure OpenAI, Google Vertex AI, Google AI Studio, AWS Bedrock, Cohere, Mistral, DeepSeek, Groq, Moonshot, Zhipu, Qwen, xAI, Together AI, Fireworks AI, Cerebras, Hugging Face, OpenRouter, Perplexity, NVIDIA NIM, Voyage AI — hosted and scaled by the provider.
vLLM, Ollama — models running on infrastructure you operate.
Any OpenAI-compatible API endpoint, for providers not in the built-in list.
Code is the value the Provider Type Catalog definition's code field carries — you'll see it on the Definitions tab and in APIops. The APNZ-6550 catalog refresh added nine of the cloud providers below (Google AI Studio, plus the xAI through NVIDIA NIM rows) — all speak an OpenAI-compatible wire format, so no new adapter family was needed:
| Provider | Code | Deployment (default) | Notes |
|---|---|---|---|
| OpenAI | openai | Cloud | GPT-4o, GPT-4 Turbo, GPT-3.5, and other OpenAI models |
| Anthropic | anthropic | Cloud | Claude models |
| Azure OpenAI | azure-openai | Cloud | OpenAI models hosted on Azure |
| Google Vertex AI | vertex | Cloud | Gemini and PaLM models |
| Google AI Studio | google-ai-studio | Cloud | Direct Gemini API surface — separate from Vertex, no GCP project or OAuth2 required |
| AWS Bedrock | bedrock | Cloud | Claude, Llama, Mistral, Titan, and Nova models |
| Cohere | cohere | Cloud | Command models |
| Mistral | mistral | Cloud | Mistral/Mixtral models |
| DeepSeek | deepseek | Cloud | DeepSeek-V3/R1 |
| Groq | groq | Cloud | Fast inference on Groq's LPU hardware |
| Moonshot (Kimi) | moonshot | Cloud | Kimi models |
| Zhipu (GLM) | zhipu | Cloud | GLM models |
| Qwen (DashScope) | qwen-dashscope | Cloud | Alibaba Qwen models |
| xAI | xai | Cloud | Grok models |
| Together AI | together | Cloud | Open-weight model aggregator |
| Fireworks AI | fireworks | Cloud | Open-weight model inference |
| Cerebras | cerebras | Cloud | Cerebras Inference |
| Hugging Face | huggingface | Cloud | Inference Providers router |
| OpenRouter | openrouter | Cloud | Multi-provider aggregator — model IDs are vendor/model-shaped |
| Perplexity | perplexity | Cloud | Perplexity Router API |
| NVIDIA NIM | nvidia-nim | Cloud | NVIDIA-hosted surface |
| Voyage AI | voyage | Cloud | Embedding-only provider |
| vLLM | vllm | On-Premise | Self-hosted inference engine |
| Ollama | ollama | On-Premise | Local LLM inference |
| Custom OpenAI-Compatible | custom-openai-compat | On-Premise (default) | Any endpoint that speaks the OpenAI API format |
Deployment Type
Each connection declares whether the LLM runs in the cloud or on-premise:
- Cloud — the LLM is managed by the provider (any "Cloud" row in the table above)
- On-Premise — the LLM runs in your infrastructure (vLLM, Ollama, a custom OpenAI-compatible endpoint)
Deployment type is auto-filled based on provider type when you create a connection, but you can override it if needed (for example, a self-hosted vLLM instance running on a cloud VM).
Credentials
All API keys and secrets are encrypted at rest and never logged. Connections are managed through Apinizer's credential system, the same system used to secure API Proxy access.
Request and Response Translation
Clients always talk to Apinizer in the OpenAI format — there's no separate Anthropic Messages or Gemini inbound format to integrate against. Apinizer translates each outbound request into the target provider's native format and normalizes the response (including streamed chunks) back to the OpenAI canonical format before it reaches your client.
Bedrock's InvokeModelWithResponseStream API uses a binary event-stream framing the worker doesn't parse. A streaming chat request routed to a Bedrock connection is rejected with a configuration error before any call reaches AWS — nothing is attempted and nothing is billed. Set streaming: false for that leg, or route streaming traffic through a different provider. Non-streaming (unary) Bedrock chat requests are unaffected.
Provider Capability Matrix
Not every connection type supports every endpoint, and not every provider gets a full request/response rewrite into its own wire format — some are forwarded in OpenAI's format as-is because the provider already speaks it (or is expected to); a smaller set is actively translated.
| Provider | Chat completions | Chat streaming | Embeddings | Audio / image (STT, TTS, image generation) |
|---|---|---|---|---|
| OpenAI, Azure OpenAI, Custom OpenAI-Compatible, vLLM, Ollama | ✅ | ✅ | ✅ | ✅ |
| Mistral, Zhipu (GLM), Qwen (DashScope), Together AI, Google AI Studio | ✅ | ✅ | ✅ | ❌ |
| DeepSeek, Groq, Moonshot (Kimi), xAI, Fireworks AI, Cerebras, Hugging Face, OpenRouter, Perplexity, NVIDIA NIM | ✅ | ✅ | ❌ (chat-only, no first-party embeddings endpoint) | ❌ |
| Cohere | ✅ | ✅ | ❌ (not yet — needs a dedicated adapter) | ❌ |
| Voyage AI | — (embedding-only provider) | — | ✅ | ❌ |
| Anthropic | ✅ (native translation) | ✅ (native translation) | ❌ (no embeddings endpoint at all — connect a Voyage AI provider instead, see Built-in Templates below) | ❌ |
| Google Vertex AI (Gemini) | ✅ (native translation) | ✅ (native translation, SSE) | ❌ (not yet) | ❌ |
| AWS Bedrock — Anthropic models | ✅ (native translation) | ❌ (see warning above) | ❌ (not yet) | ❌ |
| AWS Bedrock — Titan, Llama, Mistral, Nova | ✅ (forwarded as an OpenAI-shaped body; native translation isn't implemented for these families yet) | ❌ | ❌ (not yet) | ❌ |
A ❌ on Embeddings or Audio/Image means Apinizer rejects the request before contacting the provider — nothing is attempted, and nothing is billed.
Native Translation vs. Pass-Through
Only three connection types get an actual request/response rewrite into their own schema: Anthropic, Google Vertex AI (Gemini), and AWS Bedrock when the routed model is an Anthropic-on-Bedrock model (any model ID containing anthropic., including cross-region inference profiles like us.anthropic.*). Every other provider — including Bedrock's Titan/Llama/Mistral/Nova models and Cohere — receives the client's OpenAI-shaped request forwarded unchanged.
For the three natively-translated providers, a handful of OpenAI request fields have no equivalent and are silently dropped (logged as a WARN on the worker, never surfaced to the client):
| OpenAI field | Anthropic | Google Vertex AI (Gemini) |
|---|---|---|
model | kept in the body | not in the body — carried in the URL path |
| System / developer message | mapped to top-level system | mapped to systemInstruction |
max_tokens / max_completion_tokens | required — falls back to the model catalog's max output, then 4096, if the client omits it | optional — set only when the client provides it |
tools[].function | mapped to tools[].input_schema | mapped to functionDeclarations |
tool_choice: "none" | dropped (tool definitions are dropped too) | mapped to mode: "NONE" (tool definitions are kept) |
temperature | clamped to 1.0 if the client sends higher | forwarded as-is (0–2 range) |
stop | mapped to stop_sequences | mapped to stopSequences (only the first 5 entries) |
user | mapped to metadata.user_id | dropped |
n, presence_penalty, frequency_penalty, logit_bias, logprobs, seed, response_format, parallel_tool_calls | dropped | dropped |
| Non-text message content (image, audio, …) | dropped — text-only today | dropped — text-only today |
Usage accounting also differs by family: Anthropic's client-facing prompt_tokens adds cached tokens on top of fresh input tokens, while Gemini's promptTokenCount already includes them. Apinizer accounts for this difference internally, so cost tracking and quotas stay correct no matter which family served the request — see Reports and Analytics and Token Quotas.
When a proxy fails over between providers in different families — for example, an Anthropic primary with an OpenAI or Gemini backup — each leg's request is always rebuilt from your original client request, never from a previous leg's already-translated native body. See Failover Chains.
Model Discovery
"Model discovery" means two different things here — note which one is meant.
Client-side discovery — GET /v1/models
The connections a proxy uses for chat traffic can also be discovered through GET /v1/models, OpenAI's model-listing contract — instead of hardcoding a model name on the client side, client.models.list() (OpenAI SDK), LangChain, or LiteLLM can discover it automatically. The list only returns models actually served by that proxy's routing, not the entire catalog. See the AI Gateway Overview page's Model Discovery section for details.
Admin-side discovery — the Discover Models button
The Discover Models button in the connection form's Models section fetches the provider's live model list so you can pick which ones to add. You no longer have to copy model names out of the provider's documentation by hand, which also removes the typo-driven "model not found" failures.
How it works:
- Pick an environment from the dropdown at the top of the form — the same selection Test Connection uses. This matters in enterprise installations, where the provider is often reachable only from the Worker network.
- Click Discover Models. The request goes to the same address with the same authentication headers as Test Connection; the only difference is that the response body is read.
- The dialog lists what came back. Models already stored on the connection are marked
Already addedand cannot be selected; the rest show asNew. - Tick the ones you want and click Add Selected. They are appended to the form's table — not saved. Click Save on the connection to persist them.
The dialog also reports missing models: ones stored on the connection that the provider no longer lists. These are usually retired models, and a request will fail at runtime if routing still selects one. Apinizer does not remove them automatically — the provider may have returned a short list because of a transient fault, and a silent delete would break working routing; the call is yours.
Discovery only brings back the model id, display name, context window and max output tokens. Pricing, modality and capability fields are not returned — providers report those in inconsistent units, so importing them automatically would produce wrong cost reporting. Fill those in by hand in the table, or use Clone from Catalog to pull the catalog values.
If the provider does not expose a model-listing endpoint (some self-hosted deployments), discovery returns HTTP 404 with a "this provider does not implement model discovery" result. That is not a connection failure — add the models by hand with Add Model. AWS Bedrock and Google Vertex AI connections don't support discovery yet, same as Test Connection.
Built-in Templates
Five built-in provider templates ship with the installation: OpenAI Provider, Voyage AI Provider, DeepSeek Provider, Moonshot (Kimi) Provider and Zhipu (GLM) Provider. Each comes with the endpoint and a default model filled in, but no API key.
The intended flow is to clone a template and enter your own API key; the template itself is owned by the installation and may be refreshed by an upgrade.
The default model on these templates targets embedding generation (for example text-embedding-3-small on the OpenAI template), because that is the most common starting point for RAG and the semantic cache. A template is not locked to embeddings — after cloning it you can change the default model and use the same provider for chat/completion as well.
Creating a Connection
In the Apinizer UI, go to AI Gateway → LLM Providers. The hub has three tabs — Providers, Model Catalog, and Definitions — make sure you're on the Providers tab.
Click the Add button to open the provider form.
Choose the provider from the dropdown — any row in the Provider Categories table above.
Deployment type auto-fills:
- If the provider is vLLM, Ollama, or Custom OpenAI-Compatible → defaults to On-Premise
- Every other provider (OpenAI, Anthropic, Azure, Vertex, Bedrock, and all the newer cloud providers) → defaults to Cloud
Paste the API key or secret. Apinizer encrypts and never logs credentials.
Examples:
- OpenAI: paste your
sk-...key - Anthropic: paste your
sk-ant-...key - Azure: paste the endpoint URL and key
- vLLM / Ollama: paste the base URL (e.g.,
http://localhost:8000)
If your cloud provider has an on-premise instance (or vice versa), toggle the Deployment Type dropdown to Cloud or On-Premise.
Pick the environment to test against from the dropdown at the top of the form — the Test Connection button stays disabled until you do.
The check runs from the Worker deployed to that environment rather than from the Manager, so it exercises the same network path — firewall rules, egress restrictions, proxies — the connection will actually use once it's live, and it resolves any ${env.X} variables in the endpoint or credentials the same way a real request would.
Apinizer sends a single request to the provider's model-list endpoint using your saved credentials; any 2xx response counts as a pass.
AWS Bedrock and Google Vertex AI connections don't support Test Connection yet — their signing schemes (AWS SigV4, Google OAuth2) aren't wired into the check. Save these connections without testing; this doesn't affect chat or embedding traffic, only the pre-save probe and the periodic status shown in Reports and Analytics.
Click Save. The connection is encrypted and stored. Credentials are never visible in logs or exports.
Endpoint URL only needs the provider's base address — the request path (/chat/completions, /embeddings, /audio/transcriptions, /audio/speech, /images/generations, /responses) is appended automatically based on the request type, so it should not be typed into this field. See Effective Endpoint Paths below for how that combination is built, and how to preview it.
API Version feeds three different things depending on the provider: it's sent as the anthropic-version header on an Anthropic connection, used in the model-list probe behind Test Connection for Azure, and — for Azure OpenAI specifically — also appended to the actual chat/embedding request URL as an api-version query parameter.
Provider Definitions Tab
The same hub — AI Gateway → LLM Providers — has a third tab, Definitions, listing the provider type catalog: the built-in and custom definitions every connection above is created from (default endpoint, default auth scheme, and so on). Built-in definitions (OpenAI, Anthropic, Azure, etc.) are editable: change an endpoint or auth default and save, exactly as you would on a custom definition. Your edits are durable — no upgrade overwrites them. The shipped catalog only ever seeds: a definition is written when it is first created and is never written again, so a display name, default endpoint, default API version, default auth scheme, default auth header name, icon URL or documentation link you changed stays exactly as you left it across every future version. That is what makes it safe to point a built-in definition at your own gateway rather than the vendor's public endpoint.
Deletion behaves differently from editing. A routine version upgrade leaves a deleted definition deleted, but Apinizer periodically ships an LLM catalog-refresh migration — the provider landscape churns, so the shipped catalog is re-published from time to time (most recently APNZ-6550) — and because a refresh seeds whatever is missing, a built-in definition you deleted comes back with its shipped defaults. If you want a shipped provider out of circulation for good, leave its definition in place and remove or disable the connections that use it. Duplicate is still the way to keep a shipped definition and a variant of it side by side: copy it and edit the copy, which is a custom definition. Once a custom definition exists, you can view its details, export it, or delete it from here.
For the underlying concept, see Provider Type Catalog.
Adding a Custom Provider Definition
A custom provider definition can be created directly from this tab's own form — as well as, still, by importing a Provider Definition package through the Export/Import Wizard or through the APIops REST API. Importing one also adds a shortcut on the import completion screen straight to this tab.
Give the definition a Name and a Code — the code is what a connection's provider type resolves against, and it cannot be changed after the definition is saved. Display name, icon URL, and a documentation link are optional.
Optionally fill in a default endpoint, default API version, default auth scheme, and default auth header name — these pre-fill a new connection's form when someone picks this provider, the same way the built-in definitions do; the connection's own form still lets a user override any of them.
Choose the Wire Protocol the provider actually speaks — one of five supported families: OpenAI-compatible, Anthropic, Gemini, Bedrock, or vLLM. This is a constrained choice, not free text: only a wire protocol Apinizer has a real adapter for can be selected, and it is required. So is the Chat Completion Path (for example /chat/completions or /messages, depending on the wire protocol). Turn on Supports Embedding if the provider has an embedding endpoint — doing so makes the Embedding Path field required as well, since a provider that claims embedding support with no path to reach it would resolve to nothing at request time. Supports Responses, and the transcription/speech/image paths further down, are optional and only needed if you actually route that traffic through this provider.
Click Save. The definition becomes available in the provider-type dropdown the next time someone creates an LLM Provider connection.
Unlike a built-in provider, a custom code doesn't resolve against Apinizer's own list of known provider types — so there's no fallback layer to fill in a missing path behind the scenes for it. Leaving the wire protocol or chat path blank on a genuinely custom provider would resolve to nothing at runtime; the form requires them up front instead of failing silently later.
Effective Endpoint Paths
When you attach a connection to a provider definition, the definition's wire protocol and endpoint paths (chat completion, embedding, Responses, transcription, speech, image generation) are copied onto the connection. The Advanced section of the connection form shows those values read-only — they tell you the path a request will actually take.
Why read-only: the catalog owns them. To change a path, use the Provider Definition screen; the change is copied to connections attached to that definition from then on — connections already attached keep their own copy, so editing a definition never silently changes the behaviour of a running connection.
A path that isn't listed means the catalog says nothing about that modality, and the provider type's default applies. On older connections created before this shipped the section doesn't appear at all, and behaviour is unchanged.
Using variables in paths
Endpoint paths support the same variable resolution as the endpoint address itself:
${env.variableName} for an environment variable, #{variableName} for a context variable.
Resolution happens at request time, not at save time — so you can write
/api/${env.apiVersion}/chat once in the catalog and have the same definition resolve to v1 in
development and v2 in production.
The effective address is built as endpoint + path, and both halves go through resolution. If a
variable is not defined the text is left verbatim (the request does not fail, it simply does not go
where you expected) — identical to how the endpoint address behaves.
Query Strings and API Versioning in the Final URL
Two more rules apply once endpoint + path is turned into the address a request is actually sent to:
- A query string already embedded in the endpoint moves to the end. If the endpoint value itself carries a query string, it no longer lands ahead of the path in the final address — the path is inserted first and the original query string is appended after it, so the address stays well-formed instead of being split in two.
- Azure OpenAI connections get
api-versionappended automatically. When API Version is filled in, it's added to the request URL as anapi-versionquery parameter — unless the URL already contains one, in which case it isn't added a second time. No other provider type gets API Version appended to the URL this way; on an Anthropic connection, for example, the same field is sent as theanthropic-versionheader instead and never touches the URL.
AWS Bedrock and Google Vertex AI connections don't go through this endpoint + path combination at all — their request address is computed entirely by provider-specific code (AWS SigV4 signing for Bedrock; project, location, and an OAuth2 token for Vertex), so neither rule above, nor the paths described earlier in this section, apply to them.
Merged Address Preview
The connection form's Effective Endpoint Paths section also shows a Combined Address Preview: the endpoint combined with the resolved chat completion path, and — for provider types that support it — the resolved embedding path, applying the same rules described above. This is the actual URL a request is sent to, shown read-only so you can sanity-check a connection before saving it.
On an AWS Bedrock or Google Vertex AI connection, the preview is replaced with a note that the address is computed entirely by provider-specific code, for the same reason those two providers are excluded above. ${env.variableName} references anywhere in the endpoint or API Version are shown raw in the preview rather than resolved — resolution only happens at request time.
Using Connections in Routing
Once created, a provider connection is available for API Proxy routing. When you create an AI Gateway, you map each model to the connection that should serve it:
# Example: route requests for "gpt-4o" to the OpenAI connection
Proxy: "my-chat-api"
Models:
- name: "gpt-4o"
provider: "OpenAI" # reference the saved connection
- name: "gpt-4-turbo"
provider: "OpenAI"
- name: "claude-3-sonnet"
provider: "Anthropic" # different connection
For the bundled list of models and their default pricing, see Model Catalog and Pricing. To chain multiple connections together for failover, cost-aware, or latency-aware routing, see Routing and Failover.
LLM provider connections can also be managed through the APIops REST API; see API Reference: LLM Providers.
Monitoring Connections
- Usage by Deployment — see how many tokens were consumed by cloud vs. on-premise connections on the Analytics page's Usage Reports tab, under By Deployment Type
- Cost Attribution — each connection logs its deployment type in traffic records for accurate cost allocation
- Failover & Load Balancing — multiple connections to the same provider can be chained together for redundancy (see Routing and Failover)
Deleting a Connection
An LLM provider connection that is in use cannot be deleted. The delete is rejected while anything still points at it, and the error message names where those references live.
The reason is that a provider is a reference-type asset: it is never copied into the proxy or policy document, it is resolved by id at runtime. Without the guard, deleting one would leave dangling references behind and the live gateway would fail on the very next request.
The check covers four containers at once — API Proxy, Proxy Group, Policy Group and standalone (global) policies — across these reference paths:
- An API Proxy's AI routing: primary provider, failover chain, conditional routes, primary pool and semantic embedding provider
- The embedding provider of Semantic Cache and RAG Injection policies
- The external guardrail provider of Prompt Guard and Topic Guard policies
To delete the connection, remove those references first or point them at another provider. The same rule applies to deletes made through APIops: DELETE /apiops/projects/{projectName}/llm-providers/{providerName}/ returns 400 Bad Request for a connection that is still in use.
The Used by API Proxies panel on the connection's detail screen shows which proxies are bound to it before you delete.
Troubleshooting
Connection Test Failed
- Select an environment first — Test Connection is disabled until you do, and the check always runs through that environment's Worker, not the Manager
- Verify the API key or endpoint URL is correct
- Check that credentials have the required permissions
- Ensure the provider service is reachable from the selected environment's network — not just from the Manager's
- AWS Bedrock and Google Vertex AI connections don't support Test Connection yet — save without testing
Credentials Not Saving
- Ensure you have admin permissions
- Clear browser cache and try again
Wrong Deployment Type Selected
- Edit the connection and toggle Deployment Type to the correct value
- Deployment type changes are retroactive (logs will show the updated type going forward)