LLM Providers and Models

Overview

An LLM Provider Connection is a secure, encrypted link to an external LLM service (OpenAI, Anthropic, Azure, a self-hosted engine, etc.). Each connection holds the provider type, API credentials, deployment metadata, and the models the provider serves. It becomes available to any AI Gateway for routing once saved.

A model belongs to its provider. There is no separate model page: you add, edit and remove a model inside the provider, in its Models section (see Models).

Project scope

The AI Gateway → Models list shows providers only. It follows the same scoping rule as the Connections screens:

• Project menu (AI Gateway → Models): the list shows only the providers created in the active project.
• Admin menu (AI Governance → Models): the list shows everything — admin providers and every project's providers.

Scoping applies to the list only. Selection dropdowns — for example, picking a provider in an LLM Gateway policy — still offer the project's own providers plus the admin providers, exactly like connection selection elsewhere in Apinizer.

The search box on the list also finds a provider by one of its model names.

Provider Categories

Cloud Providers

OpenAI, Anthropic, Azure OpenAI, Google Vertex AI, Google AI Studio, AWS Bedrock, Cohere, Mistral, DeepSeek, Groq, Moonshot, Zhipu, Qwen, xAI, Together AI, Fireworks AI, Cerebras, Hugging Face, OpenRouter, Perplexity, NVIDIA NIM, Voyage AI, TypeSafe Jev — hosted and scaled by the provider.

Self-Hosted

vLLM, Ollama, TEI Classifier — models running on infrastructure you operate.

Custom

Any endpoint Apinizer doesn't have a built-in profile for — pick the adapter family it speaks and define your own paths.

Code is the internal identifier of the provider type. The cloud providers from Google AI Studio and from xAI through NVIDIA NIM all speak an OpenAI-compatible wire format, so they need no separate adapter family:

Provider	Code	Deployment (default)	Notes
OpenAI	openai	Cloud	GPT-4o, GPT-4 Turbo, GPT-3.5, and other OpenAI models
Anthropic	anthropic	Cloud	Claude models
Azure OpenAI	azure-openai	Cloud	OpenAI models hosted on Azure
Google Vertex AI	vertex	Cloud	Gemini and PaLM models
Google AI Studio	google-ai-studio	Cloud	Direct Gemini API surface — separate from Vertex, no GCP project or OAuth2 required
AWS Bedrock	bedrock	Cloud	Claude, Llama, Mistral, Titan, and Nova models
Cohere	cohere	Cloud	Command models
Mistral	mistral	Cloud	Mistral/Mixtral models
DeepSeek	deepseek	Cloud	DeepSeek-V3/R1
Groq	groq	Cloud	Fast inference on Groq's LPU hardware
Moonshot (Kimi)	moonshot	Cloud	Kimi models
Zhipu (GLM)	zhipu	Cloud	GLM models
Qwen (DashScope)	qwen-dashscope	Cloud	Alibaba Qwen models
xAI	xai	Cloud	Grok models
Together AI	together	Cloud	Open-weight model aggregator
Fireworks AI	fireworks	Cloud	Open-weight model inference
Cerebras	cerebras	Cloud	Cerebras Inference
Hugging Face	huggingface	Cloud	Inference Providers router
OpenRouter	openrouter	Cloud	Multi-provider aggregator — model IDs are vendor/model-shaped
Perplexity	perplexity	Cloud	Perplexity Router API
NVIDIA NIM	nvidia-nim	Cloud	NVIDIA-hosted surface
Voyage AI	voyage	Cloud	Embedding-only provider
TypeSafe Jev	typesafe	Cloud	Decision model. Classifier and judge slots only — see Decision-Only Providers
vLLM	vllm	On-Premise	Self-hosted inference engine
Ollama	ollama	On-Premise	Local LLM inference
Custom OpenAI-Compatible	custom-openai-compat	On-Premise (default)	Any endpoint that speaks the OpenAI API format
TEI Classifier (Self-Hosted)	tei	On-Premise	Guard classifier. Judge slot only — see Decision-Only Providers

Decision-Only Providers

A decision-only provider makes a decision for Apinizer. It does not generate text. You can use it in two places:

• As the Classifier Provider of AI routing. See Classifier Routing.
• As the LLM judge of a guard policy. See External Provider Guardrails.

You cannot use a decision-only provider as a routing target, a pool member, a failover target, or an embedding provider. These lists do not show it. Apinizer rejects a save that refers to one.

Provider type	Use	Default endpoint	Authentication	Model
TypeSafe Jev (badge Decision model)	Route classifier and judge	https://api.typesafe.ai/v1	Bearer key in the Authorization header	jev-latest
TEI Classifier (Self-Hosted) (badge Guard classifier)	Judge only	http://localhost:8080	None	meta-llama/Prompt-Guard-86M or protectai/deberta-v3-base-prompt-injection-v2 (maximum input 512 tokens)

In the Choose a provider dialog, the TypeSafe Jev tile is under Managed cloud. The TEI Classifier tile is under Self-hosted. The provider form shows an information note for both types.

TypeSafe Jev

Jev is a structured decision model. Apinizer calls POST {endpoint}/systemone and gets a typed answer with a probability. Test Connection lists the models with GET /v1/models. The chat inference test is not offered for decision-only providers.

TEI Classifier

A TEI provider is a Hugging Face Text Embeddings Inference server that serves a small sequence classifier. The classifier scores a fixed set of labels. Apinizer calls POST {endpoint}/predict. Test Connection and Discover Models read GET /info. The result shows the model id, the maximum input length, and the labels of the model. The judge uses these labels as Unsafe labels.

Prompt Guard 86M is a gated model

Hugging Face gates meta-llama/Prompt-Guard-86M. The TEI server needs a Hugging Face token and the accepted Llama licence to download it. Without them, use the open model protectai/deberta-v3-base-prompt-injection-v2 (Apache-2.0 licence, labels SAFE and INJECTION, maximum input 512 tokens). Set the Unsafe labels to INJECTION. Add the model id to the provider's models (see Models for Classifiers and Judges).

Use credentials held by the AI gateway

A decision call uses the credentials stored on the provider. Set the credentials to Provided by AI gateway. A provider set to Provided by client has no client key for the decision call, and the provider answers with 401 or 403.

Deployment Type

Each connection carries a deployment type label:

• Cloud — the LLM is managed by the provider (any "Cloud" row in the table above)
• On-Premise — the LLM runs in your infrastructure (vLLM, Ollama, TEI, a custom OpenAI-compatible endpoint)
• Other — a Custom provider. Apinizer cannot know where a custom provider runs, so it does not guess

Deployment type is filled in automatically from the provider type when you create a connection. The create and edit form has no Deployment Type field. The provider detail screen shows the value.

The deployment type is a reporting label only. It drives the Cloud / On-Premise / Other breakdown in the AI usage reports and never changes how requests are routed. Custom providers saved before this change keep their stored value.

Credentials

All API keys and secrets are encrypted at rest and never logged.

Who Holds the Credential: Gateway or Client

Every connection answers one question up front — who supplies the upstream provider's credential on a request:

Credential Source	Behavior
Provided by AI gateway (default)	Apinizer stores the key and signs every request with it — the behavior every connection had before this setting existed.
Provided by client	Apinizer stores no credential at all. Every request must already carry the provider's own header; the gateway forwards that header untouched instead of injecting a stored value. Budget, cost, and rate limits still attach to the Apinizer consumer that made the call — only the upstream key belongs to the client.

Provided by client is for installations that don't want the provider's raw key stored inside Apinizer at all — the caller (or a proxy in front of it) supplies its own key on every request. Because nothing is stored, Test Connection asks for a temporary credential just for that one probe, and that value is never saved.

Where the Key Travels: Header or Query Parameter

For the API-key family of authentication (Bearer token, an API key in a named header, or a custom header — see Credential Type Label below), a second setting decides where the key is placed on the outbound request:

API Key Location	Shape	Notes
Header (default)	<header name>: <prefix><key>	Every mainstream provider.
Query parameter	?<parameter name>=<key>	Appended to the URL — Gemini's native endpoint accepts ?key=. Discouraged when a header alternative exists, since a key placed in the URL can end up in access logs.

Basic auth, AWS Signature V4, and the Google service-account (OAuth2) flow each have a fixed transport of their own and are unaffected by this setting.

Key Prefix

The text placed in front of the key value on the wire. Left blank, it defaults to the credential type's own prefix (Bearer for a Bearer token, nothing for a raw API key). Set it explicitly to an empty value for a provider that wants the raw key with no prefix (Anthropic's x-api-key style), or to something like Token for a provider that expects Authorization: Token <key>.

Allowing a Client to Override the Stored Key

On a gateway-held connection, Allow clients to override with their own key lets an individual request bypass the stored credential: if the incoming request already carries the connection's own auth header, that value is forwarded instead of the stored key — the same pattern Kong calls allow_override. Left off (the default), any such header on the incoming request is dropped before the call is signed, so the stored key can never leak by a client simply guessing the header name.

Credential Type Label

The Credential Type shown in the form (Bearer token, API key in a header, API key as a query parameter, Basic — username + password, AWS Signature V4, Google service account, or No authentication) is not a separate stored field — it's a label derived from the underlying Auth Scheme, plus (for the API-key shapes) the API Key Location setting above:

Credential Type shown	Underlying Auth Scheme	API Key Location
Bearer token	BEARER	—
API key in a header	API_KEY_HEADER or CUSTOM	Header
API key as a query parameter	API_KEY_HEADER or CUSTOM	Query
Basic — username + password	BASIC	—
AWS Signature V4	AWS_SIGV4	—
Google service account (JSON)	OAUTH2	—
No authentication	NONE	—

API_KEY_HEADER and CUSTOM share the same "API key" credential type on screen — both send one key as a header or query parameter, and only differ in whether the header name and prefix come from the provider type's defaults or from values you typed yourself.

Request and Response Translation

Chat clients talk to Apinizer in the OpenAI format. Apinizer translates each outbound request into the target provider's native format and normalizes the response (including streamed chunks) back to the OpenAI canonical format before it reaches your client.

AWS Bedrock chat streaming isn't supported yet

Bedrock's InvokeModelWithResponseStream API uses a binary event-stream framing the worker doesn't parse. A streaming chat request routed to a Bedrock connection is rejected with a configuration error before any call reaches AWS — nothing is attempted and nothing is billed. Set streaming: false for that leg, or route streaming traffic through a different provider. Non-streaming (unary) Bedrock chat requests are unaffected.

Provider Capability Matrix

Not every connection type supports every endpoint, and not every provider gets a full request/response rewrite into its own wire format — some are forwarded in OpenAI's format as-is because the provider already speaks it (or is expected to); a smaller set is actively translated.

Provider	Chat completions	Chat streaming	Embeddings	Audio / image (STT, TTS, image generation)
OpenAI, Azure OpenAI, Custom OpenAI-Compatible, vLLM, Ollama	✅	✅	✅	✅
Mistral, Zhipu (GLM), Qwen (DashScope), Together AI, Google AI Studio	✅	✅	✅	❌
DeepSeek, Groq, Moonshot (Kimi), xAI, Fireworks AI, Cerebras, Hugging Face, OpenRouter, Perplexity, NVIDIA NIM	✅	✅	❌ (chat-only, no first-party embeddings endpoint)	❌
Cohere	✅	✅	❌ (not yet — needs a dedicated adapter)	❌
Voyage AI	— (embedding-only provider)	—	✅	❌
Anthropic	✅ (native translation)	✅ (native translation)	❌ (no embeddings endpoint at all — connect a Voyage AI provider instead, see Built-in Templates below)	❌
Google Vertex AI (Gemini)	✅ (native translation)	✅ (native translation, SSE)	❌ (not yet)	❌
AWS Bedrock — Anthropic models	✅ (native translation)	❌ (see warning above)	❌ (not yet)	❌
AWS Bedrock — Titan, Llama, Mistral, Nova	✅ (forwarded as an OpenAI-shaped body; native translation isn't implemented for these families yet)	❌	❌ (not yet)	❌

TypeSafe Jev and TEI Classifier are not in this table. They make decisions only and do not serve chat, embedding, audio, or image traffic. See Decision-Only Providers.

A ❌ on Embeddings or Audio/Image means Apinizer rejects the request before contacting the provider — nothing is attempted, and nothing is billed.

Native Translation vs. Pass-Through

Only three connection types get an actual request/response rewrite into their own schema: Anthropic, Google Vertex AI (Gemini), and AWS Bedrock when the routed model is an Anthropic-on-Bedrock model (any model ID containing anthropic., including cross-region inference profiles like us.anthropic.*). Every other provider — including Bedrock's Titan/Llama/Mistral/Nova models and Cohere — receives the client's OpenAI-shaped request forwarded unchanged.

For the three natively-translated providers, a handful of OpenAI request fields have no equivalent and are silently dropped (logged as a WARN on the worker, never surfaced to the client):

OpenAI field	Anthropic	Google Vertex AI (Gemini)
model	kept in the body	not in the body — carried in the URL path
System / developer message	mapped to top-level system	mapped to systemInstruction
max_tokens / max_completion_tokens	required — falls back to the model's max output tokens, then 4096, if the client omits it	optional — set only when the client provides it
tools[].function	mapped to tools[].input_schema	mapped to functionDeclarations
tool_choice: "none"	dropped (tool definitions are dropped too)	mapped to mode: "NONE" (tool definitions are kept)
temperature	clamped to 1.0 if the client sends higher	forwarded as-is (0–2 range)
stop	mapped to stop_sequences	mapped to stopSequences (only the first 5 entries)
user	mapped to metadata.user_id	dropped
n, presence_penalty, frequency_penalty, logit_bias, logprobs, seed, response_format, parallel_tool_calls	dropped	dropped
Non-text message content (image, audio, …)	dropped — text-only today	dropped — text-only today

Usage accounting also differs by family: Anthropic's client-facing prompt_tokens adds cached tokens on top of fresh input tokens, while Gemini's promptTokenCount already includes them. Apinizer accounts for this difference internally, so cost tracking and quotas stay correct no matter which family served the request — see Reports and Analytics and Token Rate Limit Policy.

When a proxy fails over between providers in different families — for example, an Anthropic primary with an OpenAI or Gemini backup — each leg's request is always rebuilt from your original client request, never from a previous leg's already-translated native body. See Failover Chains.

Models

A provider owns its models. Open the provider and use the Models section to switch a model on or off, set its price, and limit who may call it. There is no separate model page.

What a Model Holds

Identity and Capacity

Model ID, display name, context window, and maximum output tokens. A request for "gpt-4o" or "claude-3-sonnet" resolves to the right provider by the model ID.

Unit Pricing

Optional. Input, output, and cached-token price per 1M tokens, plus price per image, per audio minute, and per TTS character. Used to calculate the cost of every request that uses the model.

Modality and Capabilities

The model type (for example Text (Chat), Vision, Audio, Embedding, Rerank, Multimodal, Image Generation) and the capabilities of the model, so requests are not routed to an incompatible model.

Status and Access Control

The model's status (Active or Inactive) and an Access Control card that lists who may call the model.

Managing the Model List

On the provider page, the Models card shows each model with its type, context window, input and output price, access control status and status. Configure on the card opens a drawer where you change the list; you add and edit a model in a second drawer instead of inline table cells:

• Add opens the Add models drawer. It stays disabled until the Endpoint URL is filled in. The "⋯" menu at the end of a row holds Edit, which opens the Edit Model drawer pre-filled with that row, Activate or Deactivate (see Model Status), and Delete for the model.
• The model drawer holds the model identity (Model ID, Display Name), the Model Type, capacity (Context Window, Max Output Tokens), the price fields under Cost management, and the Access Control card. A model without a price shows a no price badge in Cost management.
• Routing enforces the Context Window value. When a request does not fit the context window, the Context window card on the Routing tab of the AI proxy decides what happens. A model with no context window value is not checked. See Context Window.
• Save Model writes the model drawer back into the list. Nothing is persisted until you click Save in the Configure drawer and confirm the dialog that follows (see The Provider Page).

The credentials of the provider are not in a separate panel. The Authentication part of the Connection card holds the credential type, the API key or secret, and the other credential fields (see Credentials). Change them with Configure on that card; Save persists them.

Model Status: Activate and Deactivate

Choose Deactivate in the "⋯" menu of a model row to make the gateway refuse the model for every caller. A request that names a disabled model gets HTTP 403, and the model does not appear in the GET /v1/models response. The model stays on the list, so you can choose Activate again later. The change applies when you save and confirm the provider.

The gateway's own calls follow the switch too. A switched-off model is not called by the route classifier, the LLM judge of a guard policy, or an embedding call. The route classifier makes no call and no classifier route matches. The LLM judge raises an evaluation error, and the fail mode of the policy decides what happens. An embedding call is refused before any request reaches the provider.

Access Control

The Access Control card of a model lists the consumers, applications and organizations that may call the model. When you turn it on, only the listed subjects can call the model.

Whoever may edit the provider may edit the access control of its models. An admin provider is edited by administrators. A project provider is edited by the project.

Only Listed Models

The Only listed models switch is at the top of the Models drawer, opened with Configure on the Models card. It is off by default.

• Off: The gateway sends a model ID that is not on the provider's list to the provider as is. Only the models on the list have a price and an access control. If the LLM Gateway uses Use the client's model, the model ID must also be a plain name; see Models That Are Not on the Provider's List.
• On: The gateway refuses a model ID that is not on the list. The client gets HTTP 403. A tag variant of a listed model counts as listed (for example llama3:8b for a listed llama3). A request without a model ID is not affected.

The refusal applies on every path:

• Routing, including the primary provider and the failover legs. A failover leg that does not list the model is skipped.
• The GET /v1/models list. It shows only the listed models.
• The cheaper-model downgrade of the token rate limit. The downgrade does not select an unlisted model.
• The route classifier, the LLM judge, and the embedding calls of the gateway.

A provider with this switch on needs at least one model. Apinizer rejects the save when the list is empty.

The client error is the existing POLICY_ROUTING_AI_MODEL_ACCESS_DENIED message. You can change its text in the error messages. The gateway log says that the model is not listed by its LLM provider, which allows only listed models.

Add the decision models to the list

When the switch is on, the models that the route classifier, the LLM judge and the embedding settings use on this provider must be on the list too. If one is missing, the gateway refuses those calls. See Models for Classifiers and Judges.

When you save a provider while Only listed models is on, Apinizer checks where the provider is asked for a model that is not on its list. It checks:

• LLM Gateway routing: the primary model, the model pool, failover, conditional routes, the classifier, and the semantic cache embedding.
• The embedding models of policies.
• LLM judge guardrails.
• The cheaper-model downgrade of the token rate limit.
• Knowledge bases.

The check covers API proxies, proxy groups, policy groups, and global policies. Apinizer does not block the save. The screen shows a warning that lists each place with its unlisted model IDs, for example API Proxy: chat (gpt-4o-mini). The gateway refuses these calls until you add the models to the list or change the settings.

Apinizer does not check a model ID that is written as a variable (${...}). It also does not check the Limit Plan budgets and the token budgets that you define on proxy groups or global policies, because they name no provider. The warning appears only in the Manager screen. An APIops save does not return it.

Adding Models

Add models opens a drawer with two tabs:

• Discover from provider — fetches the provider's own model list so you can pick the models to add. See Admin-side discovery below.
• Add manually — type the model ID by hand. Use it for a model the provider does not list, such as a fine-tuned model or a private deployment.

A model ID can be on the list only once for each provider. The drawer shows an error when the ID is already there.

Prices and Cost Tracking

Prices are optional. Set them under Cost management in the model drawer. Prices are stored in USD. For an introduction to how tokens are counted and priced, see AI Fundamentals. You can also manage pricing from AI Cost Settings.

Every request that reaches a provider is metered — input tokens, output tokens, and (when the provider supports it) cached tokens — and priced against the model that served it. That cost flows into:

• Reports and Analytics — cost breakdowns by person, team, model, and deployment type
• Token Rate Limit Policy and Limit Plans — USD limits are evaluated against the same price
• Routing and Failover — cost-aware routing and per-leg cost caps use the prices to decide where a request should go

A model without a price costs 0

A model without a price counts as 0 in cost reports. A USD limit does not meter such a model, so the limit never cuts it off. Set an input price for every model that a USD limit must cover. Models with no token concept (image generation, audio) have no input price and cost 0 unless you set their per-image, per-audio-minute or per-character price.

If you change a price, new requests use the new value. Historical logs keep their original cost.

Deleting a Model

Apinizer refuses to delete a model from a provider only while the model is in use. The error names where the model is used:

• The routing of LLM Gateways: primary, failover, conditional routes, and the classifier
• The embedding settings of semantic cache and RAG
• Global AI policies
• Policy groups
• Proxy groups
• Knowledge bases

Remove those references or point them at another model, then delete the model. A provider can have no models at all.

Changing a Model ID

You can change the Model ID of a model that API proxies, policies or limits already use. The names that point to it move with it.

1. Open the provider, click Configure on the Models card, and choose Edit in the "⋯" menu of the model row.
2. Change the Model ID and click Save Model.
3. Click Save in the Configure drawer.

If the model is in use, a warning lists what the change moves. Confirm the warning to continue. When you confirm, Apinizer does these steps in order:

1. It adds the new model to the provider and deploys the provider. The gateways accept both ids from this point.
2. It moves the references to the new id: AI routing of API proxies, policies on API proxies, policy groups, global policies, proxy groups, knowledge bases and the rules of limit plans.
3. It deploys again what was running. A proxy or group that has undeployed changes is not deployed.
4. It removes the old model when nothing is left that can call it.

The warning also lists:

• Places that have undeployed changes.
• Conditions that compare the model text with the old id.
• Knowledge bases.
• Limit plans: the running limit window may start again.
• Limit assignments that target the model. They do not move. End them and create ones for the new model.
• Places that take the model id from a variable (${...}, #{...}) that may name this model in an environment. They do not change, and the old model stays while a value needs it. Check the value of the variable in each environment.

When the Old Model Is Kept

Apinizer keeps the old model and names the cause in a warning in these cases:

• An API proxy or proxy group has undeployed changes. It runs the old id until its owner deploys it. After the deploy, repeat the change.
• A limit assignment targets the model. The target of an assignment cannot change. End the assignment and create one for the new model.
• An entry of another provider has no model of its own. The entry takes its id from a target that would move. Change the entry by hand.
• A value of a variable needs the old model. A place on the provider takes its model id from a variable. Apinizer resolves the variable in each environment and skips an environment that has no value for it. The old model stays in two cases: in at least one environment the value reaches the old model and would reach no model once the old one is removed, or no environment knows a value. A request value #{...} counts as unknown. Apinizer names the place. Check the value of the variable in each environment, then delete the old model from the list.
• A variable that does not need the old model does not stop the change. A variable that names another model does not stop it. A variable with a base name, such as w, that reaches the new model, such as w:new, does not stop it either. Apinizer does not rewrite references that use a variable. Apinizer does not print the values of variables in the result.
• A deploy failed. The proxy, group or provider runs the old id. Deploy it again.

Repeat the same change after you resolve the cause. A repeat is safe and finishes the rest.

Other Rules

• Two models in one save. Apinizer refuses a save in which one model takes the id of another model that also changes, for example two models that exchange ids. Change them one at a time.
• A new id that already exists. If the new id is already a model of the provider, its access list must be the same as the access list of the old model. When an access list is off, it counts as empty. If the lists differ, Apinizer writes nothing. Make the access lists the same, or use a new model id.
• Environments. An environment with no reachable gateway that does not run the model does not stop the change. It appears as failed in Deployment Results.
• Permissions. You need the permission for every asset that the change touches. This covers editing the asset, deploying the proxies and groups that were running, and managing the limit plans and assignments. If you lack it for one asset, Apinizer writes nothing and names the assets.
• Conditions on the client's model text. Apinizer does not change them. A client that keeps sending the old id still matches its conditions and reaches the new id. Update the conditions when you want to.
• Knowledge bases. The change updates the model id that the knowledge base names. The stored vectors are not computed again. If the embedding model itself changes, index the knowledge base again.

You can run the same change with the Management API. See Change a Model ID of an LLM Provider.

Models for Classifiers and Judges

The AI routing classifier and the LLM judge of guard policies use small decision models. Add these model IDs to the provider that serves them:

Provider	Model	Use
TypeSafe Jev	jev-latest	Decision model. Route classifier and judge.
TEI Classifier (Self-Hosted)	meta-llama/Prompt-Guard-86M	Guard classifier. Judge only. Maximum input 512 tokens. Gated model: see the note below.
TEI Classifier (Self-Hosted)	protectai/deberta-v3-base-prompt-injection-v2 (DeBERTa v3 Prompt Injection v2 (TEI))	Guard classifier. Judge only. Labels SAFE and INJECTION. Maximum input 512 tokens. Apache-2.0 licence.
Ollama	qwen2.5:1.5b-instruct, llama3.2:3b, granite3.3:2b, llama-guard3:1b	Small models for Standard model behaviour.
Ollama	qwen3:8b	Thinking model. Use Reasoning model behaviour, or keep Standard when thinking is turned off.

TypeSafe Jev and TEI are decision-only providers. See Decision-Only Providers, Classifier Routing, and External Provider Guardrails.

Prompt Guard 86M is a gated model

Hugging Face gates meta-llama/Prompt-Guard-86M. The TEI server needs a Hugging Face token and the accepted Llama licence to download it. Without them, use protectai/deberta-v3-base-prompt-injection-v2. It is open. As an LLM judge, set the unsafe label to INJECTION.

Model Discovery

"Model discovery" means two different things here — note which one is meant.

Client-side discovery — GET /v1/models

The connections a proxy uses for chat traffic can also be discovered through GET /v1/models, OpenAI's model-listing contract — instead of hardcoding a model name on the client side, client.models.list() (OpenAI SDK), LangChain, or LiteLLM can discover it automatically. The list only returns enabled models that the proxy's routing actually serves, not every model of the provider. See the AI Gateway Overview page's Model Discovery section for details.

Admin-side discovery — the Discover tab

The Discover from provider tab of the Add models drawer fetches the provider's live model list so you can pick which ones to add. You no longer have to copy model names out of the provider's documentation by hand, which also removes the typo-driven "model not found" failures.

Discovery needs the permission to manage AI development, because it runs inside the Models drawer. When discovery fails, the drawer also shows why: the provider does not implement a discovery endpoint (HTTP 404), the provider rejected the request (HTTP status), the environment was not found, or the worker in the selected environment did not respond.

How it works:

1. Pick an environment from the dropdown at the top of the page — the same selection Test Connection uses. This matters in enterprise installations, where the provider is often reachable only from the Worker network.
2. Open the Discover from provider tab. The worker of the selected environment sends the request to the same address with the same authentication headers as Test Connection; the only difference is that the response body is read.
3. The tab lists what came back. Models already stored on the provider are marked Already added and cannot be selected. The rest show as New. A search box filters the list.
4. Tick the ones you want and click Add Selected. They are appended to the model list — not saved. Click Save in the Configure drawer and confirm to persist them.

The panel also reports missing models: ones stored on the provider that the provider no longer lists. These are usually retired models, and a request will fail at runtime if routing still selects one. Apinizer does not remove them automatically — the provider may have returned a short list because of a transient fault, and a silent delete would break working routing. The call is yours.

Note

Discovery only brings back the model id, display name, context window and max output tokens. Pricing, modality and capability fields are not returned — providers report those in inconsistent units, so importing them automatically would produce wrong cost reporting. Set them by hand in the model drawer (pencil icon on the row). The Model Type may be left blank.

Why model type matters

A model with no type recorded appears under no value of the Model type filter on the AI Gateway → LLM Gateway list, and for models with no token concept (image generation, audio) the cost is computed as 0 USD. Because of this, the gateway logs a warning once per model whenever it prices a request for a model whose price is unknown. You can find these models with the Unspecified filter on the LLM Gateway list, which is evaluated per provider.

Warning

If the provider does not expose a model-listing endpoint (some self-hosted deployments), discovery returns HTTP 404 with a "this provider does not implement model discovery" result. That is not a connection failure — add the models with the Add manually tab. AWS Bedrock and Google Vertex AI connections don't support discovery yet, same as Test Connection.

Built-in Templates

Five built-in provider templates ship with the installation: OpenAI Provider, Voyage AI Provider, DeepSeek Provider, Moonshot (Kimi) Provider and Zhipu (GLM) Provider. Each comes with the endpoint and a default model filled in, but no API key. Being installation-owned, they are listed under the admin menu — the project-mode list shows only the project's own connections.

The intended flow is to clone a template and enter your own API key; the template itself is owned by the installation and may be refreshed by an upgrade.

Note

The default model on these templates targets embedding generation (for example text-embedding-3-small on the OpenAI template), because that is the most common starting point for RAG and the semantic cache. A template is not locked to embeddings — after cloning it you can change the default model and use the same provider for chat/completion as well.

Creating a Connection

Navigate to Connections

In the Apinizer UI, go to AI Gateway → Models. The list is a table with the columns Provider (name and endpoint), Type, Models (active models out of all models), Health (Healthy / Down / Unknown), Status (Active or Inactive) and, in the admin menu, Project. Narrow it with the search box (it also matches a model name) and the provider type and status filters; the eraser button clears them. Click a row to open the provider page. The "⋯" menu of a row holds View, Duplicate, Export and Delete.

Click Create

Click Create. The Choose a provider dialog opens. The Import button in the top bar brings in a provider you exported earlier.

Select Provider Type

Pick a tile under Managed cloud or Self-hosted — any row in the Provider Categories table above — or choose Custom provider if yours isn't listed. See Choosing a Provider Type below for what each pick fills in.

Deployment type auto-fills:

• If the provider is vLLM, Ollama, TEI, or Custom OpenAI-Compatible → set to On-Premise
• If you pick Custom provider → set to Other
• Every other provider (OpenAI, Anthropic, Azure, Vertex, Bedrock, and all the newer cloud providers) → set to Cloud

Enter Credentials

In the Authentication part of the Connection section, choose who supplies the credential and the credential type (see Credentials). Paste the API key or secret there. Apinizer encrypts and never logs credentials.

Examples:

• OpenAI: paste your sk-... key
• Anthropic: paste your sk-ant-... key
• Azure: paste the endpoint URL and key
• vLLM / Ollama: paste the base URL (e.g., http://localhost:8000)

Test the Connection (optional but recommended)

Click Test Connection at the top of the page. The button is enabled as soon as the Endpoint URL is filled in. The Test connection drawer has two tests: Reachability and Inference.

Pick the environment to test through from the dropdown at the top of the page or in the drawer. With an environment selected, the check runs from the Worker deployed to that environment rather than from the Manager, so it exercises the same network path — firewall rules, egress restrictions, proxies — the connection will actually use once it's live, and it resolves any ${env.X} variables in the endpoint or credentials the same way a real request would. With no environment selected, the Manager itself sends the request, and ${env.X} variables are not resolved.

• Reachability sends a single request to the provider's model-list endpoint using your saved credentials; any 2xx response counts as a pass. It proves the address is reachable, TLS works and the credential is accepted, not that requests are shaped correctly.
• Inference sends a real chat call with a sample prompt to one of the enabled models, with a timeout and a maximum output size you can set. Use it to prove the adapter family, the chat path and the model id together. It can also try the call as a stream. The result shows whether the model was accepted, whether token usage was read and whether a cost was computed.

Note

AWS Bedrock and Google Vertex AI connections don't support Test Connection yet — their signing schemes (AWS SigV4, Google OAuth2) aren't wired into the check. Save these connections without testing; this doesn't affect chat or embedding traffic, only the pre-save probe and the periodic status shown in Reports and Analytics.

Save

Click Save and Deploy. The connection is encrypted and stored, and the change is deployed. Credentials are never visible in logs or exports.

What goes in Endpoint URL and API Version

Endpoint URL only needs the provider's base address — the request path (/chat/completions, /embeddings, /audio/transcriptions, /audio/speech, /images/generations, /responses) is appended automatically based on the request type, so it should not be typed into this field. See Effective Endpoint Paths below for how that combination is built, and how to preview it.

API Version feeds three different things depending on the provider: it's sent as the anthropic-version header on an Anthropic connection, used in the model-list probe behind Test Connection for Azure, and — for Azure OpenAI specifically — also appended to the actual chat/embedding request URL as an api-version query parameter.

Choosing a Provider Type

Creating or changing a connection's provider opens the Choose a provider dialog instead of a plain dropdown:

• A search box filters the list as you type.
• Managed cloud and Self-hosted groups list every built-in provider type — any row in the Provider Categories table above. The provider types are shipped with Apinizer. You cannot create, edit, or delete them, from the UI or through APIops.
• Picking a tile gives the connection that provider type's endpoint and authentication scheme. The wire protocol and the request paths are the provider type's own, unless you set them yourself; an empty Name field is also suggested from the provider's code.
• At the bottom, Not listed? opens Custom provider — see below.

The Change button next to the provider chip on the connection form reopens this dialog. Re-picking overwrites the type-derived fields (endpoint, credential scheme, adapter, deployment type) with the new choice's values, but keeps the connection's name, description, stored credential, and model list untouched.

Custom Provider

Pick Custom provider when your vendor isn't in the built-in list. Apinizer knows nothing about it, so nothing is pre-filled and nothing is hidden. This is the other provider type, and it has an explicit wire protocol and explicit paths:

• Deployment Type is set to Other automatically, because Apinizer cannot know where a custom provider runs. Reports show it as Other (Custom).
• Credential Type — pick the authentication your provider expects (Bearer token, an API key in a header or query parameter, Basic, and so on; see Credential Type Label above).
• Adapter family — which wire format the provider speaks: OpenAI-compatible, Anthropic, Gemini, Bedrock, vLLM, TYPESAFE, or TEI. This field is required: Save and Deploy stays disabled until you pick one. TYPESAFE and TEI create a decision-only provider.
• Chat Completion Path comes pre-filled with /chat/completions and can be edited — it's the one path a custom provider needs to be reachable at all. The other paths (embedding, Responses, transcription, speech, image generation) start empty; fill in only the ones you actually route traffic to.
• The Advanced section shows every field with nothing hidden, since only you know which endpoints your provider serves.

The adapter and paths of a custom provider are ordinary fields stored directly on the connection, so it moves with the connection everywhere a connection moves: export and import it as an LLM Provider package through the Export/Import Wizard, or manage it through the APIops LLM Providers endpoint, where it appears with providerType: "OTHER" plus its own adapter and path fields.

Effective Endpoint Paths

The wire protocol and the endpoint paths (chat completion, embedding, Responses, transcription, speech, image generation) of a connection are the provider type's own, unless you set them. The Advanced section of the connection form shows those values under Edit paths, where they're editable — useful when a provider changed a path, or your deployment needs a different one than the default.

Editing a path here only affects this connection. It never affects any other connection using the same provider type, and an Apinizer upgrade does not overwrite it.

A path that isn't listed means nothing is set for that modality, and the provider type's default applies.

Using variables in paths

Endpoint paths support the same variable resolution as the endpoint address itself: ${env.variableName} for an environment variable, #{variableName} for a context variable. Resolution happens at request time, not at save time — so you can write /api/${env.apiVersion}/chat once and have the same connection resolve to v1 in development and v2 in production.

The effective address is built as endpoint + path, and both halves go through resolution. If a variable is not defined the text is left verbatim (the request does not fail, it simply does not go where you expected) — identical to how the endpoint address behaves.

Query Strings and API Versioning in the Final URL

Two more rules apply once endpoint + path is turned into the address a request is actually sent to:

• A query string already embedded in the endpoint moves to the end. If the endpoint value itself carries a query string, it no longer lands ahead of the path in the final address — the path is inserted first and the original query string is appended after it, so the address stays well-formed instead of being split in two.
• Azure OpenAI connections get api-version appended automatically. When API Version is filled in, it's added to the request URL as an api-version query parameter — unless the URL already contains one, in which case it isn't added a second time. No other provider type gets API Version appended to the URL this way; on an Anthropic connection, for example, the same field is sent as the anthropic-version header instead and never touches the URL.

AWS Bedrock and Google Vertex AI connections don't go through this endpoint + path combination at all — their request address is computed entirely by provider-specific code (AWS SigV4 signing for Bedrock; project, location, and an OAuth2 token for Vertex), so neither rule above, nor the paths described earlier in this section, apply to them.

Merged Address Preview

The connection form's Effective Endpoint Paths section also shows a Combined Address Preview: the endpoint combined with the resolved chat completion path, and — for provider types that support it — the resolved embedding path, applying the same rules described above. This is the actual URL a request is sent to, shown read-only so you can sanity-check a connection before saving it.

On an AWS Bedrock or Google Vertex AI connection, the preview is replaced with a note that the address is computed entirely by provider-specific code, for the same reason those two providers are excluded above. ${env.variableName} references anywhere in the endpoint or API Version are shown raw in the preview rather than resolved — resolution only happens at request time.

Using Connections in Routing

Once created, a provider connection is available for API Proxy routing. When you create an LLM Gateway, you map each model to the connection that should serve it:

# Example: route requests for "gpt-4o" to the OpenAI connection
Proxy: "my-chat-api"
Models:
  - name: "gpt-4o"
    provider: "OpenAI" # reference the saved connection
  - name: "gpt-4-turbo"
    provider: "OpenAI"
  - name: "claude-3-sonnet"
    provider: "Anthropic" # different connection

For the models of a provider and their prices, see Models. To chain multiple connections together for failover, cost-aware, or latency-aware routing, see Routing and Failover.

LLM provider connections can also be managed through the APIops REST API; see API Reference: LLM Providers.

Monitoring Connections

• Usage by Deployment — see how many tokens were consumed by cloud vs. on-premise connections on the Usage Reports tab of Reports → Metrics (AI view), under By Deployment Type
• Cost Attribution — each connection logs its deployment type in traffic records for accurate cost allocation
• Failover & Load Balancing — multiple connections to the same provider can be chained together for redundancy (see Routing and Failover)

The Provider Page

Opening a provider shows its page with four cards: General Information, Connection, Models and Advanced. Anyone who may manage AI development sees a Configure button on each card. It opens the card's section in a drawer. Save in the drawer opens a dialog that lists where the provider is used (LLM Gateways, API proxies, proxy groups, policy groups and global policies) and the gateways it reaches; Confirm saves the change and deploys it in one step. When the provider is inactive, the dialog says the edits are saved and the provider stays inactive.

At the top of the page, the environment selector and the Test Connection button run the connection test described above. The "⋯" menu holds:

Group	Item
Status	Activate or Deactivate
Transfer	Export
Impact	Where it is used
History	Audit and Rollback (users who may only view see Audit)
Last	Delete the provider

Activate and Deactivate

A provider is Active or Inactive; the list shows this in the Status column. A new provider is active.

• Activate sends the provider to the gateways. The LLM Gateways and policies that use it can send requests to it again.
• Deactivate removes the provider from every gateway right away. Requests routed to it fail, or go to a failover provider. Its settings and models stay; Activate brings it back. You confirm by typing the provider name.

Before you confirm, the dialog lists each place that uses the provider and what changes there, and the gateways the change reaches. Places that are not in force now (for example an LLM Gateway that is not deployed) are marked, because nothing changes for them.

Deleting a Connection

An LLM provider connection that is in use cannot be deleted. The delete is rejected while anything still points at it, and the error message names where those references live.

The reason is that a provider is a reference-type asset: it is never copied into the proxy or policy document, it is resolved by id at runtime. Without the guard, deleting one would leave dangling references behind and the live gateway would fail on the very next request.

The check covers four containers at once — API Proxy, Proxy Group, Policy Group and standalone (global) policies — across these reference paths:

• An API Proxy's AI routing: primary provider, failover chain, conditional routes, primary pool and semantic embedding provider
• The embedding provider of Semantic Cache and RAG Injection policies
• The external guardrail provider of Prompt Guard and Topic Guard policies

To delete the connection, remove those references first or point them at another provider. The same rule applies to deletes made through APIops: DELETE /apiops/projects/{projectName}/llm-providers/{providerName}/ returns 400 Bad Request for a connection that is still in use.

Where it is used in the "⋯" menu of the provider page lists the LLM Gateways, API proxies and other records that name the provider, so you can see what blocks the delete before you try it.

Troubleshooting

Connection Test Failed

• Select an environment to test through the Worker of that environment. With no environment selected, the Manager sends the request, and ${env.X} variables are not resolved
• Verify the API key or endpoint URL is correct
• Check that credentials have the required permissions
• Ensure the provider service is reachable from the selected environment's network — not just from the Manager's
• AWS Bedrock and Google Vertex AI connections don't support Test Connection yet — save without testing

Credentials Not Saving

• Ensure you have admin permissions
• Clear browser cache and try again

Next Steps

AI Cost Settings

Model prices, currency display, and cost tracking

Routing and Failover

Chain connections for failover and cost/latency-aware routing

Token Rate Limit Policy

Set per-provider or per-model usage limits