Update AI Routing

Endpoint

PUT /apiops/projects/{projectName}/apiProxies/{apiProxyName}/ai-routing/

Warning

This endpoint returns HTTP 400 for API proxies whose type is not AI, except an A2A proxy in AGENT_EXPOSE mode, which keeps its LLM backend in aiRouting too. aiRouting can only be set on those proxies. To create a new AI proxy in one call, use Create AI API Proxy — this endpoint only updates the routing config of a proxy that already exists.

Authentication

Requires a Personal API Access Token.

Permission: API_MANAGEMENT × MANAGE, or AI_DEVELOPMENT × MANAGE. Both are accepted on AI, MCP and A2A proxies.

Header

Authorization: Bearer YOUR_TOKEN

Request

Headers

Header	Value	Required
Authorization	Bearer {token}	Yes
Content-Type	application/json	Yes

Path Parameters

Parameter	Type	Required	Description
projectName	string	Yes	Project name
apiProxyName	string	Yes	AI API Proxy name

Request Body

The body is the full AiRouting object — it replaces the proxy's current AI routing configuration.

Full JSON Body Example

{
  "llmProviderRefName": "openai-prod",
  "modelId": "gpt-4o-mini",
  "clientModelPresentBehavior": "OVERWRITE",
  "clientModelAbsentBehavior": "KEEP_ABSENT",
  "contextWindowAction": "TRUNCATE",
  "contextKeepTurns": 2,
  "contextRagAction": "KEEP",
  "contextLastUserMessageAction": "KEEP",
  "streaming": false,
  "timeoutSeconds": 60,
  "connectTimeoutSeconds": 10,
  "streamTimeoutSeconds": 120,
  "retryCount": 0,
  "failoverStrategy": "SEQUENTIAL",
  "providerFailoverChain": [],
  "conditionalRoutes": [],
  "primaryPool": [],
  "enumAiRoutingAlgorithm": "ROUND_ROBIN",
  "circuitBreakerEnabled": false,
  "proxyEnabled": false,
  "mTLSSettings": null,
  "blockAnonymousRequests": false
}

Field Reference — Primary Routing

Field	Type	Description
llmProviderRef	string	Primary LLM provider connection id. Use llmProviderRefName instead when referencing by name
llmProviderRefName	string	Name-based sidecar for llmProviderRef. Resolved to an id within the project when present
modelId	string	Primary model id, a model of the primary LLM provider (for example gpt-4o-mini, deepseek-chat)
clientModelPresentBehavior	string	What to do when the client's request body already has a model field — USE_CLIENT, OVERWRITE, or ERROR. Default OVERWRITE when unset
clientModelAbsentBehavior	string	What to do when the client's request body has no model field — KEEP_ABSENT, ADD_ROUTING_MODEL, or ERROR. Default KEEP_ABSENT when unset
streaming	boolean	Enable SSE streaming responses. Default false. When clientStreamPresentBehavior is USE_CLIENT, it applies only to requests that carry no stream field
clientStreamPresentBehavior	string	What to do when the client's request body already has a stream field — USE_CLIENT or OVERWRITE. With USE_CLIENT, stream: true returns a streaming (SSE) response and stream: false returns a single response; with OVERWRITE, streaming always wins. The setting applies to chat and responses requests only. A body that omits the field stores no value, which behaves as OVERWRITE; the Create AI API Proxy endpoint starts new proxies with USE_CLIENT
contextWindowAction	string	What happens when a chat or responses request does not fit the selected model's context window — REJECT, TRUNCATE, or PASS. Default REJECT when unset. The input limit is the model's contextWindow (set on the LLM provider's model) minus the requested output tokens (max_completion_tokens, max_tokens, or max_output_tokens) minus a 512-token safety margin. REJECT answers HTTP 400 with error code ERR-311 (context_length_exceeded) before any provider call. TRUNCATE drops the oldest conversation turns until the request fits. PASS sends the request to the provider unchanged. No check runs when the model has no context window value. With failover, a candidate whose context window is too small is skipped. See Context Window
contextKeepTurns	integer	TRUNCATE only: how many conversation turns before the last one are never dropped. System and developer messages and the last turn are always kept. A tool call and its result are dropped together. Default 0. The Manager UI accepts 0–50
contextRagAction	string	TRUNCATE only: whether the context added by a RAG Injection policy may be shortened — KEEP or TRIM_END (shortened from its end). Default KEEP
contextLastUserMessageAction	string	TRUNCATE only: whether the gateway may shorten the last user message when dropping turns and cutting the RAG context are not enough — KEEP or TRIM_MIDDLE (the middle of the text is removed; the start and end are kept, and a note marks the removed part). At most half of the text is removed; otherwise the request is rejected. Only text is cut. Default KEEP; an omitted or null value resets to KEEP, because the body replaces the whole object
timeoutSeconds	integer	Response timeout in seconds: the longest wait for the provider's answer to a single-response (non-streaming) request. Default 60

USE_CLIENT is not honored on Azure OpenAI

Gemini, Vertex, and Bedrock carry the effective model in the URL path, which the gateway builds itself — attribution and the model that answers always agree. An Azure OpenAI deployment endpoint pins the deployment in the endpoint the operator configured, so the request body's model field cannot change which model actually answers — it could only shift the attribution (the model entry of the provider that limits, quota, semantic cache, cost calculation, and traffic logs are booked against) to a value the client supplies. Apinizer therefore does not apply clientModelPresentBehavior: "USE_CLIENT" on an AZURE_OPENAI provider: the effective model stays the configured modelId and no check against the provider's model list is run on the client's value. The upstream request carries the configured modelId in its body, exactly as it does under OVERWRITE — the client's value only stops being a reason to reject the request. OVERWRITE, ERROR, and clientModelAbsentBehavior are unaffected.

A failover entry without its own modelId inherits the configured modelId above, not the client's model, and re-evaluates USE_CLIENT against the model list of its own provider.

Field Reference — Timeouts, Limits, and Retry

Field	Type	Description
connectTimeoutSeconds	integer	Connection establish timeout (DNS + TCP + TLS) to the LLM provider. Applies to streaming and single-response requests. Default 10
streamTimeoutSeconds	integer	SSE stream idle timeout: the stream is closed when the provider sends no new chunk for this long. Default 120
maxResponseBytes	long	Maximum response body size; null or ≤ 0 = unlimited
maxChunks	long	Maximum SSE chunks; null or ≤ 0 = unlimited
retryCount	integer	How many times a single-response request is retried after a transient connection error. Streaming requests are not retried. Default 0
retryDelayType	string	NO_DELAY, FIXED, EXPONENTIAL_BACKOFF
retryFixedDelayMs	integer	Fixed retry delay in ms. Default 500
retryInitialDelayMs	integer	Initial backoff delay in ms. Default 500
retryMaxDelayMs	integer	Maximum backoff delay in ms. Default 10000
retryBackoffMultiplier	number	Backoff multiplier. Default 2.0

Field Reference — Connection Pool

Connections to each LLM provider of the proxy are reused through a pool. The proxy keeps one pool for each LLM provider in its routing (primary, conditional routes, pool entries, failover chain). The pool is created when the proxy is deployed and applies to streaming requests.

Field	Type	Description
connectionPoolManagementType	string	GENERAL, CUSTOM or NONE. GENERAL: the proxy uses its own pool sized by the connection pool defaults in AI Settings. CUSTOM: the proxy uses its own pool sized by the two fields below. NONE: no pooling; every request opens a new connection that closes when the request ends. Default GENERAL when unset
customConnectionPoolSize	integer	CUSTOM only. Idle connections kept open for reuse, for each LLM provider separately. Range 1 to 1000; a value outside the range is set to the nearest limit. Default 8 when unset
idleConnectionTimeout	integer	CUSTOM only. Seconds an idle connection is kept. Range 1 to 3600; a value outside the range is set to the nearest limit. Keep it below the provider's own idle timeout (vLLM and other uvicorn servers close idle connections after 5 seconds)

The size limits idle connections only. Connections in use are not limited and requests never wait for a connection. With GENERAL, the size and the idle timeout are taken from AI Gateway Settings on every deploy, so any value sent in these two fields is replaced; a change in AI Gateway Settings reaches the gateway with the next deploy.

Each entry of primaryPool, providerFailoverChain and conditionalRoutes also accepts the provider-specific pool fields below. They are not available on the single primary provider of aiRouting itself.

Field	Type	Description
connectionPoolSize	integer	Optional. Idle connections kept for the entry's LLM provider. Range 1 to 1000. Empty = the routing-level setting applies
idleConnectionTimeout	integer	Optional. Seconds an idle connection of the entry's provider is kept. Range 1 to 3600. Empty = the routing-level setting applies

The proxy keeps one pool for each LLM provider. When several entries use the same provider, the largest value of each field is used. A value in these fields opens a pool for the provider also when connectionPoolManagementType is NONE; the other field, if empty, takes the routing-level value (customConnectionPoolSize or idleConnectionTimeout) if the routing has one. If the routing has none, it takes the AI Gateway Settings default. The screen clears the routing-level values when None is chosen; a routing written with the API can keep them.

Example: connectionPoolManagementType is NONE, the routing has customConnectionPoolSize 8 and idleConnectionTimeout 30, and one primaryPool entry has only connectionPoolSize 100. That provider gets pool size 100 and idle timeout 30 seconds. It does not get the AI Gateway Settings value of 4 seconds.

Field Reference — Failover and Load Balancing

Field	Type	Description
failoverStrategy	string	SEQUENTIAL, ROUND_ROBIN, or CONDITIONAL
providerFailoverChain	array	Ordered failover entries — each with llmProviderRef/llmProviderRefName, optional modelId, credentialRef, timeoutMs, orderIndex, displayName
failoverStatusCodeList	array[integer]	HTTP status codes eligible for failover. Empty = default 5xx/408/429
failoverRetryCount	integer	Inner retry count per failover entry. Default 1
conditionalLatencyThresholdMs	integer	For CONDITIONAL strategy — proactive latency threshold in ms
trySiblingPoolTargets	boolean	What happens when the selected pool target fails. true or unset: the selected target, retries on the same target (if any), the other pool targets, then the failover chain. false: the selected target, retries on the same target (if any), then the failover chain directly; with an empty chain the request returns an error. Applies to primaryPool and to every conditional route targetPool. Unset behaves as true
primaryPool	array	N-primary load-balancing pool — each entry with llmProviderRef/llmProviderRefName, optional modelId, weight, displayName
enumAiRoutingAlgorithm	string	ROUND_ROBIN, WEIGHTED, RANDOM, LEAST_COST, PREFIX_AFFINITY, LRU
leastCostOutputRatio	number	Output-to-input cost weight for LEAST_COST. Default 0.25
affinityKeyHeader	string	Header for PREFIX_AFFINITY affinity key. Default session_id
affinityLoadFactor	number	Bounded-load factor for PREFIX_AFFINITY. Default 1.25

Field Reference — Conditional Routes

Field	Type	Description
conditionalRoutes	array	Ordered route variants — each with name, condition, optional llmProviderRef/llmProviderRefName, modelId, semanticUtterances, semanticThreshold, and optionally a target pool (see below)
semanticEmbeddingProviderRef	string	Embedding provider id for semantic route matching
semanticEmbeddingProviderRefName	string	Name-based sidecar for semanticEmbeddingProviderRef. Resolved to an id within the project when present (Apinizer 2026.09.4)
semanticEmbeddingModelId	string	Embedding model id. Default text-embedding-3-small

Each entry of conditionalRoutes also accepts the target pool fields below. A route with a non-empty targetPool is a target pool route: it load-balances the matching requests across its targets with its own algorithm. A route without targetPool (empty or absent) is a single-target route that uses llmProviderRef and modelId, as before. When targetPool is set, the route is not validated or routed by llmProviderRef and modelId.

Field	Type	Description
targetPool	array	Targets of the route pool. Each entry has the fields in the next table
enumAiRoutingAlgorithm	string	Algorithm of this route pool: ROUND_ROBIN, WEIGHTED, RANDOM, LRU, LEAST_COST, LEAST_LATENCY, PREFIX_AFFINITY. Default ROUND_ROBIN
affinityKeyHeader	string	Header for the PREFIX_AFFINITY key of this route. Default session_id
affinityLoadFactor	number	Bounded-load factor for PREFIX_AFFINITY of this route. Default 1.25

Fields of a targetPool entry:

Field	Type	Description
llmProviderRef	string	LLM provider connection id. Required. Use llmProviderRefName instead when referencing by name
llmProviderRefName	string	Name-based sidecar for llmProviderRef (export/import). Resolved to an id within the project when present
modelId	string	Model id. Required. A target does not inherit the model of the default target
weight	integer	Used by WEIGHTED only. A higher value receives more traffic. Empty or 0 or less counts as 1
displayName	string	Name of the target in the Manager UI
connectionPoolSize	integer	Optional provider-specific pool size. See Connection Pool above
idleConnectionTimeout	integer	Optional provider-specific idle connection timeout in seconds. See Connection Pool above

Example — a target pool route. Requests that match the route rule (for example model = qwen3-8b) are distributed across three servers with Round Robin. If the selected server fails, the other two are tried (trySiblingPoolTargets is on):

{
  "llmProviderRefName": "openai-prod",
  "modelId": "gpt-4o-mini",
  "trySiblingPoolTargets": true,
  "conditionalRoutes": [
    {
      "name": "qwen3-8b farm",
      "enumAiRoutingAlgorithm": "ROUND_ROBIN",
      "targetPool": [
        { "llmProviderRefName": "vllm-a1", "modelId": "qwen3-8b", "displayName": "a1" },
        { "llmProviderRefName": "vllm-a2", "modelId": "qwen3-8b", "displayName": "a2" },
        { "llmProviderRefName": "vllm-a3", "modelId": "qwen3-8b", "displayName": "a3" }
      ]
    }
  ]
}

The example omits the condition field, which carries the rule model = qwen3-8b. Create the rule in the Manager UI and read the route back to see the condition shape.

Each entry of conditionalRoutes also accepts the classifier fields below. A route with a classifierLabel is a classifier route: the proxy-level LLM classifier answers with one label, and the route with that label is selected. Several routes may share one label; the first one whose condition also matches wins.

Field	Type	Description
classifierLabel	string	Label of a classifier route. Lowercase letters, digits, _ and -; maximum 64 characters; must start and end with a letter or digit. none is reserved
classifierDescription	string	Required when classifierLabel is set. Maximum 500 characters. The classifier model decides by this text

The proxy-level classifier settings are fields of aiRouting itself:

Field	Type	Description
classifierProviderRef	string	Classifier LLM provider connection id. The provider must be able to make decision calls: an OpenAI-compatible chat provider (OpenAI, Azure OpenAI, DeepSeek, Moonshot, Zhipu, Mistral, Groq, Ollama, vLLM, xAI, Google AI Studio, Together, Fireworks, Cerebras, Hugging Face, OpenRouter, Perplexity, NVIDIA NIM, or a custom provider whose adapter family is OPENAI_COMPATIBLE or VLLM), Anthropic, or TypeSafe Jev. AWS Bedrock, Google Vertex AI, Cohere, the native Gemini wire format, embedding-only providers, TEI, and custom providers without an adapter family are rejected. The provider must use credentials held by the AI gateway (Provided by AI gateway)
classifierProviderRefName	string	Name-based sidecar for classifierProviderRef. Resolved to an id within the project when present
classifierModelId	string	Model id of the classifier, a model of the classifier provider. Required when at least one classifier route exists
classifierDecisionMode	string	Optional. STANDARD or REASONING. Unset is STANDARD. STANDARD sends temperature 0 and a small output limit. REASONING is for thinking models: no temperature is sent and classifierReasoningBudget is the output limit. Does not apply to TypeSafe Jev
classifierReasoningBudget	integer	Optional. Used only when classifierDecisionMode is REASONING. Upper limit for thinking plus answer, in tokens. Range 256 to 8192. Default 1024. A value outside the range is set to the nearest limit
classifierTimeoutMs	integer	Classifier timeout in milliseconds. STANDARD: default 1000, range 100 to 10000. REASONING: default 15000, up to 60000. A value that you set always wins over the default
classifierMaxInputChars	integer	Maximum characters of the last user message sent to the classifier. Default 2000. Range 200 to 20000
classifierInstructions	string	Optional extra instructions for the classifier. ${env} environment variables are resolved; the user prompt is never resolved

Example — two routes that share the label code. To split them by request size, add a condition on the context value Estimated Request Token Count (data type numeric, operator greater than) to the first route; the first route whose condition matches wins:

{
  "classifierProviderRefName": "openai-prod",
  "classifierModelId": "gpt-4o-mini",
  "classifierTimeoutMs": 1000,
  "classifierMaxInputChars": 2000,
  "classifierInstructions": "Questions about invoices are finance.",
  "conditionalRoutes": [
    {
      "name": "Large code requests",
      "classifierLabel": "code",
      "classifierDescription": "Programming questions: writing, fixing or explaining code.",
      "llmProviderRefName": "openai-prod",
      "modelId": "gpt-4o"
    },
    {
      "name": "Code requests",
      "classifierLabel": "code",
      "classifierDescription": "Programming questions: writing, fixing or explaining code.",
      "llmProviderRefName": "openai-prod",
      "modelId": "gpt-4o-mini"
    }
  ]
}

Example — a reasoning classifier. A reasoning model adds seconds and token cost to every request that reaches a classifier route, so use a longer timeout and test the routing first:

{
  "classifierProviderRefName": "anthropic-prod",
  "classifierModelId": "claude-sonnet-4-5",
  "classifierDecisionMode": "REASONING",
  "classifierReasoningBudget": 1024,
  "classifierTimeoutMs": 15000
}

Example — TypeSafe Jev as the classifier. classifierDecisionMode and classifierReasoningBudget do not apply:

{
  "classifierProviderRefName": "typesafe-jev",
  "classifierModelId": "jev-latest"
}

TypeSafe Jev and TEI are decision-only providers. TypeSafe Jev can be the classifier or a guard judge. TEI can be a guard judge only. Neither can be a routing target, a pool member, a failover target, or an embedding provider.

At request time the classifier is called at most once, only when a classifier route is reached, and only in the routing phase (after authentication and PII masking). After a classifier timeout or error, classifier calls for that proxy are skipped for 15 seconds. If it fails, times out, refuses to answer, answers with an unknown label, or has no provider, classifier routes do not match and routing continues down the list (fail-open).

Field Reference — Circuit Breaker

Field	Type	Description
circuitBreakerEnabled	boolean	Enable circuit breaker. Default false
errorWindow	integer	Error observation window in seconds. Default 10
errorThresholdValue	integer	Error threshold. Default 1
enumErrorThresholdType	string	COUNT or PERCENTAGE
sleepWindow	integer	Open-circuit sleep window in seconds. Default 10
halfOpenEnabled	boolean	Enable half-open probe. Default false

Field Reference — Outbound Proxy, mTLS, and SSL

Field	Type	Description
proxyEnabled	boolean	Route outbound LLM calls through an HTTP proxy
proxyHost	string	Outbound proxy host
proxyPort	integer	Outbound proxy port
proxyAuthorizationNeeded	boolean	Proxy requires authentication
proxyUsername	string	Proxy username
proxyPassword	string	Proxy password (encrypted at rest; blank preserves existing on update)
mTLSSettings	object	Client mTLS settings for provider connections
disableSslValidation	boolean	Accept self-signed certificates (lab only)

Field Reference — Anonymous Traffic and Trace

Field	Type	Description
blockAnonymousRequests	boolean	Reject unauthenticated requests with HTTP 401. Default false
traceMode	string	AI trace mode override — METRIC, FULL, etc.
enableAiTrace	boolean	AI trace lifecycle flag
aiTraceEnvironmentName	string	Environment where AI trace was started
aiTraceCustomQueryId	string	CustomQuery filter id for AI trace
aiTraceMaxDurationSeconds	integer	Maximum AI trace duration. Default 300

Behavior

• llmProviderRefName and semanticEmbeddingProviderRefName, classifierProviderRefName (and the name sidecars on failover entries, primary pool entries, conditional routes, and conditional route target pool entries) are resolved to connection ids before save. Apinizer looks for each name in the project of the API proxy first, then in the admin project (shared providers). A provider in the own project wins when both have the same name. A blank sidecar leaves the corresponding id field untouched; an unresolvable name does the same — it does not clear or reject the id by itself. If the id left in place (from the body, or unchanged because the name did not resolve) is missing or belongs to a different project, the save is rejected with error key aiRouting.providerOutOfScope.
• An empty routing object ({}) is accepted and replaces (clears) the existing routing. No conditional route, pool entry, failover entry, default provider, or default model is required. If the proxy is then deployed, every request fails with HTTP 503 (POLICY_ROUTING_AI_NO_SUITABLE_PROVIDER) until a target is added. Deploy is not blocked.
• Every environment the proxy is currently deployed to is marked redeploy required — redeploy to activate the new configuration.
• AI proxies do not use the classic routing object. Settings endpoints that target classic routing (circuit breaker, connection, mTLS, addresses, …) return HTTP 400 for AI proxies and point here instead.

Response

Success Response (200 OK)

{
  "status": "SUCCESS",
  "deploymentResult": {
    "success": false,
    "detailList": []
  }
}

The call does not deploy the proxy. Redeploy it to activate the change.

Error Responses

Status	Condition
400	Blank apiProxyName, empty body, proxy not found, or proxy is neither an AI proxy nor an A2A proxy in AGENT_EXPOSE mode
400	A provider id referenced anywhere in aiRouting (primary, failover, pool, conditional route, or the semantic embedding provider) is missing or belongs to a different project (error key aiRouting.providerOutOfScope). Missing and out-of-scope ids share this key so the response does not reveal whether an id exists in another project.
400	A classifier route or the classifier settings are invalid. Error keys: aiRouting.classifierProviderRequired (a classifier route exists but the classifier provider or model is missing), aiRouting.classifierProviderWire (the classifier provider cannot make decision calls), aiRouting.classifierLabelInvalid (the label does not match lowercase letters, digits, _ and -, maximum 64 characters, starting and ending with a letter or digit), aiRouting.classifierLabelReserved (the label is none), aiRouting.classifierDescriptionRequired (a classifier route has a blank description or one longer than 500 characters), aiRouting.classifierSemanticConflict (a route defines both a classifier label and semantic utterances)
400	A targetPool entry of a conditional route has no provider (error key aiRouting.primaryPoolProviderRequired) or no model (error key aiRouting.entryModelRequired)
400	A decision-only provider (TypeSafe Jev or TEI) is used as a routing target, a pool member, a failover target, or an embedding provider (error key aiRouting.decisionOnlyProvider)
400	A target field is filled but the definition produces no target, for example a provider is set but modelId is empty and there are no list entries (error key aiRouting.targetRequired). A routing with no target field filled at all is accepted, not rejected
401	Invalid or missing token
500	Unexpected server error

cURL Example

curl -X PUT \
  "https://demo.apinizer.com/apiops/projects/MyProject/apiProxies/my-ai-proxy/ai-routing/" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "llmProviderRefName": "openai-prod",
    "modelId": "gpt-4o-mini",
    "streaming": false
  }'

Related Documentation

• Create AI API Proxy
• Update MCP Routing
• Update A2A Routing
• API Proxies
• AI Gateway