Ana içeriğe geç

AI Token Rate Limit Policy

General Information​

Policy Type​

ai-token-rate-limit

Endpoints​

List Policies​

GET /apiops/projects/{projectName}/apiProxies/{apiProxyName}/policies/

Add Policy​

POST /apiops/projects/{projectName}/apiProxies/{apiProxyName}/policies/{policyName}/

Update Policy​

PUT /apiops/projects/{projectName}/apiProxies/{apiProxyName}/policies/{policyName}/

Delete Policy​

DELETE /apiops/projects/{projectName}/apiProxies/{apiProxyName}/policies/{policyName}/

Scope​

This policy configures three things that are independent of each other:

  1. The oversized-request guard — maxTokensPerRequest and maxPromptChars reject or truncate a single request that is too large, regardless of any time-windowed budget.
  2. The AI proxy protection limit — proxyBudget is the total for all callers of the AI proxy. It can be split by a variable (applyBy), with per-value limits in detailList. See AI proxy protection budgets.
  3. Cache behaviour for the time-windowed budget reservation — defaultReservationTtlSeconds, cacheConnectionTimeoutInSeconds and failOpenOnCacheError govern how the policy talks to the shared token-bucket cache.

Consumer, API client, application and organization AI limits are set with Limit Plans, not in this policy. See Limit Plans Management API.

Fields​

FieldTypeRequiredDefaultDescription
defaultReservationTtlSecondsintegerNo600How long a reserved-but-not-yet-reconciled token/USD reservation stays valid in the cache.
cacheConnectionTimeoutInSecondsintegerNo3HTTP timeout, in seconds, for calls to the token-bucket cache.
failOpenOnCacheErrorbooleanNofalseWhen the cache is unreachable: true lets the request pass unmetered; false rejects the request.
maxTokensPerRequestintegerNoUnlimitedPer-request cap on estimated tokens. null means no cap; this is independent of any time-windowed quota.
maxPromptCharsintegerNoUnlimitedPer-request cap on the raw prompt character count (the actual system + messages[].content text, not the full JSON envelope). null means no cap.
overflowActionstringNoBLOCKWhat happens when maxTokensPerRequest is exceeded. BLOCK rejects the request. TRUNCATE_OLDEST drops the oldest non-protected messages (the system messages and the last user message are always kept) until the request fits; if it still doesn't fit, the request is blocked.
overflowAction only applies to the token dimension

maxPromptChars always rejects an oversized request outright; overflowAction: TRUNCATE_OLDEST only applies to maxTokensPerRequest. The model's context window is checked by AI routing, not by this policy. A contextWindowSource field in an older package is ignored.

AI proxy protection budgets​

When a policy body carries an AI proxy protection limit (proxyBudget, and per-value rows in detailList[].budget selected by applyBy), the softCapPercents array of each budget follows these rules:

  • Omitted: the default alert thresholds [50, 80, 90] apply, the same as on the policy screen.
  • []: no alert thresholds.
  • A GET returns [] when no thresholds are set, so a body read back and sent again keeps "no thresholds".

A Token Rate Limit placed on the response or error pipe (any pipe) is counted by the AI usage screen, the AI token usage alarm and the USD price check in the same way as one in the request pipe.

The server returns 400 for an invalid protection: a per-value row without the split variable, value or limit; an invalid regular expression; a model or provider line without its model/provider or limit. See Create Global Policy, Update Global Policy and Create Policy Group.


List Policies​

Endpoint​

GET /apiops/projects/{projectName}/apiProxies/{apiProxyName}/policies/

Request​

Headers​

HeaderValue
AuthorizationBearer {token}
Content-Typeapplication/json

Path Parameters​

ParameterTypeRequiredDescription
projectNamestringYesProject name
apiProxyNamestringYesAPI Proxy name

Response​

Success Response (200 OK)​

{
"status": "SUCCESS",
"resultList": [
{
"apiProxy": {
"name": "MyAiProxy",
"requestPolicyList": [
{
"type": "ai-token-rate-limit",
"name": "tenant-token-cap",
"description": "Per-request oversized guard for the support assistant",
"active": true,
"defaultReservationTtlSeconds": 600,
"cacheConnectionTimeoutInSeconds": 3,
"failOpenOnCacheError": false,
"maxTokensPerRequest": 8000,
"maxPromptChars": 32000,
"overflowAction": "TRUNCATE_OLDEST"
}
],
"responsePolicyList": [],
"errorPolicyList": []
}
}
],
"resultCount": 1
}

cURL Example​

curl -X GET \
"https://demo.apinizer.com/apiops/projects/MyProject/apiProxies/MyAiProxy/policies/" \
-H "Authorization: Bearer YOUR_TOKEN"

Add Policy​

Endpoint​

POST /apiops/projects/{projectName}/apiProxies/{apiProxyName}/policies/{policyName}/

Request​

Headers​

HeaderValue
AuthorizationBearer {token}
Content-Typeapplication/json

Path Parameters​

ParameterTypeRequiredDescription
projectNamestringYesProject name
apiProxyNamestringYesAPI Proxy name
policyNamestringYesPolicy name

Request Body​

Full JSON Body Example​
{
"operationMetadata": {
"targetScope": "ALL",
"targetPipeline": "REQUEST",
"deploy": true,
"deployTargetEnvironmentNameList": ["production"],
"order": 1
},
"policy": {
"type": "ai-token-rate-limit",
"description": "Per-request oversized guard for the support assistant",
"active": true,
"defaultReservationTtlSeconds": 600,
"cacheConnectionTimeoutInSeconds": 3,
"failOpenOnCacheError": false,
"maxTokensPerRequest": 8000,
"maxPromptChars": 32000,
"overflowAction": "TRUNCATE_OLDEST"
}
}

Request Body Fields​

See Fields above. type (ai-token-rate-limit), description and active follow the common policy contract; name comes from the path parameter.

Response​

Success Response (200 OK)​

{
"status": "SUCCESS",
"deploymentResult": {
"success": true,
"deploymentResults": [
{
"environmentName": "production",
"success": true,
"message": "Deployment successful"
}
]
}
}

Error Responses​

CodeReason
400Blank name; proxy not found.
401Missing or invalid token.
403The token's account lacks permission on the project.

cURL Example​

curl -X POST \
"https://demo.apinizer.com/apiops/projects/MyProject/apiProxies/MyAiProxy/policies/tenant-token-cap/" \
-H "Authorization: Bearer YOUR_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"operationMetadata": {
"targetScope": "ALL",
"targetPipeline": "REQUEST",
"deploy": true,
"deployTargetEnvironmentNameList": ["production"],
"order": 1
},
"policy": {
"type": "ai-token-rate-limit",
"description": "Per-request oversized guard",
"active": true,
"maxTokensPerRequest": 8000,
"overflowAction": "TRUNCATE_OLDEST"
}
}'

Update Policy​

Endpoint​

PUT /apiops/projects/{projectName}/apiProxies/{apiProxyName}/policies/{policyName}/

Request​

The body is the same shape as Add Policy. The update replaces the policy, so send the fields you want to keep.

Full JSON Body Example — Raising the Per-Request Cap​
{
"operationMetadata": {
"targetScope": "ALL",
"targetPipeline": "REQUEST",
"deploy": true,
"deployTargetEnvironmentNameList": ["production"],
"order": 1
},
"policy": {
"type": "ai-token-rate-limit",
"description": "Per-request oversized guard for the support assistant",
"active": true,
"maxTokensPerRequest": 16000,
"maxPromptChars": 64000,
"overflowAction": "TRUNCATE_OLDEST"
}
}

Response​

Success Response (200 OK)​

{
"status": "SUCCESS",
"deploymentResult": {
"success": true,
"deploymentResults": [
{
"environmentName": "production",
"success": true,
"message": "Deployment successful"
}
]
}
}

cURL Example​

curl -X PUT \
"https://demo.apinizer.com/apiops/projects/MyProject/apiProxies/MyAiProxy/policies/tenant-token-cap/" \
-H "Authorization: Bearer YOUR_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"operationMetadata": {
"targetScope": "ALL",
"targetPipeline": "REQUEST",
"deploy": true,
"deployTargetEnvironmentNameList": ["production"]
},
"policy": {
"type": "ai-token-rate-limit",
"active": true,
"maxTokensPerRequest": 16000,
"overflowAction": "TRUNCATE_OLDEST"
}
}'

Delete Policy​

Endpoint​

DELETE /apiops/projects/{projectName}/apiProxies/{apiProxyName}/policies/{policyName}/

Request​

Headers​

HeaderValue
AuthorizationBearer {token}

Path Parameters​

ParameterTypeRequiredDescription
projectNamestringYesProject name
apiProxyNamestringYesAPI Proxy name
policyNamestringYesPolicy name

Response​

Success Response (200 OK)​

{
"status": "SUCCESS"
}

cURL Example​

curl -X DELETE \
"https://demo.apinizer.com/apiops/projects/MyProject/apiProxies/MyAiProxy/policies/tenant-token-cap/" \
-H "Authorization: Bearer YOUR_TOKEN"

Notes and Warnings​

  • The oversized-request guard is additive to any time-windowed budget. A request can pass every minute/hour/day/month budget tier and still be rejected here if it alone is larger than maxTokensPerRequest or maxPromptChars.
  • Runs at the model decision in routing. The proxy protection limit runs once in the routing stage, after the provider and model are resolved and the model access list has let the caller through, in the order of the policy chain. It never runs with a guessed model. In the request pipeline and in the timing view it appears as skipped with the reason Deferred to the model decision in routing; the real execution is a second record in routing. MCP and A2A proxies are not affected.
  • Status code order. The request body size limit (413) and the stream compatibility checks (400: a unary-only policy with a streamed request, Bedrock streaming) are evaluated before the token limit (429) and before a cache answer. An extremely long prompt gets 413, not 429.
  • TRUNCATE_OLDEST does not truncate a conversation above the body size limit (default 10 MB); the request is rejected with 413.
  • A truncated conversation is sent as truncated. When the policy shortens the conversation to fit maxTokensPerRequest, the outgoing request, including failover legs, is rebuilt from the shortened body on the same provider and model.
  • maxPromptChars counts prompt text only, not the JSON envelope (model name, stream, tool definitions, and so on) — a short user message is never blocked by the surrounding envelope size.
  • TRUNCATE_OLDEST never silently forwards an oversized request. If the protected messages (all system turns plus the last user turn) alone exceed the limit, or the body isn't parseable messages[] JSON, the request falls back to BLOCK.
  • The model's context window is not checked by this policy. AI routing checks it (default: reject with HTTP 400, error code ERR-311).
  • failOpenOnCacheError: false (the default) rejects requests when the token-bucket cache is unreachable. Set it to true only if uninterrupted traffic matters more than enforcement during a cache outage.