AI Token Rate Limit Policy
General Information
Policy Type
ai-token-rate-limit
Endpoints
List Policies
GET /apiops/projects/{projectName}/apiProxies/{apiProxyName}/policies/
Add Policy
POST /apiops/projects/{projectName}/apiProxies/{apiProxyName}/policies/{policyName}/
Update Policy
PUT /apiops/projects/{projectName}/apiProxies/{apiProxyName}/policies/{policyName}/
Delete Policy
DELETE /apiops/projects/{projectName}/apiProxies/{apiProxyName}/policies/{policyName}/
Scope
This policy configures three things that are independent of each other:
- The oversized-request guard —
maxTokensPerRequestandmaxPromptCharsreject or truncate a single request that is too large, regardless of any time-windowed budget. - The AI proxy protection limit —
proxyBudgetis the total for all callers of the AI proxy. It can be split by a variable (applyBy), with per-value limits indetailList. See AI proxy protection budgets. - Cache behaviour for the time-windowed budget reservation —
defaultReservationTtlSeconds,cacheConnectionTimeoutInSecondsandfailOpenOnCacheErrorgovern how the policy talks to the shared token-bucket cache.
Consumer, API client, application and organization AI limits are set with Limit Plans, not in this policy. See Limit Plans Management API.
Fields
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| defaultReservationTtlSeconds | integer | No | 600 | How long a reserved-but-not-yet-reconciled token/USD reservation stays valid in the cache. |
| cacheConnectionTimeoutInSeconds | integer | No | 3 | HTTP timeout, in seconds, for calls to the token-bucket cache. |
| failOpenOnCacheError | boolean | No | false | When the cache is unreachable: true lets the request pass unmetered; false rejects the request. |
| maxTokensPerRequest | integer | No | Unlimited | Per-request cap on estimated tokens. null means no cap; this is independent of any time-windowed quota. |
| maxPromptChars | integer | No | Unlimited | Per-request cap on the raw prompt character count (the actual system + messages[].content text, not the full JSON envelope). null means no cap. |
| overflowAction | string | No | BLOCK | What happens when maxTokensPerRequest is exceeded. BLOCK rejects the request. TRUNCATE_OLDEST drops the oldest non-protected messages (the system messages and the last user message are always kept) until the request fits; if it still doesn't fit, the request is blocked. |
maxPromptChars always rejects an oversized request outright; overflowAction: TRUNCATE_OLDEST only applies to maxTokensPerRequest. The model's context window is checked by AI routing, not by this policy. A contextWindowSource field in an older package is ignored.
AI proxy protection budgets
When a policy body carries an AI proxy protection limit (proxyBudget, and per-value rows in detailList[].budget selected by applyBy), the softCapPercents array of each budget follows these rules:
- Omitted: the default alert thresholds
[50, 80, 90]apply, the same as on the policy screen. []: no alert thresholds.- A
GETreturns[]when no thresholds are set, so a body read back and sent again keeps "no thresholds".
A Token Rate Limit placed on the response or error pipe (any pipe) is counted by the AI usage screen, the AI token usage alarm and the USD price check in the same way as one in the request pipe.
The server returns 400 for an invalid protection: a per-value row without the split variable, value or limit; an invalid regular expression; a model or provider line without its model/provider or limit. See Create Global Policy, Update Global Policy and Create Policy Group.
List Policies
Endpoint
GET /apiops/projects/{projectName}/apiProxies/{apiProxyName}/policies/
Request
Headers
| Header | Value |
|---|---|
| Authorization | Bearer {token} |
| Content-Type | application/json |
Path Parameters
| Parameter | Type | Required | Description |
|---|---|---|---|
| projectName | string | Yes | Project name |
| apiProxyName | string | Yes | API Proxy name |
Response
Success Response (200 OK)
{
"status": "SUCCESS",
"resultList": [
{
"apiProxy": {
"name": "MyAiProxy",
"requestPolicyList": [
{
"type": "ai-token-rate-limit",
"name": "tenant-token-cap",
"description": "Per-request oversized guard for the support assistant",
"active": true,
"defaultReservationTtlSeconds": 600,
"cacheConnectionTimeoutInSeconds": 3,
"failOpenOnCacheError": false,
"maxTokensPerRequest": 8000,
"maxPromptChars": 32000,
"overflowAction": "TRUNCATE_OLDEST"
}
],
"responsePolicyList": [],
"errorPolicyList": []
}
}
],
"resultCount": 1
}
cURL Example
curl -X GET \
"https://demo.apinizer.com/apiops/projects/MyProject/apiProxies/MyAiProxy/policies/" \
-H "Authorization: Bearer YOUR_TOKEN"
Add Policy
Endpoint
POST /apiops/projects/{projectName}/apiProxies/{apiProxyName}/policies/{policyName}/
Request
Headers
| Header | Value |
|---|---|
| Authorization | Bearer {token} |
| Content-Type | application/json |
Path Parameters
| Parameter | Type | Required | Description |
|---|---|---|---|
| projectName | string | Yes | Project name |
| apiProxyName | string | Yes | API Proxy name |
| policyName | string | Yes | Policy name |
Request Body
Full JSON Body Example
{
"operationMetadata": {
"targetScope": "ALL",
"targetPipeline": "REQUEST",
"deploy": true,
"deployTargetEnvironmentNameList": ["production"],
"order": 1
},
"policy": {
"type": "ai-token-rate-limit",
"description": "Per-request oversized guard for the support assistant",
"active": true,
"defaultReservationTtlSeconds": 600,
"cacheConnectionTimeoutInSeconds": 3,
"failOpenOnCacheError": false,
"maxTokensPerRequest": 8000,
"maxPromptChars": 32000,
"overflowAction": "TRUNCATE_OLDEST"
}
}
Request Body Fields
See Fields above. type (ai-token-rate-limit), description and active follow the common policy contract; name comes from the path parameter.