Token Rate Limit Policy
The Token Quotas screen and the budgets it saved are removed: organization, application, project and platform budgets, and the default budgets on the AI Gateway tab of System Settings. Apinizer did not convert them. Recreate each limit in one of these places:
- A limit for a caller (consumer, API client, application or organization): an AI Token Limit Limit Plan.
- A limit for a project or for the whole platform was removed: Project and Platform Limits Removed.
- A limit for one LLM Gateway: LLM Gateway Protection on this policy.
Token limit management belongs to the AI Gateway license module: without that module the AI Gateway screens are hidden in the Manager.
The API is not gated. The Management API and the APIops endpoints answer normally on an installation whose license does not name AI Gateway.
Losing the module deletes nothing: existing configuration and usage counters are preserved. Workers that are already deployed keep enforcing the limits they were given.
What the Policy Does
The Token Rate Limit policy protects one LLM Gateway. It does two jobs:
- LLM Gateway Protection limits the tokens and cost that all callers of the proxy spend together.
- Oversized Guard blocks a single request that is too large.
The policy does not set limits for each caller. Limits for a consumer, API client, application or organization are set on Limit Plans. The gateway applies them to every AI request, with or without this policy.
Deleting or deactivating the policy removes only the protection of the proxy. Caller limits keep working.
A new LLM Gateway is created with no policies. Add the policy to the request policy list of an LLM Gateway to use it.
Add Token Rate Limit to the request policy list of the LLM Gateway.
In the LLM Gateway Protection section, turn on Apply the LLM Gateway limit. When it is off, the limit fields are inactive.
Set the token windows, the USD windows, Hard Cap (%), Alert Thresholds (%) and Overflow Action.
Deploy the LLM Gateway. Then compare the limits with the real use in Reports and Analytics.
LLM Gateway Protection
LLM Gateway Limit is the total for every caller that reaches the proxy. The Apply the LLM Gateway limit switch sits above the limit fields. When it is off, the fields are inactive and no limit is applied to the proxy.
The limit is a budget form with these fields:
| Field | Description |
|---|---|
| Tokens/Minute, Tokens/Hour, Tokens/Day, Tokens/Month | Token limits for each window. |
| USD/Minute, USD/Hour, USD/Day, USD/Month | Cost limits for each window. USD/Minute accepts up to 4 decimals; the other fields accept 2. |
| Hard Cap (%) | The percentage of a limit at which the overflow action starts. The default, 100, means when the limit is full. |
| Alert Thresholds (%) | Percentages at which an alert is raised. |
| Overflow Action | What happens when a limit is reached. See Overflow Actions. |
Token limits and USD limits are independent. You can set only token limits, only USD limits, or both.
Windows are aligned to the UTC calendar. A minute, hour, day or month window renews at the start of the next minute, hour, day (00:00 UTC) or month. It does not renew a fixed time after the first request.
Model Limits and Provider Limits
Under Model limits you give one model its own limit. Under Provider limits you give one LLM provider its own limit. A request is counted in the proxy limit and in every line that covers it. The first limit that fills stops the request.
Each line needs a model or a provider and at least one limit: tokens or USD per hour, day or month. The lines have no minute window, for tokens or for USD. The screen refuses a model or a provider that is listed twice. A model line counts the model on every provider that serves it; this differs from a model limit on a Limit Plan, which names one model of one provider.
The policy has no request-count limits. A request-count limit (Requests/Minute, Hour, Day or Month) is set on an AI Token Limit Limit Plan; the policy refuses it when you save.
Split by Variable
By default the proxy limit is one total for every caller. With Split by Variable, you select a variable, for example a tenant header. The limit is then counted separately for each value of the variable.
Under Per-Value Limits, click Add Value to give one value its own limit. Each row has these fields:
| Field | Description |
|---|---|
| Value | The value of the variable that the row covers. |
| Regular Expression | When on, the row covers every value that matches the expression. The expression needs a capture group, for example ^(team)-.*$. The first capture group is the value the row counts under. Values that share a capture group share one counter. |
| Limit form | The row's own token, USD, hard cap, alert and overflow settings. |
| Active | The row must be on and must set at least one limit. The server rejects a row that does not. |
A request whose value matches a row is counted against that row. Other values are counted against the default limit, each value on its own.
Per-value rows need a variable. The screen asks you to select a variable or remove the rows. Rows work like the per-value rows of the API Throttling policy.
A USD limit meters only models that have an input price. The price is "price per million input tokens" on the model, in the Models section of its LLM provider. A model without a price counts as 0 in cost reports. A USD limit does not meter such a model, so the limit never cuts it off. Apinizer does not refuse to save the limit.
Set an input price for every model that the USD limit must cover, or use a token limit instead. See Limit Plans.
This also applies to Use the client's model (Routing > Client "model" field). With this option the client can ask the routed LLM provider for any enabled model, failover providers included. A USD limit meters only those models that have a price. A provider whose models were added with Discover from provider (all models added) usually contains unpriced models, such as speech or image models. Set their price, or deactivate them with Deactivate in the model's row menu. When the provider does not have Only listed models on, the client can also ask for a model that is not on the list if its ID is a plain name. Such a model has no price, so the USD limit never meters it. See Models That Are Not on the Provider's List.
Azure OpenAI is the exception: the deployment pins the model, so the client's value is not applied.
The same rule for the per-value rows (a row needs its key and at least one limit, and a regular expression must be valid) also runs when an export package replaces a Token Rate Limit policy or policy group and when a policy version is rolled back.
Overflow Actions
When a request would exceed a limit, you choose what happens next:
The request is rejected outright. The 429 response also carries a Retry-After header (in seconds) stating when the window reopens.
The request is routed to the backup provider in the failover chain instead of being rejected. If the routing has no backup provider, the request is rejected with HTTP 429. It is not sent to the over-budget primary.
The request goes through as usual; only a warning is raised.
The request is automatically redirected to the Cheaper Model instead of being blocked.
If no cheaper model is defined, the provider does not allow the switch, or the caller has no access to the cheaper model, the request is rejected with HTTP 429 instead.
The Retry-After value is the time remaining until the exceeded window (minute/hour/day/month) closes, but it is capped by the apinizer.limit.retryAfterClampSeconds JVM system property on the worker (default 86400, i.e. one day) — so a monthly window never reports a 30-day wait, at most one day. A client that retries earlier simply gets a new 429 with an updated Retry-After; counter accuracy is unaffected. The clamp applies to every window, not only monthly ones.
Per-Request Size Limit
Separate from the proxy limit, you can cap the size of a single request. The policy checks this before the time-windowed limit, as a cheap pre-flight step. It lives in the Oversized Guard section:
This section works on its own. An installation that only wants to cap request size can add the Token Rate Limit policy and fill in these fields alone, with no LLM Gateway Limit — the limit tier never engages and the request never reaches the cache. The Quota Reservation & Cache Behavior settings only matter once the LLM Gateway Limit is on.
| Field | Description | Default |
|---|---|---|
| Max Tokens Per Request | Blocks a single request whose estimated token count exceeds this value. Empty = no limit. | — |
| Max Prompt Characters | Blocks a single request whose prompt character count exceeds this value — only the prompt text itself is counted, not JSON envelope fields such as model or stream. Empty = no limit. | — |
| On Overflow | What happens when Max Tokens Per Request is exceeded: Block rejects the request; Truncate Oldest Messages drops the oldest conversation turns instead, until the request fits. Character-count overflow always blocks, regardless of this setting. | Block |
When a request is truncated, Apinizer always preserves every system message and the last user message — if those alone still exceed the limit, it falls back to Block rather than send a partial conversation. A successful truncation is tracked in three places: the X-Apinizer-AI-Truncated response header (number of dropped messages), the Guardrail Hits report (as truncated, alongside oversized), and — if AI Trace is active on the proxy — that request's trace detail.
This is a different concept from the Overflow Actions above: the proxy limit runs out over time, but the per-request size limit is about a single request that is too large.
The policy no longer has a Limit Source field. Apinizer checks the model's context window on the Routing tab of the AI proxy. See Context Window. A policy that used Model Context Window now follows the routing default. The gateway rejects a request that does not fit with HTTP 400 and error code ERR-311.
When the Limit Is Checked
The LLM Gateway protection limit runs in routing, after the provider and model are final and the model access list has let the caller through. It runs in the order of the policy chain and sees the caller that the authentication policy resolved. In tracing, the policy shows as skipped in the request pipeline with the reason Deferred to the model decision in routing, and as a second record in routing.
Status codes follow this order:
- The request body size limit (
413) and the stream compatibility checks (400) are checked first. - The token limit (
429) and the cache answer follow.
A very long prompt gets 413, not 429. Truncate Oldest Messages does not shorten a conversation that is above the body size limit (default 10 MB). That request gets 413.
When the policy shortens a conversation to the context window, Apinizer rebuilds the outgoing request from the shortened body. This includes failover legs, which use the same provider and model.
What the Caller Receives
When the overflow action is Block, the Token Rate Limit policy answers with HTTP 429, error code ERR-263, and a Retry-After header. The default body is:
{"error":{"code":429,"message":"AI token quota exceeded","type":"insufficient_quota"}}
You can change the message on the Error Messages screen. For a limit that comes from a Limit Plan, you can also change it on the plan's Error Message Customization tab.
Threshold Alarms
When usage crosses an alert threshold (Alert Thresholds (%), from 1 to 99 in ascending order; Limit Plans use 50, 80 and 90 by default), the request's traffic record is marked and the event appears in the Active alerts panel of Analytic → Dashboard (AI view). A request that passes because its overflow action is Alert Only is recorded the same way.
Crossing a threshold does not notify anyone on its own. To be notified, create an alert rule: either a rule on the Quota Usage & Alerts screen with AI Token and/or AI Cost selected in its counter types (an empty counter-type list covers the gateway request counters only), or an alarm with the AI Quota Soft-Cap Hit Count or AI Quota Alert-Only Count trigger.
How Limit Tracking Stays Accurate
Before a request is sent to the model, Apinizer estimates its token cost and reserves that amount against every applicable limit and window. Once the response finishes — including streamed responses — the reservation is adjusted to match the actual tokens consumed. This keeps concurrent requests from over-counting or under-counting against your limits.
Counters are kept in the shared cache service, so every worker uses the same counter and a worker restart does not reset them.
The Quota Reservation & Cache Behavior section holds the reservation settings: Default Reservation TTL (s), Cache Connection Timeout (s) and Fail Open on Cache Error. They apply to the LLM Gateway Limit of this policy only. A Limit Plan uses its own cache settings, or the settings of its assignment.
When the Counter Store Is Unreachable
If the cache service that holds the counters cannot be reached, the Fail Open on Cache Error switch decides what happens to the LLM Gateway Limit of this policy. A Limit Plan has its own setting, If the Counter Store Is Unreachable; see Advanced Settings.
- Off (the default, fail-closed): the request is rejected with HTTP
503and error code ERR-307 (default message: "AI token limit could not be checked: the counter store is unreachable (0)"). You can customize the message on the Error Messages screen or on this policy's error message tab. - On (fail-open): the request passes without a limit while the cache is down.
Limit Plans and This Policy
The policy limit and the limits of Limit Plans are independent. A request must pass both, and the first limit to fill stops it. The policy limit is the total of the proxy. The plans limit each caller.
Monitoring Limit Usage
Reports and Analytics shows what was consumed:
- Total tokens and Total cost for the selected period on Analytic → Dashboard (AI view), and the breakdown by person, group, application and model on the Usage Reports tab of Reports → Metrics (AI view)
- This month's cost and the cumulative cost chart on the Cost & Consumers tab of Reports → Metrics (AI view); a project has no AI budget of its own, so there is no budget line
- Active alerts on Analytic → Dashboard (AI view), raised as usage approaches a limit
For the headroom left in each counter, use Quota Usage & Alerts.
The Quota Usage & Alerts screen shows the usage of the counters, AI token and AI cost rows included. LLM Gateway rows on that screen come from this policy: one row for an undivided proxy limit, and one row for each per-value row that has a fixed value. A policy that is split by a variable counts each value of its default limit on its own, so the default limit has no single row.
The AI Token Usage Percentage alarm with the LLM Gateway scope reads the monthly token limit of this policy's LLM Gateway Protection. A limit that is split by a variable has no single total, so the alarm does not read it. With no monthly token limit the alarm does not fire. See Alarm.
The Limits tab of an identity shows only what its plans define, not usage.
Reading the same numbers through the Management API
What has been consumed — tokens, cost in US dollars, remaining headroom, percentage and reset time, per organization, consumer, application, API client or API proxy — is read through the Usage Reports API. It has five read-only GET endpoints, intended for chargeback exports and budget-alarm jobs.
Three properties of those figures are worth knowing before you build a report on them:
- A consumer, an organization and an LLM Gateway are read from the hourly usage rollup — the same table the AI reports use, so the numbers match what you see there. It is refreshed about once an hour and is kept independently of traffic-log retention, so a question about last quarter still answers correctly. Its grain is a whole UTC day, and the response says which days it actually measured.
- An application and an API client are read from the traffic log instead, because the rollup carries no application dimension. Their figures are therefore bounded by how long the traffic log is retained, and traffic recorded before 2026.09.2 cannot be attributed to a specific API client at all.
- The budget block is live, not historical. It always describes the windows that are open right now, regardless of the period you asked for, and states how fresh it is (the counters are synchronized about once a minute).
If a source cannot be read, the call still answers successfully and marks the result as partial, naming the environments it could not reach — it never reports an unread source as zero consumption.
Best Practices
- Start with Alert Only: make Alert Only the first overflow action of a new limit. You learn the real consumption before you reject traffic.
- Use a Limit Plan for per-caller limits: the limit in this policy is the total of the proxy.
- Limit expensive models separately: restrict expensive models more with Model limits.
- Watch the threshold alerts: follow them in the Active alerts panel of Analytic → Dashboard (AI view), and define an alert rule to be notified.
Troubleshooting
A limit was exceeded unexpectedly
- Check the Limit Plans of the caller. A plan limit may have filled before the proxy limit.
- Check the Model limits and Provider limits lines.
A limit change has no effect
- After you save the policy, only new requests use the new limit; requests in progress are not affected.
- Check that the policy is deployed to the LLM Gateway.
Next Steps
Set reusable limits for organizations, consumers, applications and API clients
Configure the failover chain used when a limit routes traffic elsewhere
Set per-model pricing that USD limits are calculated against
Monitor limit and cost usage over time