Semantic Cache
Semantic Cache must run after Prompt Template, Prompt Decorator and RAG Injection. The cache key is derived from the fully assembled prompt; if the cache runs earlier the key is computed from an incomplete text and genuinely different requests can share one answer.
If you save the policy list in an order that breaks this rule, Apinizer corrects the order when you save and tells you what was moved and why. This keeps the order you see on the Develop screen identical to the real execution order shown in tracing.
The cache lookup runs in routing, after the provider and model are final and the model access list has let the caller through. A caller that is not allowed to use the model gets 403 model_access_denied, not the cached answer. The lookup always sees the caller that the authentication policy resolved.
Matching Modes
A request that is literally identical to a previous one reuses its cached response.
An opt-in mode that uses the vector database to find requests that are near-identical in meaning, even when the wording differs, and reuses the cached answer for them too.
Both modes exist to avoid sending the same question to the model twice — lowering both response latency and cost.
Similarity Threshold
In vector-similarity mode the Similarity Threshold (default: 0.95) sets the minimum score a candidate needs to count as a hit. Lowering it yields more hits; raising it yields more precise ones.
Setting it to 0 genuinely disables the threshold — no candidate is rejected on score. But that does not mean "everything matches": three more filters run before the threshold.
- Scope isolation — under Proxy Isolated strategy (the default), a candidate is only accepted within the same model, the same API proxy and the same identity scope; under Global strategy the proxy is not in the key so candidates from different proxies can match. A higher-scoring record from a different identity scope is always rejected on purpose; otherwise one caller's response could be returned to another. When the LLM Gateway's Client "stream" field is set to Use the client's value, the response mode (streaming or single response) is part of the scope too: a response saved for a streaming request is never returned as a similarity hit to a request that asks for a single response, and the other way around. With Overwrite with the proxy setting the scope does not change.
- Candidate count — only the nearest few records are fetched from the vector database. On a large collection the correct same-scope record can fall outside that list; raise the Top-K Query Count when that happens.
- Entry lifetime — even on a vector match the response body is read from the exact cache entry. If that entry has expired via TTL the result is still a miss.
If you are not getting the hit you expect, check these in order — lowering the threshold further will not help.
Choosing What Gets Embedded
By default, the vector-similarity layer embeds only the request's last user message. Three settings widen that window — without touching the exact-match cache key, which always hashes the complete request body regardless of these settings:
- Trailing Message Count — how many trailing messages, scanning backward from the end of the conversation, to collect for the embedding.
1(default) reproduces the original behavior — only the last user message. - Include System / Assistant / Tool Messages — three independent toggles that widen which roles count toward that trailing window; user messages are always included regardless. All three default to off, matching the behavior before these settings existed.
Raising the trailing count (and including more roles) can improve the hit rate on multi-turn conversations where the useful context isn't just the latest message, at the cost of a slightly less precise match — a longer embedded text has more room to differ between two requests a user would still consider "the same question."
Anonymous Requests
The cache key always includes the model and the caller's identity; under Proxy Isolated strategy (the default) the API proxy is included too, so one tenant's response is never returned to another. On an API proxy without authentication a request carries no such identity, and the policy decides what to do:
| Option | Behaviour |
|---|---|
| Do not apply the policy (default) | The cache is bypassed and the request goes straight to the provider |
| Consume from a shared pool | All anonymous requests share a single cache bucket |
| Reject the request | Anonymous requests are turned away with HTTP 401 |
The default matches the behaviour before this setting existed. With the shared pool, tenant isolation is not expected — one caller's response can be returned to another; it suits only installations that deliberately allow anonymous access and are comfortable sharing responses across callers.
The same three options exist on the retry-storm guardrail — see Advanced Guardrails.
The Masking Gate
The semantic cache skips the write — for both unary and streaming responses — when the proxy has any of these active policies on the response side:
- A PII Mask policy with at least one rule that is not detect-only. Detect-only rules do not disable caching.
- A DLP Guard policy with at least one Block or Mask rule, or with file signature detection enabled — even when all explicit rules are Flag-only. Flag-only rules without file signature detection do not disable caching.
These checks apply to policies nested inside policy groups as well.
When the write is skipped, the response is not cached at all — neither the raw nor any processed version is stored. Response-side policies still run on what is sent to the caller.
This prevents a scenario where a cache entry recorded without masking would be served unmasked on a later hit.
Cache Key Strategy
The Cache Key Strategy field controls whether the API proxy is included in the cache key:
| Strategy | Key components | Isolation |
|---|---|---|
| Proxy Isolated (default) | Model + API proxy + consumer | Each proxy has its own cache bucket |
| Global | Model + consumer | For the same caller, the same answer can be shared across proxies using Global |
Under Global, two requests for the same question from the same caller arriving at two different proxies can share a cached answer — the proxy is not in the key. Use Global only in single-tenant installations where sharing responses across proxies is intentional and safe.
Under Global strategy, a response cached by one proxy can be returned to the same caller through a different proxy. Only use this in environments where cross-proxy response sharing is intentional and there is no tenant isolation requirement.
There is no project dimension in the key. Under Global strategy, a response cached on one proxy can be served through a proxy in a different project — caller identity is still always in the key, but project boundaries do not isolate cache entries.
Cache Lifetime
Each cached entry carries a configurable TTL (sec) (default: 3,600 seconds). Once it expires, it is automatically removed and is no longer returned as a cache hit. Setting TTL to 0 bypasses the cache entirely — every request goes to the provider.
Cleaning up the similarity layer
When similarity matching is enabled, every cached entry also has a counterpart in the vector database: the embedding of the request, plus a pointer back to the cached response. When the response entry expires under its TTL, that vector record does not disappear with it — what remains is a record pointing at a response that no longer exists.
Leftover records never produce a wrong answer (once the pointer leads nowhere, the request simply goes to the provider as usual). Left to accumulate, however, they cause two problems: as the collection grows they crowd the nearest-neighbour list and push valid matches out of it, lowering your hit rate; and embeddings derived from prompt text stay around with no retention policy of their own.
Apinizer handles this for you:
- PostgreSQL (pgvector) and Qdrant — a maintenance task runs on the Manager every day at 04:45 (in the Manager's time zone) and deletes expired semantic-cache records. It only ever targets records the semantic cache itself wrote; knowledge base (RAG) data sharing the same collection is never touched.
- Redis — records are given their own expiry the moment they are written, so Redis removes them on its own and no maintenance task is involved.
Set SETTING_AI_SEMANTIC_CACHE_VECTOR_PURGE_ENABLED to false in the Manager environment to turn the maintenance task off. Unless you have a reason to, leave it at its default (on).
The cleanup relies on an expiry stamp carried by each record. Vector records written by earlier versions do not carry that stamp, so the maintenance task does not see them and will not delete them. That is a deliberate safety choice: without the stamp there is no certainty about what wrote a record, so leaving it in place beats removing it. The set shrinks on its own over time — asking the same prompt again overwrites the record with a freshly stamped one. If you need the older records gone sooner, talk to your system administrator; where the collection is not shared for any other purpose, its records can be deleted wholesale. Delete the records, not the collection itself — see the note below.
Apinizer creates the vector collection the first time the policy needs it and does not check for it again afterwards. If the collection is dropped from outside Apinizer, it is not created again until the workers are restarted or the vector database connection is redeployed; until then similarity lookups and writes on it fail.
Respecting Cache-Control
By default, the semantic cache ignores any Cache-Control header a client sends and manages caching entirely through its own settings. Turning on Honor Cache-Control Header changes that:
| Client sends | Effect |
|---|---|
Cache-Control: no-store | The cache is bypassed entirely for that request — nothing is looked up, and the response isn't stored either |
Cache-Control: no-cache | The lookup is skipped, forcing a fresh call to the provider, but the response is still written to the cache afterward |
| Anything else, or the setting left off | No effect — unchanged behavior |
This gives an individual client a way to opt out of caching for a specific request — a debugging session, or a request that must always see the model's live answer — without an administrator having to disable the cache for everyone.
Monitoring Cache Efficiency
The Cache Efficiency report in Reports and Analytics shows how much the cache is saving you:
- Hit rate for exact and similarity matches
- Estimated cost savings, based on what those requests would otherwise have cost against your model pricing
Configuring with APIops
Semantic Cache can be added, updated, or removed on an API proxy through the same generic policy endpoint every policy type shares:
POST /apiops/projects/{projectName}/apiProxies/{apiProxyName}/policies/{policyName}/
PUT /apiops/projects/{projectName}/apiProxies/{apiProxyName}/policies/{policyName}/
Set policy.type to ai-semantic-cache. When vector-similarity matching is turned on (vectorEnabled: true), vectorDbName and embeddingProviderName are resolved server-side by name — the same name-based-reference rule every APIops resource follows, never by internal ID — so the same request body is portable across environments as long as the referenced connections exist there too.
See API Reference: Semantic Cache Policy for the full field list, request/response examples, and error codes.
Configuration Reference
| Setting | Default | Notes |
|---|---|---|
| TTL (sec) | 3,600 | 0 bypasses the cache — all requests go to the provider |
| Cache Connection Timeout (sec) | 3 | Minimum 1; on timeout the request goes to the provider |
| Max Cacheable Size (bytes) | 262,144 (256 KB) | 0 = no limit (not recommended); applies to both unary and streaming responses |
| Vector Cache Enabled | Off | Turns on vector-similarity matching. The fields below apply only when it is on |
| Vector DB Connection | — | Required. The vector database connection that stores the request embeddings |
| Embedding Provider | — | Required. Only providers that support embeddings are listed |
| Embedding Model ID | text-embedding-3-small | Pick from the enabled embedding models of the selected provider, or type a model ID |
| Embedding Dimension | 1536 | Must match the dimension the embedding model produces |
| Collection Name | — | Required. The collection that holds the request embeddings |
| Similarity Threshold | 0.95 | Minimum score for a vector match to count as a hit |
| Top-K Query Count | 1 | Number of candidates fetched from the vector database for scope filtering and scoring |
| Honor Cache-Control Header | Off | See Respecting Cache-Control |