Semantic Cache
Semantic Cache must run after Prompt Template, Prompt Decorator and RAG Injection. The cache key is derived from the fully assembled prompt; if the cache runs earlier the key is computed from an incomplete text and genuinely different requests can share one answer.
If you save the policy list in an order that breaks this rule, Apinizer corrects the order when you save and tells you what was moved and why. This keeps the order you see on the Develop screen identical to the real execution order shown in tracing.
Matching Modes
A request that is literally identical to a previous one reuses its cached response.
An opt-in mode that uses the vector database to find requests that are near-identical in meaning, even when the wording differs, and reuses the cached answer for them too.
Both modes exist to avoid sending the same question to the model twice — lowering both response latency and cost.
Similarity Threshold
In vector-similarity mode the similarity threshold sets the minimum score a candidate needs to count as a hit. Lowering it yields more hits; raising it yields more precise ones.
Setting it to 0 genuinely disables the threshold — no candidate is rejected on score. But that does not mean "everything matches": three more filters run before the threshold.
- Scope isolation — a candidate is only accepted within the same model, the same API proxy and the same identity scope. A higher-scoring record from another scope is rejected on purpose; otherwise one caller's response could be returned to another.
- Candidate count — only the nearest few records are fetched from the vector database. On a large collection the correct same-scope record can fall outside that list; raise the nearest-neighbour count when that happens.
- Entry lifetime — even on a vector match the response body is read from the exact cache entry. If that entry has expired via TTL the result is still a miss.
If you are not getting the hit you expect, check these in order — lowering the threshold further will not help.
Choosing What Gets Embedded
By default, the vector-similarity layer embeds only the request's last user message. Three settings widen that window — without touching the exact-match cache key, which always hashes the complete request body regardless of these settings:
- Trailing Message Count — how many trailing messages, scanning backward from the end of the conversation, to collect for the embedding.
1(default) reproduces the original behavior — only the last user message. - Include System / Assistant / Tool Messages — three independent toggles that widen which roles count toward that trailing window; user messages are always included regardless. All three default to off, matching the behavior before these settings existed.
Raising the trailing count (and including more roles) can improve the hit rate on multi-turn conversations where the useful context isn't just the latest message, at the cost of a slightly less precise match — a longer embedded text has more room to differ between two requests a user would still consider "the same question."
Anonymous Requests
The cache key is built from the model, the API proxy and the caller's identity, so one tenant's response is never returned to another. On an API proxy without authentication a request carries no such identity, and the policy decides what to do:
| Option | Behaviour |
|---|---|
| Do not apply the policy (default) | The cache is bypassed and the request goes straight to the provider |
| Consume from a shared pool | All anonymous requests share a single cache bucket |
| Reject the request | Anonymous requests are turned away with HTTP 401 |
The default matches the behaviour before this setting existed. With the shared pool, tenant isolation is not expected — one caller's response can be returned to another; it suits only installations that deliberately allow anonymous access and are comfortable sharing responses across callers.
The same three options exist on the retry-storm guardrail — see Advanced Guardrails.
The Masking Gate
If a response requires personal-data masking or data-loss protection before it is returned, Apinizer never stores the raw response in the cache — only the masked version can be cached. This closes off a scenario where masking could otherwise be bypassed simply by hitting a cache entry recorded before masking ran.
Cache Lifetime
Each cached entry carries a configurable time-to-live (TTL). Once it expires, it is automatically removed and is no longer returned as a cache hit.
Respecting Cache-Control
By default, the semantic cache ignores any Cache-Control header a client sends and manages caching entirely through its own settings. Turning on Honor Cache-Control Header changes that:
| Client sends | Effect |
|---|---|
Cache-Control: no-store | The cache is bypassed entirely for that request — nothing is looked up, and the response isn't stored either |
Cache-Control: no-cache | The lookup is skipped, forcing a fresh call to the provider, but the response is still written to the cache afterward |
| Anything else, or the setting left off | No effect — unchanged behavior |
This gives an individual client a way to opt out of caching for a specific request — a debugging session, or a request that must always see the model's live answer — without an administrator having to disable the cache for everyone.
Monitoring Cache Efficiency
The Cache Efficiency report in Reports and Analytics shows how much the cache is saving you:
- Hit rate for exact and similarity matches
- Estimated cost savings, based on what those requests would otherwise have cost against your model pricing