Tracing and Replay
Overview
AI Gateway builds an end-to-end chain of requests (trace group / DAG) in real time, enabling two capabilities:
- Trace Detail — Inspect every step of a request (policies, routing decision, provider request and response) in a single screen
- Replay — Re-execute a captured trace as a new, real request
When the effective data retention profile is anything other than Standard (No Payload or No Persist), starting a trace is rejected — not even an authorized user can turn on tracing for that proxy/project/installation. Stopping a trace that is already running is not affected by this restriction.
Turning Live Trace On
Tracing is off by default. To turn it on for an LLM Gateway:
Open the LLM Gateway and go to its Trace tab. The gateway must be deployed to at least one environment.
Select the environment to trace. Optionally pick a filter (custom query) to trace only matching traffic; the default is All Traffic.
Click Start. Live Trace stops automatically after 5 minutes, or earlier when you click Stop.
While Live Trace is on, Apinizer also captures the raw request, response and stream chunks of the traced requests.
How the Trace Group (DAG) Is Built
When Live Trace is on, a unique trace ID is generated at the start of each request and carried through the entire lifecycle of that request:
- The main request and any sub-calls (for example, an agent tool calling another LLM Gateway, or an A2A task chain) are correlated under the same trace group ID
- Each sub-step references a parent trace ID, establishing a parent-child relationship between steps
Access and Retention
Trace data is sensitive — it can contain full request and response bodies. Viewing it requires the AI analytics view permission, scoped to your project, so only people with legitimate access to a project's AI traffic can inspect it. Trace records are also retained for a limited period and purged automatically rather than kept indefinitely.
Trace Detail
In the LLM Gateway's Trace tab, open a request to see its detail:
- The request flow from the client through each policy to the provider, with the duration and status (success/error/skipped) of every step
- The request and response bodies before and after each policy
- The routing decision, the request sent to the provider, and the provider's response
A Policy's Internal Steps
AI policies that reach the cache (semantic cache, token rate limit) do real work in two separate lifecycle passes: the request-phase decision (cache lookup, token reservation) and a stream-end pass that runs once the response is complete (writing the response to the cache, reconciling the reservation against the tokens actually consumed).
Both passes are genuine executions, so each gets its own node in the flow — the trace shows what really ran, not a simplified summary. The second node carries a ↺ badge; hovering it names the work done in that pass rather than the phase itself ("Stream end — response written to the cache", "Stream end — token reservation settled"). Clicking the node shows the same label in the detail panel next to a "Why a second record?" line, so it reads as a second phase rather than a duplicated policy.
The token rate limit (proxy protection limit), the semantic cache lookup, and the loop guard run in routing, after the provider and model are final and the model access check has passed. In the request pipeline these three policies show as skipped with the reason Deferred to the model decision in routing. The timing view also marks that row as skipped. The real execution appears as a second record in routing. This applies to LLM Gateways only; MCP and A2A proxies are not affected.
Whether a second node appears depends on the request:
| Situation | Second node |
|---|---|
| Semantic cache MISS | Yes — the response is written to the cache at stream end |
| Semantic cache HIT | No — nothing needs re-caching, so no stream-end pass is dispatched |
| Token rate limit | Yes, one per configured policy — the reservation is reconciled or released |
The Execution Details button on the top bar opens the cache calls behind the node: the operation (RESERVE / RECONCILE / RELEASE / GET / PUT), the bucket, the cache key, the result and the duration — the same behaviour as the API Call policy's detail window on the API proxy trace screen.
Why the Trace Order Can Differ From Develop
The trace shows the actual execution order, and for LLM Gateways that is not always the order you arranged on the Develop canvas. Before running the request pipeline the gateway applies a defensive reordering so that a misconfigured order cannot silently break correctness or a guardrail:
- Prompt Template runs before Prompt Decorator — the Decorator injects into the
messages[]array the Template produces. - Prompt Template / Prompt Decorator run before RAG Injection and every guard/cache policy — those need to see the fully assembled prompt, not the raw client body.
- RAG Injection runs before Prompt Guard, Context Integrity, DLP Guard and Semantic Cache — content RAG injects must be scanned by the guards and must be part of the cache key, otherwise a guard is defeated and the cache can be poisoned.
So if a node appears earlier in the trace than in Develop, the request was not mishandled — the gateway corrected the order. The Develop screen warns you whenever one of these rules is violated: when you reorder policies by drag and drop, and when the flow screen loads with that order (including the return trip after adding a new policy). Fix the configuration and the warning stops, making both screens agree.
For the semantic cache, that window also shows the cached conversation alongside the decision itself (hit/miss, match source — exact or vector, similarity, cache key, TTL, model): the request's prompt text, plus the response body that was served from the cache on a hit, or written to it on the write phase. "Why did this request come from cache?" and "what response was stored?" are then answered in one window, without comparing raw request/response bodies along the flow.
Replay
Replay re-executes a captured trace: the gateway sends the trace's provider-bound request again through the same policy chain and records the result as a new trace. Use it to reproduce a customer-reported issue or to compare behavior after a configuration change.
Replay is a live request, not a playback of the stored response:
- A real call is made to the LLM provider, so tokens, cost and quota are consumed and guardrails run again.
- If the original request has a side effect (for example, a tool call that writes to an external system), that side effect can happen again. Use Replay for read-only or analysis-oriented requests.
- You must choose the environment whose gateway runs the request. Only one gateway instance of that environment runs it.
- Only traces recorded while Live Trace was on can be replayed, because only they keep the request body. Replay of any other record is rejected.
- Replay needs the AI Development – Manage permission on the project. The AI analytics view permission that lets you read traces is not enough.
Replay is available through the Manager REST API (POST /api/ai-trace/{id}/replay?projectId=...&environmentName=...). The Trace tab does not have a Replay button.
Metrics and Traces: OpenTelemetry and Prometheus
Beyond the trace views in the UI, AI Gateway usage signals (tokens, cost, latency, guardrail outcomes) reach your own observability stack through two complementary paths:
- Prometheus — the gateway's existing
apinizer_ai_*counters/timers (provider, model, project, status tags) on its Prometheus endpoint, unchanged and always on regardless of the setting below. - Native OpenTelemetry (OTLP) export — an optional, agent-free product feature: W3C
traceparentpropagation plus a retrospective span tree (SERVER + per-attempt CLIENT + request/response spans) for all traffic, and a GenAI semantic-convention attribute/metric layer (gen_ai.*,apinizer.ai.*) for AI traffic specifically — exported straight from the product to your OTLP collector, no javaagent required. Content capture (prompts/completions on spans) is off by default.
See OpenTelemetry for the full attribute/metric reference, setup steps, and a downloadable Grafana dashboard.
Next Steps
See how tool-call sequences show up in the trace chain
See how agent tasks are traced across the chain
Review performance and latency metrics
Correlate guardrail triggers within the trace chain
Configure native OTLP export and import the Grafana dashboard