Manual ILM Policy and Template Creation

Warning

If log management will be done through Apinizer, these operations do not need to be performed, these operations are done automatically by Apinizer.

Variables

Dynamic values and their descriptions in the requests are shown in the table below.

Variable	Description
`<ELASTICSEARCH_IP_ADDRESS>`	Host information of the Elasticsearch cluster.
`<ELASTICSEARCH_PORT>`	Port information of the Elasticsearch cluster.
`<INDEX_KEY>`	Unique identifier for the index.
`<INDEX_PATTERN_NAME>`	Index pattern name. It is the index name followed by `` (e.g.: apinizer-log-apiproxy-default).
`<ILM_POLICY_NAME>`	ILM policy name (e.g.: apinizer-log-ilm-policy-default).
`<NUMBER_OF_SHARDS>`	Number of shards for the index (Apinizer default: 1).
`<NUMBER_OF_REPLICAS>`	Number of replicas for the index (Apinizer default: 0).
`<REFRESH_INTERVAL>`	Index refresh interval (Apinizer default: 5s).
`<LOAD_FIXED_BITSET_FILTERS_EAGERLY>`	The `index.load_fixed_bitset_filters_eagerly` setting (Apinizer default: true).

Creating ILM Policies

Creating ILM Policy Request Named apinizer-log-ilm-policy-default

curl -XPUT "http://<ELASTICSEARCH_IP_ADDRESS>:<ELASTICSEARCH_PORT>/_ilm/policy/apinizer-log-ilm-policy-<INDEX_KEY>" \
  -H 'Content-Type: application/json' \
  --data-binary @- <<'EOF'
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "30d",
            "max_size": "30gb",
            "max_docs": 15000000
          },
          "set_priority": {
            "priority": null
          }
        }
      },
      "warm": {
        "actions": {
          "readonly": {},
          "allocate": {
            "number_of_replicas": 0,
            "include": {},
            "exclude": {}
          },
          "shrink": {
            "number_of_shards": 1
          },
          "forcemerge": {
            "max_num_segments": 1
          },
          "set_priority": {
            "priority": null
          }
        }
      },
      "cold": {
        "min_age": "90d",
        "actions": {
          "set_priority": {
            "priority": null
          }
        }
      }
    }
  }
}
EOF

Creating Template

Creating Template Request Named apinizer-log-apiproxy-template-default

curl -X PUT "http://<ELASTICSEARCH_IP_ADDRESS>:<ELASTICSEARCH_PORT>/_index_template/apinizer-log-apiproxy-template-<INDEX_KEY>" \
  -H 'Content-Type: application/json' \
  --data-binary @- <<'EOF'
{
  "index_patterns": ["<INDEX_PATTERN_NAME>"],
  "data_stream": {},
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "<ILM_POLICY_NAME>"
        },
        "number_of_shards": "<NUMBER_OF_SHARDS>",
        "number_of_replicas": "<NUMBER_OF_REPLICAS>",
        "refresh_interval": "<REFRESH_INTERVAL>",
        "load_fixed_bitset_filters_eagerly": "<LOAD_FIXED_BITSET_FILTERS_EAGERLY>"
      }
    },
    "mappings": {
      "properties": {
        "aprp": {
          "type": "keyword"
        },
        "fcrb": {
          "type": "text"
        },
        "hr1ru": {
          "type": "keyword"
        },
        "aci": {
          "type": "keyword"
        },
        "subi": {
          "type": "keyword"
        },
        "subp": {
          "type": "keyword"
        },
        "subpl": {
          "type": "keyword"
        },
        "subo": {
          "type": "keyword"
        },
        "ack": {
          "type": "keyword"
        },
        "cot": {
          "type": "keyword"
        },
        "coi": {
          "type": "keyword"
        },
        "coo": {
          "type": "keyword"
        },
        "usub": {
          "type": "keyword",
          "ignore_above": 256
        },
        "uids": {
          "type": "keyword",
          "ignore_above": 256
        },
        "ocid": {
          "type": "keyword",
          "ignore_above": 256
        },
        "mcprc": {
          "type": "keyword"
        },
        "mcpap": {
          "type": "keyword"
        },
        "tapt": {
          "type": "integer"
        },
        "trqpt": {
          "type": "integer"
        },
        "trspt": {
          "type": "integer"
        },
        "hr1rp": {
          "type": "keyword"
        },
        "uok": {
          "ignore_above": 50,
          "type": "keyword"
        },
        "sr2t": {
          "type": "integer"
        },
        "fcrp": {
          "type": "nested",
          "properties": {
            "k": {
              "type": "keyword"
            },
            "v": {
              "type": "keyword"
            }
          }
        },
        "api": {
          "type": "keyword"
        },
        "appi": {
          "type": "keyword"
        },
        "appin": {
          "type": "keyword"
        },
        "fcrh": {
          "type": "nested",
          "properties": {
            "k": {
              "type": "keyword"
            },
            "v": {
              "type": "keyword"
            }
          }
        },
        "apn": {
          "type": "keyword"
        },
        "pet": {
          "ignore_above": 75,
          "type": "keyword"
        },
        "apr": {
          "type": "short"
        },
        "tbah": {
          "type": "nested",
          "properties": {
            "k": {
              "type": "keyword"
            },
            "v": {
              "type": "keyword"
            }
          }
        },
        "ei": {
          "type": "keyword"
        },
        "apmi": {
          "type": "keyword"
        },
        "hr1qs": {
          "type": "keyword"
        },
        "tbap": {
          "type": "nested",
          "properties": {
            "k": {
              "type": "keyword"
            },
            "v": {
              "type": "keyword"
            }
          }
        },
        "ra": {
          "type": "keyword"
        },
        "et": {
          "ignore_above": 75,
          "type": "keyword"
        },
        "tba": {
          "type": "text"
        },
        "hr1rh": {
          "type": "keyword"
        },
        "trt": {
          "type": "integer"
        },
        "rrc": {
          "type": "integer"
        },
        "rfc": {
          "type": "integer"
        },
        "icr": {
          "type": "boolean"
        },
        "crs": {
          "type": "keyword"
        },
        "mc": {
          "type": "integer"
        },
        "msc": {
          "type": "integer"
        },
        "mfc": {
          "type": "integer"
        },
        "mtd": {
          "type": "long"
        },
        "ma": {
          "type": "keyword"
        },
        "sr1t": {
          "type": "integer"
        },
        "hr1ra": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword"
            }
          }
        },
        "loc": {
          "type": "geo_point"
        },
        "xff": {
          "type": "keyword"
        },
        "apmn": {
          "type": "keyword"
        },
        "hr1hm": {
          "ignore_above": 50,
          "type": "keyword"
        },
        "rt": {
          "ignore_above": 7,
          "type": "keyword"
        },
        "hr1lp": {
          "type": "keyword"
        },
        "hr1ln": {
          "type": "keyword"
        },
        "hr1pi": {
          "type": "keyword"
        },
        "sc": {
          "type": "short"
        },
        "fbarh": {
          "type": "nested",
          "properties": {
            "k": {
              "type": "keyword"
            },
            "v": {
              "type": "keyword"
            }
          }
        },
        "tcb": {
          "type": "text"
        },
        "hr1rsi": {
          "type": "keyword"
        },
        "tch": {
          "type": "nested",
          "properties": {
            "k": {
              "type": "keyword"
            },
            "v": {
              "type": "keyword"
            }
          }
        },
        "eat": {
          "type": "keyword"
        },
        "hr1r1u": {
          "type": "keyword"
        },
        "hr1p": {
          "type": "keyword"
        },
        "hr1s": {
          "type": "keyword"
        },
        "fbarb": {
          "type": "text"
        },
        "@timestamp": {
          "format": "yyyy-MM-dd'T'HH:mm:ss.S'Z'||yyyy-MM-dd'T'HH:mm:ss.SS'Z'||yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",
          "type": "date"
        },
        "ch": {
          "type": "boolean"
        },
        "hr1ce": {
          "type": "keyword"
        },
        "hr1sp": {
          "type": "keyword"
        },
        "hr1sn": {
          "type": "keyword"
        },
        "hr1la": {
          "type": "keyword"
        },
        "hr1ct": {
          "ignore_above": 50,
          "type": "keyword"
        },
        "pi": {
          "type": "keyword"
        },
        "hr1cp": {
          "type": "keyword"
        },
        "hr1cl": {
          "type": "keyword"
        },
        "aip": {
          "type": "keyword"
        },
        "aipid": {
          "type": "keyword"
        },
        "aivs": {
          "type": "keyword"
        },
        "aim": {
          "type": "keyword"
        },
        "aiti": {
          "type": "long"
        },
        "aito": {
          "type": "long"
        },
        "aitc": {
          "type": "long"
        },
        "aicm": {
          "type": "long"
        },
        "aicim": {
          "type": "long"
        },
        "aicom": {
          "type": "long"
        },
        "aiccm": {
          "type": "long"
        },
        "aittft": {
          "type": "long"
        },
        "aitpot": {
          "type": "long"
        },
        "ailat": {
          "type": "long"
        },
        "aisch": {
          "type": "boolean"
        },
        "aigh": {
          "type": "keyword"
        },
        "aigc": {
          "type": "keyword"
        },
        "aigv": {
          "type": "keyword"
        },
        "aircl": {
          "type": "keyword"
        },
        "aircs": {
          "type": "keyword"
        },
        "aircms": {
          "type": "long"
        },
        "aircti": {
          "type": "long"
        },
        "aircto": {
          "type": "long"
        },
        "airccm": {
          "type": "long"
        },
        "aipd": {
          "type": "boolean"
        },
        "aifr": {
          "type": "keyword"
        },
        "aistr": {
          "type": "boolean"
        },
        "aivk": {
          "type": "keyword"
        },
        "aitid": {
          "type": "keyword"
        },
        "aiff": {
          "type": "keyword"
        },
        "aiqsc": {
          "type": "integer"
        },
        "aiqao": {
          "type": "boolean"
        },
        "aidt": {
          "type": "keyword"
        },
        "aigtms": {
          "type": "long"
        },
        "aiinf": {
          "type": "long"
        },
        "aiovh": {
          "type": "long"
        },
        "mcptn": {
          "type": "keyword"
        },
        "mcpjrid": {
          "type": "keyword"
        },
        "mcptno": {
          "ignore_above": 256,
          "type": "keyword"
        },
        "mcptco": {
          "type": "integer"
        },
        "a2atk": {
          "type": "keyword"
        },
        "a2act": {
          "type": "keyword"
        },
        "a2ats": {
          "type": "keyword"
        },
        "a2asno": {
          "ignore_above": 256,
          "type": "keyword"
        },
        "a2aaco": {
          "type": "integer"
        },
        "afr": { "type": "keyword", "ignore_above": 256 },
        "rfr": { "type": "keyword" },
        "rco": { "type": "keyword" },
        "rec": { "type": "keyword" },
        "red": { "type": "keyword", "ignore_above": 256 },
        "rdsel": { "type": "long" },
        "rddns": { "type": "long" },
        "rdtcp": { "type": "long" },
        "rdtls": { "type": "long" },
        "rdttfb": { "type": "long" },
        "rdbrd": { "type": "long" },
        "rdpw": { "type": "long" },
        "usc": { "type": "short" },
        "uip": { "type": "keyword" },
        "crf": { "type": "boolean" },
        "rtr": { "type": "boolean" },
        "gw": { "type": "keyword" },
        "pll": { "type": "short" },
        "plp": { "type": "short" },
        "pla": { "type": "short" },
        "plm": { "type": "short" },
        "cct": { "type": "integer" },
        "crt": { "type": "integer" },
        "ccrt": { "type": "integer" },
        "cwms": { "type": "integer" },
        "cr": { "type": "keyword" },
        "cl": { "type": "keyword" },
        "pxt": {
          "type": "nested",
          "properties": {
            "o": { "type": "integer" },
            "t": { "type": "keyword" },
            "d": { "type": "integer" },
            "f": { "type": "keyword", "ignore_above": 1 }
          }
        }
      }
    }
  }
}
EOF

Template Data Structure Table

Field Name	Data Stored in Field/Description	Java Mapping Type	Elastic Search Mapping Type
`@timestamp`	Time Stamp	Instant	date format: yyyy-MM-dd'T'HH:mm:ss.S'Z' yyyy-MM-dd'T'HH:mm:ss.SS'Z' yyyy-MM-dd'T'HH:mm:ss.SSS'Z'
`aci`	Apinizer Correlation ID	String	keyword
`ei`	Environment ID	String	keyword
`pi`	Project ID	String	keyword
`appi`	Application ID	String	keyword
`appin`	Application Name	String	keyword
`api`	API Proxy ID	String	keyword
`apn`	API Proxy Name	String	keyword
`eat`	API Type	Enum	keyword
`aprp`	API Proxy Relative Path	String	keyword
`apmi`	API Proxy Method ID	String	keyword
`apmn`	API Proxy Method Name	String	keyword
`xff`	X Forwarded For	String	keyword
`hr1ra`	HTTP Request Remote Address	String	text, keyword
`hr1hm`	HTTP Request HTTP Method	String	keyword ignore_above: 50
`hr1ct`	HTTP Request Content Type	String	keyword ignore_above: 50
`hr1pi`	HTTP Request Path Info	String	keyword
`uok`	Username or Key	String	keyword ignore_above: 50
`ack`	API Client Key	String	keyword
`cot`	Client Owner Type	String	keyword
`coi`	Client Owner Id	String	keyword
`coo`	Client Organization Id	String	keyword
`hr1cp`	HTTP Request Context Path	String	keyword
`hr1qs`	HTTP Request Query String	String	keyword
`hr1ru`	HTTP Request Remote User	String	keyword
`hr1rsi`	HTTP Request Requested SessionID	String	keyword
`hr1r1u`	HTTP Request Request URI	String	keyword
`hr1ce`	HTTP Request Character Encoding	String	keyword
`hr1cl`	HTTP Request Content Length	String	keyword
`hr1p`	HTTP Request Protocol	String	keyword
`hr1s`	HTTP Request Scheme	String	keyword
`hr1sn`	HTTP Request Server Name	String	keyword
`hr1sp`	HTTP Request Server Port	String	keyword
`loc`	Location	GeoLocation{ Double lat, Double lon }	geo_point
`hr1rh`	HTTP Request Remote Host	String	keyword
`hr1rp`	HTTP Request Remote Port	String	keyword
`hr1ln`	HTTP Request Local Name	String	keyword
`hr1la`	HTTP Request Local Addr	String	keyword
`hr1lp`	HTTP Request Local Port	String	keyword
`trqpt`	Time Request Pipeline Total	Integer	integer
`trspt`	Time Response Pipeline Total	Integer	integer
`rrc`	Routing Retry Count	Integer	integer
`rfc`	Routing Failover Count	Integer	integer
`trt`	Time Routing Total	Integer	integer
`tapt`	Time API Proxy Total	Integer	integer
`ch`	Cache Hit	Boolean	boolean
`sr1t`	Size Request Total	Integer	integer
`sr2t`	Size Response Total	Integer	integer
`sc`	Status Code	int	short
`rt`	Result Type	Enum	keyword ignore_above: 7
`et`	Error Type	Enum	keyword ignore_above: 75
`pet`	Parent Error Type	Enum	keyword ignore_above: 75
`afr`	Authentication Failure Reason	Enum	keyword ignore_above: 256
`fcrp`	From Client Read Only Parameter	Map<String,String>	nested k: keyword v: keyword
`fcrh`	From Client Read Only Header	Map<String,String>	nested k: keyword v: keyword
`fcrb`	From Client Read Only Body	String	text
`tbap`	To Backend API Parameter	Map<String,String>	nested k: keyword v: keyword
`tbah`	To Backend API Header	Map<String,String>	nested k: keyword v: keyword
`tba`	To Backend API Body	String	text
`fbarh`	From Backend API Read Only Header	Map<String,String>	nested k: keyword v: keyword
`fbarb`	From Backend API Read Only Body	String	text
`tch`	To Client Header	Map<String,String>	nested k: keyword v: keyword
`tcb`	To Client Body	String	text
`pxt`	Policy Execution Times (written only when the option is enabled on the log target)	List<PolicyExecutionTiming>	nested o: integer t: keyword d: integer f: keyword ignore_above: 1
`usub`	Stable user ID: the `sub` claim of an IdP token, or the client itself for a client_credentials token	String	keyword ignore_above: 256
`uids`	Token issuer (`iss`) of an IdP token; `apinizer` for a token issued by Apinizer	String	keyword ignore_above: 256
`ocid`	OAuth client the token was issued to (`azp`, otherwise `client_id`)	String	keyword ignore_above: 256
`subi`	Subscription ID	String	keyword
`subp`	Subscription product ID	String	keyword
`subpl`	Subscription plan ID	String	keyword
`subo`	Consumer organization ID of the subscription	String	keyword
`apr`	API Proxy Revision	Integer	short
`ra`	Routing Address	String	keyword
`icr`	Whether the request was routed as a canary request	Boolean	boolean
`crs`	Canary routing status	Enum	keyword
`mc`	Number of mirror requests sent	Integer	integer
`msc`	Number of successful mirror requests	Integer	integer
`mfc`	Number of failed mirror requests	Integer	integer
`mtd`	Total duration of the mirror requests	Long	long
`ma`	Mirror addresses	List<String>	keyword
`rfr`	Routing failure reason	String	keyword
`rco`	Confidence of the failure reason (HIGH, MEDIUM, LOW)	String	keyword
`rec`	Exception class name	String	keyword
`red`	Sanitized and truncated exception message	String	keyword ignore_above: 256
`rdsel`	Upstream selection time (ns)	Long	long
`rddns`	DNS resolve time (ns)	Long	long
`rdtcp`	TCP connect time (ns)	Long	long
`rdtls`	TLS handshake time (ns)	Long	long
`rdttfb`	Time to first byte (ns); -1 when there was no response	Long	long
`rdbrd`	Response body read time (ns)	Long	long
`rdpw`	Time spent waiting for a connection from the pool (ns)	Long	long
`usc`	Raw status code returned by the backend (separate from the status returned to the client)	Short	short
`uip`	Upstream ip:port actually connected to	String	keyword
`crf`	Whether a pooled connection was reused	Boolean	boolean
`rtr`	Whether the request reached a backend that responded	Boolean	boolean
`gw`	Worker pod/host that processed the request	String	keyword
`pll`	Connections leased in the pool	Short	short
`plp`	Pending connection requests in the pool	Short	short
`pla`	Available connections in the pool	Short	short
`plm`	Maximum connections in the pool	Short	short
`cct`	Configured connect timeout (ms)	Integer	integer
`crt`	Configured read timeout (ms)	Integer	integer
`ccrt`	Configured connection-request (pool lease) timeout (ms)	Integer	integer
`cwms`	Time spent writing the response to the client (ms); -1 for streaming responses	Integer	integer
`mcptn`	Name of the MCP tool called by the request	String	keyword
`mcpjrid`	JSON-RPC correlation ID	String	keyword
`mcprc`	Outcome class of a `tools/call`: OK, TOOL_ERROR, JSONRPC_ERROR, TRANSPORT, POLICY_REJECT	String	keyword
`mcpap`	Approval state of a `tools/call` held for approval	String	keyword
`mcptno`	Distinct names of the outbound MCP tools called during the request, comma-separated (cut at 256 characters, marked with `,…`)	String	keyword ignore_above: 256
`mcptco`	Total number of outbound tool calls during the request	Integer	integer
`a2atk`	A2A task ID	String	keyword
`a2act`	A2A context ID (multi-turn correlation)	String	keyword
`a2ats`	Latest A2A task state, for example COMPLETED or FAILED	String	keyword
`a2asno`	Distinct names of the outbound agent skills called during the request, comma-separated (cut at 256 characters)	String	keyword ignore_above: 256
`a2aaco`	Total number of outbound agent calls during the request	Integer	integer
`aip`	LLM provider type (for example openai, anthropic)	String	keyword
`aipid`	ID of the LLM provider record that served the request	String	keyword
`aim`	Model name	String	keyword
`aidt`	Deployment type: CLOUD, ON_PREM, or OTHER	String	keyword
`aiti`	Prompt (input) tokens	Long	long
`aito`	Completion (output) tokens	Long	long
`aitc`	Cached prompt tokens	Long	long
`aicm`	Cost in micro USD (cost × 1,000,000)	Long	long
`aicim`	Input token cost in micro USD	Long	long
`aicom`	Output token cost in micro USD	Long	long
`aiccm`	Cached token cost in micro USD	Long	long
`aittft`	Time to first token (ms)	Long	long
`aitpot`	Average time per output token (ms)	Long	long
`ailat`	Total latency (ms)	Long	long
`aigtms`	Total time of the guardrail stage (ms)	Long	long
`aiinf`	Upstream LLM HTTP latency (ms)	Long	long
`aiovh`	Gateway overhead: total minus guardrail minus inference (ms)	Long	long
`aisch`	Whether the semantic cache returned a hit	Boolean	boolean
`aigh`	Guard that was triggered	String	keyword
`aigc`	Category of the triggered rule	String	keyword
`aigv`	Verdict of the external guardrail: SAFE, UNSAFE, TIMEOUT, ERROR, or SKIP	String	keyword
`aipd`	Whether PII was detected	Boolean	boolean
`aifr`	Finish reason (stop, length, tool_use, content_filter)	String	keyword
`aistr`	Whether the response was streamed	Boolean	boolean
`aivk`	Identity of the authenticated caller (the principal, not a record ID)	String	keyword
`aitid`	AI trace record reference	String	keyword
`aiff`	Primary provider that failed over	String	keyword
`aiqsc`	Soft-cap percentage that was crossed (for example 80)	Integer	integer
`aiqao`	Whether an ALERT_ONLY overflow fired and the request proceeded	Boolean	boolean
`aircl`	Label returned by the route classifier	String	keyword
`aircs`	Route classifier outcome: MATCHED, NONE, UNKNOWN_LABEL, TIMEOUT, ERROR, TRUNCATED, BACKOFF, NO_PROVIDER, or NO_INPUT	String	keyword
`aircms`	Route classifier call latency (ms)	Long	long
`aircti`	Route classifier prompt tokens	Long	long
`aircto`	Route classifier completion tokens	Long	long
`airccm`	Route classifier cost in micro USD (not included in `aicm`)	Long	long
`aivs`	Why the request continued without the vector layer (RAG, semantic cache, or Topic Guard)	String	keyword
`cr`	Pipeline position where the Log policy captured the record (FROM_CLIENT, TO_BACKEND, FROM_BACKEND, TO_CLIENT); written only in Log policy records	Enum	keyword
`cl`	Assignment level of the Log policy (API_PROXY_GROUP, API_PROXY, API_PROXY_METHOD); written only in Log policy records	Enum	keyword

Info

You can review the data example created in this data structure on the API Traffic Log Record Data Structure page.

Index Creation and Querying

When creating indices or querying with data stream, requests are sent according to the value in the index_patterns metadata within the template.

Index Creation

curl -XPOST "http://<ELASTICSEARCH_IP_ADDRESS>:<ELASTICSEARCH_PORT>/apinizer-log-apiproxy-<INDEX_KEY>/_doc" -H 'Content-Type: application/json' -d'{"@timestamp": "2023-05-31T11:40:36.376Z"}'

Running Query

curl -XGET "http://<ELASTICSEARCH_IP_ADDRESS>:<ELASTICSEARCH_PORT>/apinizer-log-apiproxy-<INDEX_KEY>/_search" \
  -H 'Content-Type: application/json' \
  --data-binary @- <<'EOF'
{
  "from": 0,
  "size": 10,
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "@timestamp": {
              "from": "now-10m/m",
              "to": "now/m",
              "include_lower": true,
              "include_upper": true
            }
          }
        }
      ],
      "adjust_pure_negative": true,
      "boost": 1
    }
  }, 
  "sort": [
    {
      "@timestamp": {
        "order": "desc"
      }
    }
  ]
}
EOF

Info

The templates on this page are for the API traffic log. The index templates for SIEM destinations are different. See Elasticsearch Index Templates for SIEM Destinations.

Variables​

Creating ILM Policies​

Creating ILM Policy Request Named apinizer-log-ilm-policy-default​

Creating Template​

Creating Template Request Named apinizer-log-apiproxy-template-default​

Template Data Structure Table​

Index Creation and Querying​

Index Creation​

Running Query​