Checking Indexes

Check Document Count

curl -X GET "<ELASTICSEARCH_IP>:9200/<INDEX_KEY>/_doc/_count"
BASH

Check Index with Where Clause

curl -X GET "<ELASTICSEARCH_IP>:9200/*/_search?pretty=true&q=apiGatewayName:KPS+XYS"
CODE
curl -X GET "<ELASTICSEARCH_IP>:9200/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "apiGatewayName": "TEST KPS GW"
          }
        },
        {
          "range": {
            "created": {
              "gte": "now-7d/d",
              "lt": "now-5d/d"
            }}}]}}}'
CODE
curl -X GET "<ELASTICSEARCH_IP>:9200/_search?pretty" -H 'Content-Type: application/json' -d'
{
    "_source": ["contentType"],   
    "size": 50,
    "query": {
        "match_all": {}
    }}'
CODE

Finding Requests That Have the Different CID From Response

curl -X GET "<ELASTICSEARCH_IP>:9200/_search?pretty" -H 'Content-Type: application/json' -d'
{
    "query": {
        "bool": {
            "filter": [
                {
                    "script": {
                        "script": {
                            "source": "doc['headerRequestFromClient.APINIZER-CORRELATION-ID.keyword'].value !=  doc['headerResponseToClient.APINIZER-CORRELATION-ID.keyword'].value",
                            "lang": "painless"
                        }}},
                {
                    "range": {
                        "created": {                            
                          "gte": "2021-06-28T16:30:32.000"                           
                        }}},
				{
				  "term":{
					"instanceId":2
				  }}]}}}'
CODE

Finding Request on Specific Interval

curl -X GET "<ELASTICSEARCH_IP>:9200/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
     "bool": {
      "filter": [
        {
          "match": {
            "api": "26"
          }},
        {
          "range": {
            "created": {
              "gte": "2020-06-08T15:08:00.000",
              "lte": "2020-06-08T15:12:00.000"
            }}}]}},   
    "aggs" : {
        "reqs_over_time" : {
            "date_histogram" : {
                "field" : "created",
                "interval" : "1s"
            }}}}'
CODE

Find Documents by Correlation ID

curl -X GET "<ELASTICSEARCH_IP>:9200/_search?pretty" -H 'Content-Type: application/json' -d' { "query" : { "match":{ "aci": "c3d8523e-e3ac-497b-ac7a-76853198c239" }}}'
BASH

Delete by Index Name

curl -X DELETE "<ELASTICSEARCH_IP>:9200/<INDEX_KEY>"
BASH


Deleting Indexes with Given Word

curl -X DELETE "<ELASTICSEARCH_IP>:9200/*metric*"
BASH


Changing Elasticsearch Cluster's Read_Only Status

curl -X PUT "<ELASTICSEARCH_IP>:9200/_all/_settings?wait_for_completion=false" -H "Content-Type: application/json"  -d'
{
    "index.blocks.read_only_allow_delete": null,
    "index.blocks.write": null
}'
BASH

For only one index:

curl -X PUT "<ELASTICSEARCH_IP>:9200/<INDEX_KEY>/_settings?pretty" -H 'Content-Type: application/json' -d' 
{
    "index.blocks.read_only_allow_delete": null,
    "index.blocks.write": null
}'
BASH

Deleting Logs Up to a Specific Date

curl -X POST "<ELASTICSEARCH_IP>:9200/.ds-apinizer-log-apiproxy-<LOG_KEY>-000*/_update_by_query?pretty" -H 'Content-Type: application/json' -d '
{
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "@timestamp": {
              "lte": "2024-04-20T00:00:00.000Z" 
            }
          }
       }
      ]
    }
  },
  "script": {
    "source": "ctx._source.remove(\"tba\"); ctx._source.remove(\"fbarb\"); ctx._source.remove(\"tcb\")"
  }
}'
BASH

To delete by api proxy id value instead of time range, "match": { "api": "64ac03067e8f7400cf4adbdd" } filter can be used instead of "range": { "@timestamp": { "lte": "2024-04-20T00:00:00.000Z"  } } filter.

To examine the data structure of Elasticsearch and identify fields to be deleted, visit the following link: API Traffic Log Data Structure.

Switching to New Index with Rollover

http://<ELASTICSEARCH_IP>:9200/apinizer-log-apiproxy-<INDEX_KEY>/_rollover
BASH

Search

Search Documents on Specific Index

curl -X GET "<ELASTICSEARCH_IP>:9200/<INDEX_KEY>/_search?pretty=true&q=*:*"
BASH

Search Documents on Index with Criteria

curl -X GET "<ELASTICSEARCH_IP>:9200/*/_search?pretty=true&q=apiProxyName:Petstore+API"
BASH

Search Documents by Interval

curl -X GET "<ELASTICSEARCH_IP>:9200/_search?pretty" -H 'Content-Type: application/json' -d
'{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "apiProxyName": "Petstore API"
          }
        },
        {
          "range": {
            "created": {
              "gte": "now-7d/d",
              "lt": "now-5d/d"
            }
          }
        }
      ]
    }
  }
}'
BASH

Search with Conditions and Aggregating the Results

curl -X GET "<ELASTICSEARCH_IP>:9200/_search?pretty" -H 'Content-Type: application/json' -d
'{
  "query": {
     "bool": {
      "filter": [
        {
          "match": {
            "api": "26"
          }
        },
        {
          "range": {
            "created": {
              "gte": "2020-06-08T15:08:00.000",
              "lte": "2020-06-08T15:12:00.000"
            }
          }
        }
      ]
	}    
  }, 	
	"aggs" : {
        "reqs_over_time" : {
            "date_histogram" : {
                "field" : "created",
                "interval" : "1s"
            }
        }
    }
}'
BASH

Search with Limitations

curl -X GET "<ELASTICSEARCH_IP>:9200/_search?pretty" -H 'Content-Type: application/json' -d'
{
	"_source": ["contentType"],    
    "size": 50,
    "query": {
        "match_all": {}
    }
}'
BASH

Update

Update Document

curl -X PUT "<ELASTICSEARCH_IP>:9200/<INDEX_KEY>/doc/1?pretty&pretty" -H 'Content-Type: application/json' -d'
{
  "name": "John Doe"
}'
BASH


Deleting Some Key and Values

curl -X POST "<ELASTICSEARCH_IP>:9200/_update_by_query?pretty" -H 'Content-Type: application/json' -d'
{
  "script" : 
	"ctx._source.headerRequestFromClient.remove('header-name-1');
	 ctx._source.headerRequestToTarget.remove('header-name-2');",
  "query": { "match_all": {} }
}
BASH

Deleting Specific Values by Criteria

curl -X  POST "<ELASTICSEARCH_IP>:9200/*/_update_by_query?pretty&conflicts=proceed&requests_per_second=200" -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool" : {
      "filter": {
        "exists": {
            "field": "headerRequestFromClient.user_username"
          }
      },
      "must_not" : {
       "term": {
          "headerRequestFromClient.user_password": ""
        }
      }
    }
  },
  "script":  "ctx._source.headerRequestFromClient.remove(\"user_password\");"
}
BASH
  • Execution Reject error will be prevented by requests_per_second key's value.
  • Batch Size is 1000 by default. Waiting time between two request is set by giving 5 (=1000/200) 
 http://<ELASTICSEARCH_IP>:9200/*/_update_by_query?conflicts=proceed&wait_for_completion=true
{
  "script": {
    "inline": 
	"ctx._source.remove('apiGatewayApiMethodId');
	 ctx._source.remove('bodyRequestToTarget');
	 ctx._source.remove('bodyResponseFromTarget');
	 ctx._source.remove('bodyResponseToClient');
	 ctx._source.remove('headerRequestFromClient');
	 ctx._source.remove('headerRequestToTarget');
	 ctx._source.remove('headerResponseFromTarget');
	 ctx._source.remove('headerResponseToClient');
	 ctx._source.remove('paramRequestFromClient');
	 ctx._source.remove('paramRequestToTarget');",
    "lang": "painless"
  },
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "created": {
             "gte": "2019-02-01T20:03:12.963",
              "lte": "2019-04-30T20:03:12.963"
            }
          }
        }
      ],
      "adjust_pure_negative": true,
      "boost": 1
    }
  }
}
BASH

Setting Replica Number

 curl -X PUT "<ELASTICSEARCH_IP>:9200/_template/template_genel?pretty" -H 'Content-Type: application/json' -d'
{
  "index_patterns": ["*log*", "*metric*", "*db*"],
  "settings": {
    "number_of_shards": 1,
	"number_of_replicas": 0
  } 
}
'  
BASH
curl -XPUT '<ELASTICSEARCH_IP>:9200/*/_settings' -H 'Content-Type: application/json' -d'
{
	"index" : {            
		"number_of_replicas" : 0
	}    
}'
BASH

Shard Allocation

curl -X PUT "<ELASTICSEARCH_IP>:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
    "transient" : {
        "cluster.routing.allocation.enable" : "all"
    }
}'
BASH

Shard Limit

http://<ELASTICSEARCH_IP>:9200/_cluster/settings
{
  "persistent" : {
    "cluster.routing.allocation.total_shards_per_node" : 2000 ,
    "cluster.max_shards_per_node":2000
  }
}
BASH

Log

Changing Log Level

curl -X PUT "<ELASTICSEARCH_IP>:9200/_cluster/settings" -H 'Content-Type: application/json' -d'{"transient":{"logger._root":"DEBUG"}}'
curl -X PUT "<ELASTICSEARCH_IP>:9200/_cluster/settings" -H 'Content-Type: application/json' -d'{"transient":{"logger._root":"INFO"}}'
BASH

ShowLog Settings

curl -X PUT "<ELASTICSEARCH_IP>:9200/*log*/_settings?pretty" -H 'Content-Type: application/json' -d'
{
    "index.search.slowlog.threshold.fetch.trace": "200ms",
    "index.search.slowlog.level": "trace"
}'
BASH

Other

_cat APIs

curl -X GET "<ELASTICSEARCH_IP>:9200/_cat/indices/*?v&s=index&pretty"

curl -X GET "<ELASTICSEARCH_IP>:9200/_cat/thread_pool?v&h=id,node_name,ip,name,core,queue,rejected,completed,max"
BASH

_nodes APIs

curl -X GET "<ELASTICSEARCH_IP>:9200/_nodes/os?pretty"

curl -X GET "<ELASTICSEARCH_IP>:9200/_nodes/jvm?pretty"

curl -X GET "<ELASTICSEARCH_IP>:9200/_nodes/thread_pool?pretty"

curl -X GET "<ELASTICSEARCH_IP>:9200/_nodes/stats/process?filter_path=**.max_file_descriptors"
BASH

_cluster APIs

curl -X GET "<ELASTICSEARCH_IP>:9200/_cluster/stats?pretty"

curl -XGET '<ELASTICSEARCH_IP>:9200/_cluster/state?pretty=true' > result.json
BASH

Flush

curl -X POST "<ELASTICSEARCH_IP>:9200/*log*/_flush/synced?pretty"
BASH

Removing the Log Writing Barrier

curl -XPUT 'http://<ELASTICSEARCH_IP>:9200/*log*/_settings' -H 'Content-Type: application/json' -d'{"index": {"blocks": {"read_only_allow_delete": null}}}'
BASH


General Information About Snapshots

curl 'http://<ELASTICSEARCH_IP>:9200/_snapshot?pretty'
BASH


Repository and Snapshot Details

curl 'http://<ELASTICSEARCH_IP>:9200/_slm/policy/apinizer-slm-policy-<INDEX_KEY>?pretty' 
BASH


Detailed Examination of the Repository and Snapshot Names Obtained from the Previous Command

curl -XGET "http://<ELASTICSEARCH_IP>:9200/_snapshot/apinizer-repository-<INDEX_KEY>/apinizer-snapshot-<INDEX_KEY>-2023.03.13-m33h8zcpq_if4swyzn0wrq?pretty"

curl -XGET "http://<ELASTICSEARCH_IP>:9200/_snapshot/apinizer-repository-<INDEX_KEY>/apinizer-snapshot-<INDEX_KEY>-2023.03.13-m33h8zcpq_if4swyzn0wrq/_status?pretty"
BASH


Deleting All Settings Related to Snapshots

curl -XDELETE 'http://<ELASTICSEARCH_IP>:9200/_snapshot/apinizer-repository-<INDEX_KEY>?pretty'
BASH