Prerequisite: JQ (JSON Processor) Installation
JQ package needs to be installed on the server for the bash script to work properly.
1. Installing EPEL Repository
yum install epel-release -y
2. Updating Server
The following script needs to be saved to a directory with the name script.sh and made executable with the chmod +x script.sh command.
#!/bin/bash
es_url='http://<ELASTICSEARCH_IP>:9200'
index=apinizer-log-apiproxy-<XXXX>
response=$(curl -X GET -s $es_url/$index/_search?scroll=1m -H 'Content-Type: application/json' -d @query.json)
scroll_id=$(echo $response | jq -r ._scroll_id)
hits_count=$(echo $response | jq -r '.hits.hits | length')
hits_so_far=${hits_count}
echo Got initial response with $hits_count hits and scroll ID $scroll_id
# process first page of results here (ex. put the response into result.json)
echo $response | jq . >> result.json
while [ "$hits_count" != "0" ]; do
response=$(curl -X GET -s $es_url/_search/scroll -H 'Content-Type: application/json' -d "{ \"scroll\": \"1m\", \"scroll_id\": \"$scroll_id\" }")
scroll_id=$(echo $response | jq -r ._scroll_id)
hits_count=$(echo $response | jq -r '.hits.hits | length')
hits_so_far=$((hits_so_far + hits_count))
echo "Got response with $hits_count hits (hits so far: $hits_so_far), new scroll ID $scroll_id"
# process page of results (ex. put the response into result.json)
echo $response | jq . >> result.json
done
echo Done!
#script reference: https://gist.github.com/toripiyo/8b14e8a387069bae372d49296b0077d7
Example Query
The following query needs to be saved with the name query.json in the same directory as the script.sh file.Since this query needs to be sent to Apinizer Elasticsearch address, the address where the request is made and the index name need to be corrected according to your environment.
{
"from": 0,
"size": 3000000,
"query": {
"bool": {
"filter": [
{
"bool": {
"filter": [
{
"bool": {
"filter": [
{
"match": {
"uok": {
"query": "username",
"operator": "OR",
"prefix_length": 0,
"max_expansions": 50,
"fuzzy_transpositions": true,
"lenient": false,
"zero_terms_query": "NONE",
"auto_generate_synonyms_phrase_query": true,
"boost": 1.0
}
}
}
],
"adjust_pure_negative": true,
"boost": 1.0
}
}
],
"adjust_pure_negative": true,
"boost": 1.0
}
},
{
"bool": {
"filter": [
{
"bool": {
"should": [
{
"term": {
"pi": {
"value": "6130d19b59f2007bff548d29",
"boost": 1.0
}
}
}
],
"adjust_pure_negative": true,
"boost": 1.0
}
},
{
"range": {
"@timestamp": {
"from": "now-4320m/m",
"to": "now/m",
"include_lower": true,
"include_upper": true,
"boost": 1.0
}
}
}
],
"adjust_pure_negative": true,
"boost": 1.0
}
}
],
"adjust_pure_negative": true,
"boost": 1.0
}
},
"_source": {
"includes": [
"@timestamp",
"uok",
"fcrb",
"sc",
"pet",
"rt",
"tch",
"tcb",
"hr1ra",
"et",
"fcrh"
],
"excludes": []
}
}
You can visit this page to see what the fields in this query mean. Running Script
Run the ./script.sh command from terminal to run the script.
Afterwards, information notes start coming as below and results accumulate in the result.json file.
