Backup Policy

First, the method of backup must be decided. The commonly used methods are described below.

Method 1: Backing Up the path.data Location

The path.data location defined in the Elasticsearch configuration file can be backed up by system administrators incrementally or at regular intervals.

bilgi

Access to historical data always remains just as easy.

uyarı

Both the active and backup disks keep growing continuously. If an issue occurs on the primary disk, the system must be reinstalled to access the data on the backup disk.

Method 2: Server-Level Disk Backup (RAID-0)

The server hosting Elasticsearch can be backed up using RAID-0 or at regular intervals.

bilgi

Access to historical data always remains just as easy. If an issue occurs on the primary disk, the backup disk can be brought into use immediately through network rerouting.

uyarı

Both the active and backup disks keep growing continuously.

Method 3: Elasticsearch Snapshot API

Elasticsearch data can be dumped using the Snapshot API. A snapshot policy can be configured to export backup files to a designated system location; these backup files should then be separately backed up to a different server.

bilgi

Access to historical data always remains just as easy.

uyarı

Both the active and backup disks keep growing continuously.

bilgi

Recommendation

Regardless of which method above is used, once regular backups are configured, logs that have already been backed up can be set to be deleted automatically via Elasticsearch ILM, or manually deleted as needed using the Elasticsearch API. This allows the active server to continue operating with a much lower disk footprint. However, in this case, an application must be installed on the backup disk to access historical data, or the backups must be transferred to a separate server and worked with there; from that point on, only the backup disk keeps growing.

Manual Elasticsearch Backup and Restore

This section explains how to create a Snapshot Lifecycle Management (SLM) policy for automatically backing up logs on Elasticsearch via a cron definition, as well as how to take an on-demand backup and restore it.

Variables

The dynamic values used in the requests and their descriptions are listed in the table below.

Variable	Description
`<ELASTICSEARCH_IP_ADDRESS>`	The host information of the Elasticsearch cluster.
`<INDEX_KEY>`	This value must be unique, as it serves as an identifier at the cluster level. Therefore, the same value must be used in all requests.

Specifying the Backup File Location

Add the path.repo field to the elasticsearch.yml configuration file on all nodes in the cluster, specifying the file location where backup files will be stored.

uyarı

If this setting is added to the configuration file after the initial setup, the node must be restarted.

path:
  repo:
    - /backups/my_backup_location

Defining the Snapshot Repository

The repository holds the information about where the files to be backed up during the snapshot process will be stored.

curl -X PUT "http://<ELASTICSEARCH_IP_ADDRESS>:9200/_snapshot/apinizer-repository-<INDEX_KEY>?pretty" -H 'Content-Type: application/json' -d'
{
  "type": "fs",
  "settings": {
    "location": "/backups/my_backup_location",
 	"compress": true
  }
}
'

Verify Location Request

It must be checked whether Elasticsearch has access to the file location.

bilgi

If verification succeeds, the list of nodes using the repository is returned. If verification fails, the request returns an error.

curl -X POST "http://&#60;ELASTICSEARCH_IP_ADDRESS&#62;:9200/_snapshot/apinizer-repository-&#60;INDEX_KEY&#62;/_verify?pretty"

If backups will be taken automatically, the SLM Policy commands must be run. If a backup will be taken at a specific time, the On-Demand Backup commands must be run.

Creating an SLM Policy

Snapshot Policy Creation Request

curl -X PUT "http://<ELASTICSEARCH_IP_ADDRESS>:9200/_slm/policy/apinizer-slm-policy-<INDEX_KEY>?pretty" -H 'Content-Type: application/json' -d'
{
  "schedule": "0 0 0 ? * 1#1 *", 
  "name": "<apinizer-snapshot-<INDEX_KEY>-{now/d}>", 
  "repository": "apinizer-repository-<INDEX_KEY>", 
  "config": { 
    "indices": ["apinizer-log-apiproxy-<INDEX_KEY>"],  
	"ignore_unavailable": false,
    "partial": false   
  },
  "retention": { 
    "expire_after": "30d", 
    "min_count": 5, 
    "max_count": 50 
  }
}
'

Manually Running the Policy Request

curl -X POST "http://&#60;ELASTICSEARCH_IP_ADDRESS&#62;:9200/_slm/policy/apinizer-slm-policy-&#60;INDEX_KEY&#62;/_execute?pretty"

Viewing Snapshot Records

curl -X GET "http://&#60;ELASTICSEARCH_IP_ADDRESS&#62;:9200/_snapshot/apinizer-repository-&#60;INDEX_KEY&#62;/apinizer-snapshot-&#60;INDEX_KEY&#62;*?pretty"

bilgi

Backups (snapshots) are usually kept within the same environment where indexing takes place. Afterward, these snapshots may need to be stored in a different environment. In Elasticsearch, simply transferring the files does not automatically make the data available on another Elasticsearch cluster. In addition to transferring the files, the snapshot structure itself must also be migrated in the same way. Transferring only the files can both corrupt the structure within the snapshot and prevent the backup from being restored.

When moving a snapshot to another cluster, the repository must be created first, followed by the snapshot.

bilgi

Once the snapshot process is complete, the indices that have been backed up are not deleted. They are only deleted if the delete phase is enabled in the index's ILM policy.

Taking an On-Demand Backup

Snapshot Creation Request

curl –XPUT "http://&#60;ELASTICSEARCH_IP_ADDRESS&#62;:9200/_snapshot/apinizer-repository-&#60;INDEX_KEY&#62;/apinizer-snapshot-&#60;INDEX_KEY&#62;?wait_for_completion=true" -H 'Content-Type: application/json' –d '{
  "indices":"index001, index002, index003",
  "ignore_unavailable":true,
  "include_global_state": false
}'

Restoring All or Part of the Indices in a Snapshot

ipucu

Multiple values can be entered in the indices field of the command, and the wildcard value * can also be used.

curl -XPOST "http://&#60;ELASTICSEARCH_IP_ADDRESS&#62;:9200/_snapshot/apinizer-repository-&#60;INDEX_KEY&#62;/apinizer-snapshot-&#60;INDEX_KEY&#62;/_restore?pretty" -H 'Content-Type: application/json' -d '{
  "indices":"index00*",
  "ignore_unavailable":true,
  "include_global_state": false
}'

Elasticsearch Snapshot Transfer and Restore Script

After the snapshot policy has been configured, this script transfers the snapshot files created at a given location to the backup server, restores them there, and keeps them in a state ready to be read.

The following are required to run the script:

Both the log server and the backup server must be running on a Linux server that supports shell scripting.
The backup server must be running an Elasticsearch version that is the same as, or supported alongside, the existing Elasticsearch server.
The log server and the backup server must be able to communicate via protocols such as ssh and scp.
The log server must support crontab (available by default on most popular Linux distributions).
Basic Linux shell knowledge.

The steps performed by the script are as follows:

The repository is checked.
The name and location of the snapshot file are retrieved.
The snapshot file is sent to the restore server.
The snapshot restore process is started.
The status of the restore process is checked.

#!/bin/bash

#Logs will be written to a file
current_date=$(date +'%d-%m-%Y')
exec > logfile$current_date.log 2>&1

#Server IP's need to be set
es_snapshot_ip="<ELASTICSEARCH_IP_ADDRESS>"
es_restore_ip="<ELASTICSEARCH_BACKUP_SERVER>"
repository_dst_location="<BACKUP_PATH_REPO>"
log_key="<INDEX_KEY>"


time_start=`date +%s`

echo -e "\n\nScript has started on \"`date`\""

es_snapshot_address="http://$es_snapshot_ip:9200"
es_restore_address="http://$es_restore_ip:9200"
echo "Variables:"
echo " es_snapshot_address: $es_snapshot_address"
echo " es_restore_address: $es_restore_address"
echo " repository_dst_location: $repository_dst_location"


##Show repositories, take name and path
repository_name=$(curl -XGET -s "$es_snapshot_address/_snapshot/_all" |  jq -r 'keys[] | select(contains("repository"))')
repository_src_location=$(curl -XGET -s "$es_snapshot_address/_snapshot/_all" | jq -r ' .[].settings.location ' | head -1)
echo " repository_name: $repository_name"
echo " repository_src_location: $repository_src_location"

##Show snapshots on repository
echo -e "Command to be used: curl -XGET -s \"$es_snapshot_address/_snapshot/apinizer-repository-$log_key/_all\" | jq '.snapshots[].snapshot' | tr -d '\"' \n"
snapshot_name=$(curl -XGET -s "$es_snapshot_address/_snapshot/apinizer-repository-$log_key/_all" | jq '.snapshots[].snapshot' | tr -d '"')
echo " snapshot_name: $snapshot_name"
time_1=`date +%s`
echo -e "\nduration - since beginning: $((time_1-time_start)) seconds"

##Move Snapshot files to remote server
echo "---Moving Snapshot to remote server: Started"
size_snapshot=$(du -sh $repository_src_location)
echo "Snapshot file size: $size_snapshot"

size_dst_initial=$(ssh elasticsearch@$es_restore_ip "du -sh ${repository_dst_location/}")
echo "Target disk size before moving snapshot: $size_dst_initial"

echo "scp -r $repository_src_location/* elasticsearch@$es_restore_ip:$repository_dst_location/ &"
scp -r $repository_src_location/* elasticsearch@$es_restore_ip:$repository_dst_location/ &
SCP_PID=$!
wait $SCP_PID

echo "---Moving Snapshot to remote server: Done"
size_dst_afterscp=$(ssh elasticsearch@$es_restore_ip "du -sh ${repository_dst_location/}")

echo "Target disk size after moving snapshot: $size_dst_afterscp"

time_2=`date +%s`
echo "---duration - scp: $((time_2-time_1)) seconds"

if [ "$size_dst_initial" = "$size_dst_afterscp" ];
then
  echo "Moving snapshot file has failed. Script is being terminated."
  exit
fi



##Register repository on remote server
time_3=`date +%s`
echo -e "Command to be used: curl -XPUT \"$es_restore_address/_snapshot/$repository_name?pretty\" -H \"Content-Type: application/json\" -d '{   \"type\": \"fs\",   \"settings\": {     \"compress\" : \"true\",     \"location\": \"$repository_dst_location\"   } }' \n"
curl -XPUT "$es_restore_address/_snapshot/$repository_name?pretty" -H "Content-Type: application/json" -d '{
  "type": "fs",
  "settings": {
    "compress" : "true",
    "location": "'$repository_dst_location'"
  }
}'


##Start restoring snapshot
echo -e "\nRestore: Started"
echo "Command to be used: curl -XPOST -s \"$es_restore_address/_snapshot/$repository_name/$snapshot_name/_restore?pretty\" -H \"Content-Type: application/json\" -d '{   \"indices\": \".ds-apinizer-log-token-$log_key-*,.ds-apinizer-token-oauth-$log_key-*,.ds-apinizer-log-apiproxy-$log_key-*\",   \"rename_pattern\": \"(.ds-apinizer-)(.*$)\",   \"rename_replacement\": \"restored_$1$2\" }'"
index_to_close_list=()

while [ true ]
do
curl -XPOST -s "$es_restore_address/_snapshot/$repository_name/$snapshot_name/_restore?pretty" -H "Content-Type: application/json" -d '{
  "indices": ".ds-apinizer-log-token-$log_key-*,.ds-apinizer-log-apiproxy-$log_key-*",
  "rename_pattern": "(.ds-apinizer-)(.*$)",
  "rename_replacement": "restored_$1$2"
}' -o  restore.output

is_error_exist=$(grep -oPm 1 'error' < restore.output)
if [ "$is_error_exist" = "error" ];then
##There are always expected to be at least 1 conflicted index (The last one). Those indexes needs to be closed to write on them
	index_to_close=$(grep -oPm 1 'restored_.ds-apinizer-.*$log_key-\d{6}' < restore.output)
	curl -XPOST -s "$es_restore_address/$index_to_close/_close?pretty" >> closed_indexes.output
	index_to_close_list+=($index_to_close)
	echo "Conflicted index has closed: $index_to_close"
else
	echo "There is no conflicted indeks. Script is being continued."
	break
fi
done

##Check restore process hourly
echo -e "Command to be used: curl -XGET -s \"$es_restore_address/_cluster/state\" | jq '.restore.snapshots[].state' \n"
echo "Restore process will be checked every hour before continuing to script."
while [ true ]
do
sleep 3600
restore_result=$(curl -XGET -s "$es_restore_address/_cluster/state" | jq '.restore.snapshots[].state')


if [[ "$restore_result" = "STARTED" ] || [ "$restore_result" = "INIT" ]];then
	echo "Status of Restore process as per _cluster/state: $restore_result. Restore is in progress."
elif [ "$restore_result" = "DONE" ]; then
	echo "Status of Restore process as per _cluster/state: $restore_result. Continuing to script."
	break
elif [ "$restore_result" = "" ]; then
	echo "Status of Restore process could not obtained from _cluster/state. Continuing to script."
	break
fi
done


time_4=`date +%s`
echo "duration - restore: $((time_4-time_3)) seconds"

##Open closed indexes if there are any
for index in $index_to_close_list; do
  curl -XPOST -s "$es_restore_address/$index/_open"
done

##Setting visibility of restored indexes to visible
echo -e "\nSet visibility of restored indexes: Started"
cluster_dst_state=$(curl -s "$es_restore_address/_cluster/state")
restored_indices=$(echo "$cluster_dst_state" | jq '.metadata.indices | keys | .[]' | grep '^"restored_.*"')
restored_indices=${restored_indices//\"}

echo -e "Command to be used: curl -XPUT -s \"$es_restore_address/INDEX/_settings?pretty\" -H 'Content-Type: application/json' -d'{     \"index.hidden\": false   }' \n"
for index in $restored_indices; do
  curl -XPUT -s "$es_restore_address/$index/_settings?pretty" -H 'Content-Type: application/json' -d'{
    "index.hidden": false
  }'
done

echo "Set visibility of restored indexes: Done"
time_5=`date +%s`
echo "duration - restored visibility: $((time_5-time_4)) seconds"




##Making sure of if the restore process done. Index counts should be the same as snapshot file has
echo -e "\nChecking restore results: Started"
echo -e "Command to be used: curl -s \"$es_restore_address/_snapshot/$repository_name/$snapshot_name?pretty\" \n"
snapshot_json=$(curl -s "$es_restore_address/_snapshot/$repository_name/$snapshot_name?pretty")
snapshot_indices_array=$(echo "$snapshot_json" | jq -r '.snapshots[0].indices[]')
snapshot_index_count=$(echo "$snapshot_indices_array" | grep -c ".")
echo "Total number of indices in snapshot: $snapshot_index_count"

recovery_info=$(curl -s "$es_restore_address/_cat/recovery")
filtered_lines=$(echo "$recovery_info" | grep "$snapshot_name" | awk '$14 == "100.0%" || $4 == "100.0%"')

completed_count=$(echo "$filtered_lines" | grep -c "100.0%")
uncompleted_count=$(echo "$filtered_lines" | grep -cv "100.0%")

echo "-Restore completed: $completed_count"
echo "-Restore uncompleted: $uncompleted_count"

if [ "$uncompleted_count" -gt 0 ];
  then
	echo -e "\nUncompleted Indices:"
	echo "$filtered_lines" | grep -v "100.0%"
	echo -e "\n\n---There are indexes that could not be restored!---\n\n"
  elif [ "$uncompleted_count" -e 0 ];
  then
	echo -e "\n\n---Restore was successful."
	echo "Snapshot file will be deleted by Elasticsearch according to SLM policy."
    echo -e "To manually delete, following command can be used: curl -XDELETE \"$es_snapshot_address/_snapshot/$repository_name/$snapshot_name\" \n"
  else
	echo "Checking restore results has failed. Please check results manually."
fi
echo "Checking restore results: Done"
time_6=`date +%s`

echo "duration - checking restore results: $((time_6-time_5)) seconds"


time_end=`date +%s`
echo "\nduration - total time of script: $((time_end-time_start)) seconds"

##Clear the variables set to shell just in case
unset time_1 time_2 time_3 time_4 time_5 time_6 time_start time_end snapshot_json snapshot_indices_array snapshot_index_count recovery_info filtered_lines completed_count uncompleted_count current_date es_snapshot_address es_restore_address repository_dst_location repository_name repository_src_location snapshot_name cluster_dst_state restored_indices size_dst_afterscp size_dst_initial size_snapshot restore_result es_snapshot_ip es_restore_ip is_error_exist index_to_close_list index_to_close apinizer_adres
echo "Used variables has been cleansed."



echo -e "\n\nScript is done on \"`date`\" \n"

echo "Note: If there is a error log like -All shards failed-, those indexes needs to be deleted from remote cluster and restore process needs to be initialized partially."
##Script Ends##

How It Works:

Save the script to a suitable location on your log server, e.g. under the name "ESMoveSnapshotAndRestore.sh". The script can be copied using shell editors such as vi or nano on Linux, or saved to a file on a Windows machine and transferred via sftp using an application such as WinSCP or MobaXterm.
Grant execute permission with chmod +x ESMoveSnapshotAndRestore.sh.
To avoid password prompts during scp and allow automatic connection, the script relies on scp's ssh key authentication feature.
- A key is generated on the log server using ssh-keygen, and pushed to the backup server using ssh-copy-id.
If not already installed, install the jq package on the log server. Use apt install jq on Ubuntu or yum install jq on RedHat.
Verify the data.path and repo.path values in the Elasticsearch config files on both servers.
Set the variables in the script to match your environment.
- ELASTICSEARCH_SERVER
- ELASTICSEARCH_BACKUP_SERVER
- LOG_KEY
- BACKUP_PATH_REPO

Usage:

Before running the script, enter your own Elasticsearch variable values.

chmod +x /path/to/ESMoveSnapshotAndRestore.sh
./path/to/ESMoveSnapshotAndRestore.sh &

This process can be run manually, or repeated at set intervals. To repeat it automatically, this entry needs to be added to the Linux cron job settings.

CronJob Usage:

Open the cron editor by running the following command in the terminal:

crontab -e

Add a line to the opened editor according to how frequently you want the script to run.

For example, to run it at 23:00 on the 3rd day of every month:

0 23 3 * * /path/to/ESMoveSnapshotAndRestore.sh

To save the line you added, press Esc, type :wq, and press Enter.

bilgi

In both methods, when the script runs, its output will be written to a file in the same folder as the script, in the format "logfile<DATE>.log".

Method 1: Backing Up the path.data Location​

Method 2: Server-Level Disk Backup (RAID-0)​

Method 3: Elasticsearch Snapshot API​

Manual Elasticsearch Backup and Restore​

Variables​

Specifying the Backup File Location​

Defining the Snapshot Repository​

Verify Location Request​

Creating an SLM Policy​

Snapshot Policy Creation Request​

Manually Running the Policy Request​

Viewing Snapshot Records​

Taking an On-Demand Backup​

Snapshot Creation Request​

Restoring All or Part of the Indices in a Snapshot​

Elasticsearch Snapshot Transfer and Restore Script​

How It Works:​

Usage:​

CronJob Usage:​

Method 1: Backing Up the path.data Location

Method 2: Server-Level Disk Backup (RAID-0)

Method 3: Elasticsearch Snapshot API

Manual Elasticsearch Backup and Restore

Variables

Specifying the Backup File Location

Defining the Snapshot Repository

Verify Location Request

Creating an SLM Policy

Snapshot Policy Creation Request

Manually Running the Policy Request

Viewing Snapshot Records

Taking an On-Demand Backup

Snapshot Creation Request

Restoring All or Part of the Indices in a Snapshot

Elasticsearch Snapshot Transfer and Restore Script

How It Works:

Usage:

CronJob Usage: