Curator Operations

Installation

FOR CENTOS
-------------------------------------------
# link: https://www.elastic.co/guide/en/elasticsearch/client/curator/current/yum-repository.html
-------------------------------------------

$vi /etc/yum.repos.d/curator.repo
içerik:
[curator-5]
name=CentOS/RHEL 6 repository for Elasticsearch Curator 5.x packages
baseurl=https://packages.elastic.co/curator/5/centos/6
gpgcheck=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
 
$yum install elasticsearch-curator 
 
running curator every day at 03.00
crontab -e

0 3 * * * /usr/bin/curator --config /mnt/ElasticData/curator/curator.yml /mnt/ElasticData/curator/readonly_shrink.yml
 
service crond restart

sudo tail -900f /var/log/cron

# Troubleshooting:
# https://logz.io/blog/elasticsearch-cheat-sheet/
# https://www.elastic.co/blog/red-elasticsearch-cluster-panic-no-longer

BASH

FOR UBUNTU
-------------------------------------------
# link: https://www.elastic.co/guide/en/elasticsearch/client/curator/current/apt-repository.html
-------------------------------------------

BASH

Curator.yml

To run Curator, we need the curator.yml file. This file specifies how Curator should connect and where:

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
client:
  hosts:
    - 10.6.1.11
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile: curator.log
  logformat: default
# default: blacklist: ['elasticsearch', 'urllib3']

YML

Snapshot

To take a snapshot, we need a YAML file containing snapshot settings.

In addition to curator.yml, we create a snapshot.yml file:

actions:
  1:
    action: snapshot
    options:
      disable_action: False
      repository: "es_apinizer_snapshot_20200914"
      ignore_empty_list: True
      wait_interval: 10
      max_wait: -1
    filters:
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 90

YML

We create a repository in Elasticsearch first and then run Curator with the snapshot.yml using the following commands:

mkdir -p /LOGARSIV/arsiv/apinizerLogBackup/es_apinizer_snapshot_20200914

# EKLEMEK için:
es_repo_mgr  --config /mnt/ElasticData/curator/curator.yml  create fs --repository es_apinizer_snapshot_20200914 --location /LOGARSIV/arsiv/apinizerLogBackup/es_apinizer_snapshot_20200914 --compression true

# SİLMEK için:
es_repo_mgr  --config /mnt/ElasticData/curator/curator.yml  delete --repository es_apinizer_snapshot_20200914
 
curator --config /mnt/ElasticData/curator/curator.yml /mnt/ElasticData/curator/snapshot.yml &

curl -X GET "<ELASTICSEARCH_IP>:9200/_snapshot/_status?pretty" > status.json

curl -X GET "<ELASTICSEARCH_IP>:9200/_snapshot/_all?pretty"

curl -X GET "<ELASTICSEARCH_IP>:9200/_cat/snapshots/es_apinizer_snapshot_20200914?v&s=id&pretty"

BASH

Delete

To be able to delete the desired indexes, we need a YAML file containing delete settings.

In addition to curator.yml, we create a delete.yml file:

actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 90 days
    options:
      ignore_empty_list: True
      disable_action: False
    filters:
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 90

YML

We run the following command to make Curator delete the indexes:

curator --config /mnt/ElasticData/curator/curator.yml /mnt/ElasticData/curator/delete.yml &

BASH

Readonly & Shrink

To set the mode of the desired indexes to readonly and be able to shrink them, we need a YAML file containing these settings.

In addition to curator.yml, we create a readonly_shrink.yml file:

actions:
  1:
    action: index_settings
    description: >-
      Set log indices older than 2 days to be read only (block writes)
    options:
      disable_action: False
      index_settings:
        index:
          blocks:
            write: True
      ignore_unavailable: False
      preserve_existing: False
      indices: 
      continue_if_exception: True
    filters:
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 2
  2:
    action: shrink
    options:
      disable_action: False
      ignore_empty_list: True
      shrink_node: DETERMINISTIC
      node_filters:
        permit_masters: True
      number_of_shards: 1
      number_of_replicas: 0
      indices:
      continue_if_exception: True
      shrink_prefix:
      shrink_suffix: '-shrink'
      delete_after: True
      post_allocation:
        allocation_type: include
        key: node_tag
        value: cold
      wait_for_active_shards: 1
      wait_for_completion: True
      wait_for_rebalance: True
      wait_interval: 9
      max_wait: -1
    filters:
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 2

YML

We run the following command to make Curator set the indexes to readonly and then shrink them:

curator --config /mnt/ElasticData/curator/curator.yml /mnt/ElasticData/curator/readonly_shrink.yml &

BASH