Elasticsearch Cluster

All API Traffics are kept in Elasticsearch database in Apinizer.

Elasticsearch API has a powerful structure for protecting, scaling and fast execution of data that needs to be monitored and analyzed beyond recording their traffic.

On this module, the operations of defining nodes that make up an Elasticsearch cluster, creating connection pooling for connection management, managing Apinizer Index Lifecycle Policies and viewing Apinizer Index Templates are performed.

Before creating an environment in Apinizer, you need to define your existing Elasticsearch server/clusters to Apinizer.

The picture below shows the Elasticsearch Cluster settings:

The fields used for Elasticsearch Cluster configuration are shown below.

Field	Description
Name	It is the name information for the created cluster.
Log Key	It is an abbreviated key used for the created cluster. It facilitates cluster management by using index, template, ILM and SLM policy naming. When the information in the Log Key field is changed, all the template and ILM policies of the cluster also change. Therefore, this field should not be changed unless necessary. If the ILM Policy and Template information defined in a deleted Elasticsearch cluster are to be used in a new cluster definition, the log key information must be the same.
Description	An explanation can be written to facilitate the management of the created cluster.
Enable to authenticate	It must be activated to implement the Simple Authentication method. Click to apply the configuration settings and for detailed information.
Username	If authentication is enabled, the user name of the server must be entered.
Password	If authentication is activated, the password of the server must be entered.
Scheme Type	If authentication is enabled, the schema type of the server must be selected.
Host	The IP or host name of the server must be entered.
Port	The port information of the server must be entered.
Socket Reuse Address	It is set whether to reuse the socket address in the connection pool.
Socket Keep Alive	It is set whether to keep the socket alive in the connection pool.
IO Threads	The number of threads for the connection pool is entered.
Max Connection Per Host	The maximum number of connections per host in the connection pool is entered.
Max Connection Total	The maximum number of connections that should be in the connection pool is entered.
Enable to administrate cluster	It must be active to apply the Apinizer Index Lifecycle Policies and Index Templates defined by Apinizer.
Apinizer Index Lifecycle Policies	By default, Index Lifecycle Policies defined by Apinizer are managed within this section.
Apinizer Index Templates	By default, Index Templates defined by Apinizer are displayed in this section.
Test Connection	After entering the information of the Elasticsearch cluster, a connection test can be performed by clicking the button.

Editing the Apinizer Index Lifecycle

Phases of default lifecycle policies can be edited when enabled to manage the cluster.

An index lifecycle policy allows defining rules for specific actions such as a rollover or force merge within a specified period of an index and automates the process.

Index lifecycle management defines 4 different phases in which indexes can take place:

1. Hot phase: It is the phase where active querying and writing operations are performed on the index. A new index is created over the pseudo-index name given to very large or very old indexes. It is generally recommended to be stored in SSD space to achieve the best performance requirements.
2. Warm phase: In this phase, the index can be queried, but the index is in read-only mode. Shards can be reserved for less performing hardware. For faster searches, the number of shards can be reduced and segments can be merged to reduce the number.
3. Cold phase: Less queries are made on indexes and queries take longer to be answered. It is still too early to delete data. E.g; In the case of security monitoring, old data may be needed to support its investigation. Therefore, hardware with less performance can be used and the number of replicas can be reduced.
4. Delete phase: When the index is no longer needed, it can be defined when it is safe to delete.

• Log politikası için Delete Fazı etkin halde değildir.
• Trace politikası için Cold Fazı etkinleştirilemez.
• Token politikası içinse Cold ve Delete Fazı etkinleştirilemez.
• The Delete Phase is not active for the log policy.
• The Cold Phase cannot be enabled for the trace policy.
• For token policy, The Cold Phase and The Delete Phase cannot be enabled.

The picture below shows the Index Lifecycle settings:

The fields used to configure the Index Lifecycle are shown below.

Updating Index Lifecycle Management Policies and Templates

An error can occur when applying the change to Elasticsearch, either when the cluster definition is updated or at the time of initial creation. This can be seen from the boxes in the "Status of Index Lifecycle Policies, Status of Index Templates" columns on the screen where cluster definitions are listed.

When these boxes are clicked, the relevant status is updated on Elasticsearch.

The picture below shows the list of Elasticsearch Clusters:

When the Elasticsearch Cluster definiton is deleted, the relevant indexes, the policies and the templates will not be deleted. However, policies belonging to the cluster's Index Lifecycle Management (ILM) disable all phases. Thus, unnecessary resource consumption is be prevented. 

For instance; Supposing any index in the deleted cluster is assigned 5 to the minimum age of the index in the hot phase of the ILM policy. A new index will be created every 5 days. This action is considered necessary in order not to increase the number of indexes and shards in the cluster.

You can access our experience with Elasticsearch on our Medium blog.