All API Traffics are kept in Elasticsearch database in Apinizer.

Elasticsearch API has a powerful structure for protecting, scaling and fast execution of data that needs to be monitored and analyzed beyond recording their traffic.

On this module, the operations of defining nodes that make up an Elasticsearch cluster, creating connection pooling for connection management, managing Apinizer Index Lifecycle Policies and viewing Apinizer Index Templates are performed.

Before creating an environment in Apinizer, you need to define your existing Elasticsearch server/clusters to Apinizer.

The image containing the Elasticsearch Cluster settings is given below:


The fields used for Elasticsearch Cluster configuration are shown below.

Field

Description

Name

It is the name information for the created cluster.

Log Key

It is an abbreviated key used for the created cluster. Makes it easy to manage Elasticsearch clusters.

Additionally, it is used for naming indexes and templates.

Description

An explanation can be written to facilitate the management of the created cluster.

Enable to authenticate

It must be activated to implement the Simple Authentication method.

Click to apply the configuration settings and for detailed information.

Username

If authentication is enabled, the user name of the server must be entered.

Password

If authentication is activated, the password of the server must be entered.

Scheme Type

If authentication is enabled, the schema type of the server must be selected.

Host

The IP or host name of the server must be entered.

Port

The port information of the server must be entered.

Socket Reuse Address

It is set whether to reuse the socket address in the connection pool.

Socket Keep Alive

It is set whether to keep the socket alive in the connection pool.

IO Threads

The number of threads for the connection pool is entered.

Max Connection Per Host

The maximum number of connections per host in the connection pool is entered.

Max Connection Total

The maximum number of connections that should be in the connection pool is entered.

Enable to administrate cluster

It must be active to apply the Apinizer Index Lifecycle Policies and Index Templates defined by Apinizer.

Apinizer Index Lifecycle Policies

By default, Index Lifecycle Policies defined by Apinizer are managed within this section.

Apinizer Index Templates

By default, Index Templates defined by Apinizer are displayed in this section.

Test Connection

After entering the information of the Elasticsearch cluster, a connection test can be performed by clicking the button.

Editing the Apinizer Index Lifecycle

Phases of default lifecycle policies can be edited when enabled to manage the cluster.

An index lifecycle policy allows defining rules for specific actions such as a rollover or force merge within a specified period of an index and automates the process.

Index lifecycle management defines 4 different phases in which indexes can take place:

  1. Hot phase: It is the phase where active querying and writing operations are performed on the index. A new index is created over the pseudo-index name given to very large or very old indexes. It is generally recommended to be stored in SSD space to achieve the best performance requirements.
  2. Warm phase: In this phase, the index can be queried, but the index is in read-only mode. Shards can be reserved for less performing hardware. For faster searches, the number of shards can be reduced and segments can be merged to reduce the number.
  3. Cold phase: Less queries are made on indexes and queries take longer to be answered. It is still too early to delete data. E.g; In the case of security monitoring, old data may be needed to support its investigation. Therefore, hardware with less performance can be used and the number of replicas can be reduced.
  4. Delete phase: When the index is no longer needed, it can be defined when it is safe to delete.
  • Log politikası için Delete Fazı etkin halde değildir.
  • Trace politikası için Cold Fazı etkinleştirilemez.
  • Token politikası içinse Cold ve Delete Fazı etkinleştirilemez.
  • The Delete Phase is not active for the log policy.
  • The Cold Phase cannot be enabled for the trace policy.
  • For token policy, The Cold Phase and The Delete Phase cannot be enabled.


The image containing the Index Lifecycle settings is given below:


The fields used to configure the Index Lifecycle are shown below.

Field

Description

Policy Name

The policy name information entered to configure Index Lifecycle Management. It is automatically generated over the specific naming theme.

By default, there are 3 index lifecycles;

  • apinizer-log-ilm-policy-{logKey}
  • apinizer-trace-ilm-policy-{logKey}
  • apinizer-token-ilm-policy-{logKey}

Hot Phrase






Enable to Warm phase

It is activated to apply the hot phase to the indexes.

Age information of the index in this phase in days (d) is entered.

Maximum Index Size

The maximum size of the index on the primary shard is entered in gigabytes (gb).

The maximum number of documents in the index. This issue does not include replica shard documents.

Warm Phrase



Enable to Warm phase

It is activated to apply the Warm phase to the indexes.

Number of Primary Shards for Shrink Index

While applying the shrink action, a new index can be created by reducing the number of primary shards in the index. Therefore, the number of primary shards desired to have indexes that pass this phase is entered.

Number of Segments for Force Merge

With the force merge action, the number of segments in the shard containing the indexes is reduced. The desired segment number of an index is entered.

Replica shard number of indexes can be updated.

Cold Phrase


Enable to Cold phase

It is activated to apply the delete phase to indexes.

Minimum Age of Cold Phase

The indices enter the cold phase with the minimum age parameter. The index does not enter phase until it is older than the minimum age. The minimum age of the cold phase is entered in days (d).

Number of Replica Shard

Enter the number of replica shards to which an index will be assigned for the cold phase.

Delete Phrase

Enable to Delete phase

It is activated to apply the delete phase to indexes.

Minimum Age of Delete Phase

Indexes enter the delete phase with a minimum age parameter. The index does not enter phase until it is older than the minimum age. The minimum age of the delete phase is entered in days (d).

You can access our experience with Elasticsearch on our Medium blog.