Elasticsearch
This page explains how to configure Elasticsearch connection settings.
The picture below shows the Elasticsearch Connection definition:
The fields used for Elasticsearch Configuration are shown in the table below.
Field | Description |
---|---|
Enabled | Indicates the active state of the setting. |
Name | The name of the setting to be made. |
Type | The operation to be performed via Elasticsearch API is selected. |
Description | Description can be entered. |
Authenticate | This field is activated to authenticate. |
Username | Username of the Elasticsearch server. |
Password | Password of the Elasticsearch server. |
Scheme Type | Selected as HTTP or HTTPS. |
Host | It is the IP or access address information of the server. |
Port | It is the port information of the server. |
Socket Reuse Address | Socket reuse address information. |
Socket Keep Alive | Socket keep alive information. |
IO Threads | IO threads information |
Max Connection Per Host | The maximum number of connections per host is entered. |
Max Connection Total | The maximum number of connections is entered. |
Connection Timeout | The default timeout value for the Elasticsearch Index API for logging API Traffic requests is set to 5 seconds. |
Enable to administrate cluster | This field is activated so that Apinizer can perform Index creation, Template creation and Index Lifecycle management operations in the Elasticsearch cluster. If this field is active, other administrative fields also become active; if it is inactive, only the index name value to be read/written is needed. |
Index Name | Target Index name. |
Template Name | Target Template name. |
Number of Replicas | This field describes the default value and function of specifying how many copies of the data will exist. The default value is 1, indicating that only the original data will be retained, with no backup. |
Number of Shards | This field explains the number of shards per index. Its default value is set to 1. |
Index Refresh Interval | This field describes a parameter that determines how often the index will be updated, in seconds. Its default value is set to 5. |
Editing the Apinizer Index Lifecycle
Phases of default lifecycle policies can be edited when enabled to manage the cluster.
An index lifecycle policy allows defining rules for specific actions such as a rollover or force merge within a specified period of an index and automates the process.
Index lifecycle management defines 4 different phases in which indexes can take place:
- Hot phase: It is the phase where active querying and writing operations are performed on the index. A new index is created over the pseudo-index name given to very large or very old indexes. It is generally recommended to be stored in SSD space to achieve the best performance requirements.
- Warm phase: In this phase, the index can be queried, but the index is in read-only mode. Shards can be reserved for less performing hardware. For faster searches, the number of shards can be reduced and segments can be merged to reduce the number.
- Cold phase: Less queries are made on indexes and queries take longer to be answered. It is still too early to delete data. E.g; In the case of security monitoring, old data may be needed to support its investigation. Therefore, hardware with less performance can be used and the number of replicas can be reduced.
- Delete phase: When the index is no longer needed, it can be defined when it is safe to delete.
Delete phase, is disabled by default in order to prevent unconsciously log loss.
The picture below shows the Index Lifecycle settings:
The fields used to configure the Index Lifecycle are shown below.
Field | Description | |
---|---|---|
Policy Name | The policy name information entered to configure Index Lifecycle Management. | |
Hot Phrase | Enable to Warm phase | It is activated to apply the hot phase to the indexes. |
Age information of the index in this phase in days (d) is entered. | ||
Maximum Index Size | The maximum size of the index on the primary shard is entered in gigabytes (gb). | |
The maximum number of documents in the index. This issue does not include replica shard documents. | ||
Warm Phrase | Enable to Warm phase | It is activated to apply the Warm phase to the indexes. |
Number of Primary Shards for Shrink Index | While applying the shrink action, a new index can be created by reducing the number of primary shards in the index. Therefore, the number of primary shards desired to have indexes that pass this phase is entered. | |
Number of Segments for Force Merge | With the force merge action, the number of segments in the shard containing the indexes is reduced. The desired segment number of an index is entered. | |
Replica shard number of indexes can be updated. | ||
Cold Phrase | Enable to Cold phase | It is activated to apply the delete phase to indexes. |
Minimum Age of Cold Phase | The indices enter the cold phase with the minimum age parameter. The index does not enter phase until it is older than the minimum age. The minimum age of the cold phase is entered in days (d). | |
Number of Replica Shard | Enter the number of replica shards to which an index will be assigned for the cold phase. | |
Delete Phrase | Enable to Delete phase | It is activated to apply the delete phase to indexes. |
Minimum Age of Delete Phase | Indexes enter the delete phase with a minimum age parameter. The index does not enter phase until it is older than the minimum age. The minimum age of the delete phase is entered in days (d). |
When the Elasticsearch Cluster definiton is deleted, the relevant indexes, the policies and the templates will not be deleted. However, policies belonging to the cluster's Index Lifecycle Management (ILM) disable all phases. Thus, unnecessary resource consumption is be prevented.
For instance; Supposing any index in the deleted cluster is assigned 5 to the minimum age of the index in the hot phase of the ILM policy. A new index will be created every 5 days. This action is considered necessary in order not to increase the number of indexes and shards in the cluster.
Elasticsearch Monitoring
To monitor the status of the Elasticsearch cluster, the monitoring page can be accessed by clicking the "Monitor" option from the menu at the end of the line you want to monitor in the list of Elasticsearch connections.
Click here for detailed information about Elasticsearch Monitor.
Elasticsearch Rollover
To enable Elasticsearch cluster to automatically create a new index and route new data to this new index, you can access the 'Rollover' option from the menu at the end of the row in the list where Elasticsearch connections are located.