Visualizing API Traffic Logs with Kibana
Kibana is a user interface program used to visualize and analyze Elasticsearch data. Kibana communicates with an Elasticsearch cluster to retrieve data. Kibana enables users to visualize Elasticsearch data with graphs and visual analyses.
One of the key features of Kibana is its ability to monitor logs saved in Elasticsearch in real-time. This allows users to track and analyze log data in real-time.
Kibana is a powerful tool for users to understand, analyze, and share data. With the creation of visual analyses and reports, users can interpret and share data more effectively.
Kibana Installation (Windows)
For a compatible version of Kibana, you can visit https://www.elastic.co/downloads/past-releases/kibana-7-9-2 Basic version.
You can choose Kibana's Basic version or the Free version. The Basic version typically comes with paid subscriptions, while the Free version is an open-source alternative.
For a compatible version of Kibana, you can visit https://www.elastic.co/downloads/past-releases/kibana-oss-7-9-2 Free version.
- After selecting the appropriate version for your operating system from the links above, simply click on the relevant link to initiate the download process. Once the download is complete, you can proceed with the installation of Kibana.
- Extract the downloaded file and save it to the target folder.
- Navigate into the Kibana directory, then enter the bin folder, and copy the file path.
For configuring the server details of Apinizer's Elasticsearch Integration in the 'kibana.yml' file inside the Config folder, you can adjust the information as follows.
You can open Command Prompt (cmd) as an administrator and navigate to the copied Kibana file using the 'cd' command.
cd C:\kibana-7.9.2- To start Kibana, execute the following command with the .bat extension.
- Kibana should start after a few seconds.
bin/kibana.batIf everything is fine, you should see a result like this.
- Running by default, Kibana uses port 5601.
- To check if Kibana is running, you can navigate to "localhost:5061" in your browser.
- Upon logging in, you will see the Kibana interface connected to Elasticsearch
Kibana Installation (Linux)
The Kibana v7.9.2 Linux archive can be downloaded and installed as follows:
We download version 7.9.2 of Kibana using the 'curl' command.
curl -O https://artifacts.elastic.co/downloads/kibana/kibana-7.9.2-linux-x86_64.tar.gzWe download the SHA512 verification file for Kibana, then we verify its content using the SHA512 algorithm.
curl https://artifacts.elastic.co/downloads/kibana/kibana-7.9.2-linux-x86_64.tar.gz.sha512 | shasum -a 512 -c -We extract the Kibana archive file using the 'tar' command.
tar -xzf kibana-7.9.2-linux-x86_64.tar.gzWe navigate to the directory where Kibana has been extracted (kibana-7.9.2-linux-x86_64/).
cd kibana-7.9.2-linux-x86_64/For configuration, the server information for Apinizer's Elasticsearch Integration is edited in the 'kibana.yml' file inside the Config folder.
vi config/kibana.yml
Start Kibana server as below:
./bin/kibanaIf everything is fine, you should see a result like this.
Integration of Apinizer API Traffic Logs With Kibana
Kibana enables the creation of various types of visualizations, including data tables, time series, heat maps, bar charts, area charts, and more. The most appropriate visualization format is selected based on the usage purpose before starting the operations.
The "Overview" section of Apinizer provides the ability to display general information about API Proxies, total request counts, and details such as successful, failed, and blocked requests on a single page. This section allows users to easily monitor and manage API traffic.
Through Kibana integration, users can create customized reports in addition to general information and delve into API traffic in more detail. This enables users to understand API performance and make improvements. Below are examples of several Kibana Visualize and Dashboard samples prepared in a similar manner to the Apinizer Overview section, demonstrating how they are created.
You can visit the Manually Create ILM Policy and Template on Elasticsearch page for the Template Data Structure Table, where you can view the descriptions of the data held by ILM (Index Lifecycle Management) policies according to field names.
Creating Visualizations and Dashboards
The visualizations created with Kibana are consolidated into a visualization panel, providing a comprehensive overview of existing data in the Apinizer Overview section, including more customized data.
To create this visualization, click Visualize link from left menu as below:
Then, click "Create Visulatization" button:
Now you can select the desired chart creation format.
Creating Sample Graph
Creating a graph showing the request count status within the last 7 days.
The graph shows the status of request numbers within the last 7 days, grouped based on timestamps. For each timestamp interval, the unique count of requests in the 'Size Request Total' field has been calculated using the unique count metric. Additionally, the dataset indicates the various statuses present based on the 'Result Status' term.
Field | Aggregation | Apinizer Elasticsearch Index Field Name for Query |
|---|---|---|
Metric (Dimensionality) | Unique Count | sr1t(Size Request Total) |
| Buckets (What Information Will Be in the Data Set) | Date histogram | @timestamp(For Time Interval Data) |
Terms | rt(Result Type) |
- In the Metrics field, the metric s1rt (size of requests) is entered to determine the number of unique items. The number of unique items provides information about the diversity and overall summary of the dataset's content.
- To show the number of requests based on a 7-day time frame, the Date Histogram metric is used along with the @timestamp field, and the x-axis is determined by this time interval. To group the results, the Terms term is used in the Split Series section based on commonly encountered values, grouping the statuses. This method is employed to visualize the distribution of requests over time and the distribution of different statuses.
- Following the steps, it visually represents how frequently certain statuses occur within a specific time period.
Creating a graph showing the request count status by methods
The chart is grouped by API Proxy Method names, displaying the request count for each method. Additionally, different outcome statuses for each method are also shown.
Field | Aggregation | Apinizer Elasticsearch Index Field Name for Query |
|---|---|---|
Metric (Dimensionality) | Count | - |
| Buckets (What Information Will Be in the Data Set) | Terms | apmn(API Proxy Method Name) |
Terms | rt(Result Type) |
- The Metrics field utilizes Count to determine the total number of methods in the dataset, and no field is specified.
- To determine the x-axis, the Buckets field employs apm (API Proxy Method name). Results are grouped based on commonly encountered values using the Terms term in the Split Series section.
- Following the steps, it groups API Proxy Method names and visually presents the request status for each method.
Creating a chart displaying API Proxy usage rates
The chart is grouped by API Proxy names, displaying the usage count for each API Proxy. This chart can be used to determine which API Proxies are being utilized more.
Field | Aggregation | Apinizer Elasticsearch Index Field Name for Query |
|---|---|---|
Metric (Dimensionality) | Count | - |
| Buckets (What Information Will Be in the Data Set) | Terms | apn(API Proxy Name) |
- Count is used in the Metrics field to determine the total number of proxies in the dataset, and no field is specified.
- In the Split Series section, proxies are grouped using the Terms term.
- The Buckets section indicates restrictions regarding the number of API Proxies and how they will be grouped according to API Proxy names. The Metrics section determines the slice size.
Creating a chart showing the request count and status by API Proxies
This chart illustrates the most frequent associations of API Proxies with different statuses. For instance, it can be used to determine under which circumstances an API Proxy tends to fail more often or succeed more frequently.
Alan | Aggregation | Apinizer Elasticsearch Index Field Name for Query |
|---|---|---|
Metric (Dimensionality) | Count | - |
| Buckets (What Information Will Be in the Data Set) | Terms | apn(API Proxy Name) |
| Terms | rt(Result Type) |
- Count is used in the Metrics field to determine the total number of API proxies in the dataset, and no field is specified.
- To determine the x-axis, the Buckets field employs apn (API Proxy name). Results are grouped based on commonly encountered values using the Terms term in the Split Series section, grouping the Result statuses.
- Following the steps, it groups requests by API Proxy names and visually presents the request status for each Proxy.
Creating a Histogram graph displaying requests by status code
This histogram graph will display the number of requests by status codes. By determining how many requests occurred for each status code, you can visualize the distribution of requests. This allows you to observe which status codes are more or less frequent.
Field | Aggregation | Apinizer Elasticsearch Index Field Name for Query |
|---|---|---|
Metric (Dimensionality) | Count | - |
| Buckets (What Information Will Be in the Data Set) | Terms | sc(Status Code) |
- Count is used in the Metrics field to determine the total number of requests in the dataset, and no field is specified.
- sc is used in the Buckets field to determine the status code based on the total number of requests.
- This visualization clearly displays the number of requests with different status codes. For example, it may include the proportion of requests with a 200 (Successful) status code out of the total requests, the number of requests with a 404 (Not Found) status code, the number of requests with a 500 (Server Error) status code, and so on.
Creating Dashboard Panels and Adding Visualizations
Click Dashboard link from left menu:
Click "Create Dashboard" button:
The area marked in red is used to add any existing visualization created within it.
The area marked in green, on the other hand, is used to create a new visualization and add it to the dashboard panel.
Adding Saved Visualizations to the Dashboard Panel
Data Analysis and Filtering
The section that allows for analyzing data stored in indexes, obtaining detailed information about the structure of each field, and visualizing findings. Additionally, searches can be personalized, saved, and these customized search and filtering options can be placed into a control panel.
The chart above shows the total number of requests within the last 15 minutes. Below the bars, there is a list of Elasticsearch documents returned by the search.
The filtering process based on the time interval can be optionally performed and visualized.
Alongside the data, all fields can be displayed on a row-by-row basis. Clicking on the arrow icon to expand the row provides you with details in either table format or JSON format.
JSON Format
The desired data can be formatted into a table. Clicking on the red marked box allows you to add that data in table format when expanding a row.
Clicking on the box marked in green allows you to filter the data. This functions similarly to KQL (Kibana Query Language) queries.
KQL is the query language used for searching and analyzing data stored in Elasticsearch. When simple queries are written, they are automatically converted into Elasticsearch DSL Query format in the background for searching. Complex Elasticsearch queries can be performed using KQL in a single line.
Querying the Number of Successful Requests in the Last 24 Hours (status code: 200)
After saving the search data, it can be shared with others using the share button located in the top right corner, offering various sharing options.
You can visit the Manually Create ILM Policy and Template on Elasticsearch page for the Template Data Structure Table, where you can view the descriptions of the data held by ILM (Index Lifecycle Management) policies according to field names.