Skip to main content

​
Analytics Engine Concept

Data Collection

Data collection from API traffic

Analysis

Data analysis and metric calculation

Visualization

Dashboards and reports

Real-Time

Real-time metrics

​
Analytics Engine Features

​
Data Collection

API traffic and all logging are done asynchronously. This way, API Gateway’s performance is not affected and logging operations are performed in the background.
  • Request URL
  • HTTP method
  • Request headers
  • Request body (optional)
  • Timestamp
  • Response status code
  • Response headers
  • Response body (optional)
  • Response time
  • Error type
  • Error message
  • Error time
All logging operations are performed asynchronously. This way, API Gateway’s response time is not affected and high performance is maintained.

​
Metrics

Performance Metrics

  • Response time (average, min, max)
  • Throughput (requests/second)
  • Latency
  • Error rate

Usage Metrics

  • API call count
  • Unique user count
  • Endpoint usage
  • Method-based usage

Error Metrics

  • Error count
  • Error rate
  • Error type distribution
  • Error trend

Traffic Metrics

  • Total traffic
  • Traffic trend
  • Peak hours
  • Geographic distribution

​
Visualization

  • Real-time metrics
  • Charts and graphs
  • Customizable widgets
  • Filtering and time range selection
  • Ready reports
  • Custom report creation
  • Report scheduling
  • Report export (PDF, CSV, Excel)
  • Metric-based alarms
  • Threshold definition
  • Notification channels (Email, SMS, Webhook)

​
Analytics Engine Components

Log Collector

Component that collects log records from API Gateways
  • Log Aggregation: Collecting log records
  • Log Parsing: Parsing log records
  • Log Storage: Storing log records

Data Processor

Component that processes collected data and calculates metrics
  • Data Transformation: Data transformation
  • Metric Calculation: Metric calculation
  • Data Aggregation: Data aggregation

Storage Layer

Layer where analytics data is stored
  • Elasticsearch: Log and metric data
  • Time Series Database: Time series data
  • Data Retention: Data retention policies

​
Analytics Engine Usage Scenarios

Performance Monitoring

  1. API response time monitoring
  2. Throughput analysis
  3. Latency trend analysis
  4. Bottleneck detection

Usage Analysis

  1. Most used APIs
  2. User-based usage
  3. Endpoint-based usage
  4. Time-based usage trend

Error Analysis

  1. Error rate monitoring
  2. Error type analysis
  3. Error trend analysis
  4. Error resolution tracking

​
Analytics Engine and Other Components Relationship

Analytics Engine receives log records from API Gateways asynchronously, processes them, and produces analytics data. Workflow: 
API Gateway
   │
   │ Asynchronous Log Records
   ▼
Analytics Engine
   │
   │ Analytics Data
   ▼
Storage Layer (Elasticsearch)
   │
   │ Visualization
   ▼
Dashboard / Reports

1. Asynchronous Data Collection

Log records are collected asynchronously from API GatewaysRequest, Response, and Error logs are collected in the background. API Gateway’s performance is not affected.

2. Data Processing

Log records are processed and metrics are calculatedData transformation and metric calculation are performed.

3. Data Storage

Analytics data is saved to storage layerLog and metric data are stored in Elasticsearch.

4. Visualization

Dashboards and reports are createdUsers can visualize metrics.
Analytics Engine provides valuable information about API performance and usage by processing log records from API Gateways asynchronously. Thanks to asynchronous logging, API Gateway’s response time is not affected and high performance is maintained.

​
Data Storage and Retention

​
API Traffic Logs and ILM

API traffic logs are stored in Elasticsearch and Index Lifecycle Management (ILM) is directly supported. ILM policies can be managed through Apinizer. Additionally, logs can be sent to different targets through connectors:

Elasticsearch Storage

API traffic logs are stored in Elasticsearch
  • Real-time log storage
  • Search and analysis capabilities
  • Scalable storage
  • Direct viewing and management on Apinizer screen
  • Kibana visualization support

ILM Support

Elasticsearch ILM policies are managed from Apinizer
  • Index lifecycle stages (hot, warm, cold, delete)
  • Automatic index rollover
  • Retention policies
Apinizer provides special analytics and reporting features for API traffic logs stored in Elasticsearch:
  • Analytics Dashboard: Direct viewing on Apinizer screen
  • Reporting: Creating and managing custom reports
  • Real-Time Metrics: Live traffic and performance metrics
  • Filtering and Querying: Advanced filtering and querying capabilities
You can view, analyze, and create reports for API traffic logs from the Apinizer screen. All analytics and reporting operations can be managed through Apinizer.
If desired, logs in Elasticsearch can be visualized with Kibana:
  • Kibana Dashboard: Creating dashboards on Kibana
  • Visualization: Kibana’s visualization tools
  • Advanced Analysis: Kibana’s analysis capabilities
Thanks to Kibana integration, logs in Elasticsearch can be used with Kibana’s powerful visualization and analysis tools.
API traffic logs can be sent to different targets through connectors:
  • Kafka: Sending logs to message queue
  • Syslog: Sending logs to syslog servers
  • SIEM Tools: Integration with security information and event management tools
  • Relational Database: Writing log records to relational databases
Thanks to connectors, API traffic logs can be sent to multiple targets in parallel. This way, logs are both stored in Elasticsearch and transferred to other systems.

​
Apinizer Application Logs

Automatic cleanup tasks can be configured for logs created by various components of the Apinizer platform:
  • Manager Logs: Software logs of Manager application
  • Gateway Logs: Software logs of Gateway application
  • Cache Logs: Software logs of Cache application
  • Audit Records: User change records
  • ACL Audit Records: API Proxy and group access records
  • Token Requests: Token request records
  • API Manager User Token Blacklist: Token records kept when logout is performed
  • Alert Logs: Alarm records
  • Anomaly Detector Logs: Anomaly detection logs
  • Uptime Monitor Logs: Uptime monitoring logs
  • Task Flow Logs: Task Flow operation logs
  • Report Logs: Report operation logs
  • API Proxy Trace Logs: Logs kept when trace feature is enabled
  • User Notifications: User notifications
  • Environment Metrics: Environment (Namespace) metrics
Application log cleanup tasks can be run hourly or daily. Records before the specified date are automatically deleted. For detailed information, see the Application Log Cleanup Tasks page.

​
Unsent API Traffic Logs

When API traffic logs cannot be sent to the connector, a failover connector is used. If Apinizer configuration database is selected as the failover connector, accumulated data can be transferred to main connectors:
Unsent API traffic logs can be transferred manually. If an error is encountered during the operation, it is stopped halfway without data loss and can be restarted later. For detailed information, see the Transfer Unsent API Traffic Logs page.

​
Prometheus and Grafana Integration

Analytics Engine metrics can be integrated with Prometheus and Grafana:

Prometheus Integration

Apinizer metrics are exported to Prometheus
  • Gateway metrics
  • Cache metrics
  • Custom PromQL queries

Grafana Dashboard

Prometheus data is visualized in Grafana
  • Sample dashboards
  • Customizable graphs
  • Real-time metrics
For detailed information about Prometheus and Grafana integration, see the Integrating Apinizer Metrics with Prometheus and Grafana page.

​
Next Steps

API Traffic

Viewing and analyzing API traffic logs

Prometheus and Grafana

Prometheus and Grafana integration

Application Log Cleanup

Application log cleanup tasks

Data Log Layer

Learn about Data Log Layer architecture