Analytics Engine
Analytics Engine Concept
Data collection from API traffic
Data analysis and metric calculation
Dashboards and reports
Real-time metrics
Analytics Engine Features
Data Collection
API traffic and all logging are done asynchronously. This way, API Gateway's performance is not affected and logging operations are performed in the background.
Request Logs
- Request URL
- HTTP method
- Request headers
- Request body (optional)
- Timestamp
Response Logs
- Response status code
- Response headers
- Response body (optional)
- Response time
Error Logs
- Error type
- Error message
- Error time
All logging operations are performed asynchronously. This way, API Gateway's response time is not affected and high performance is maintained.
Metrics
- Response time (average, min, max)
- Throughput (requests/second)
- Latency
- Error rate
- API call count
- Unique user count
- Endpoint usage
- Method-based usage
- Error count
- Error rate
- Error type distribution
- Error trend
- Total traffic
- Traffic trend
- Peak hours
- Geographic distribution
Visualization
Dashboard
- Real-time metrics
- Charts and graphs
- Customizable widgets
- Filtering and time range selection
Reports
- Ready reports
- Custom report creation
- Report scheduling
- Report export (PDF, CSV, Excel)
Alerting
- Metric-based alarms
- Threshold definition
- Notification channels (Email, SMS, Webhook)
Analytics Engine Components
Component that collects log records from API Gateways
- Log Aggregation: Collecting log records
- Log Parsing: Parsing log records
- Log Storage: Storing log records
Component that processes collected data and calculates metrics
- Data Transformation: Data transformation
- Metric Calculation: Metric calculation
- Data Aggregation: Data aggregation
Layer where analytics data is stored
- Elasticsearch: Log and metric data
- Time Series Database: Time series data
- Data Retention: Data retention policies
Analytics Engine Usage Scenarios
- API response time monitoring
- Throughput analysis
- Latency trend analysis
- Bottleneck detection
- Most used APIs
- User-based usage
- Endpoint-based usage
- Time-based usage trend
- Error rate monitoring
- Error type analysis
- Error trend analysis
- Error resolution tracking
Analytics Engine and Other Components Relationship
Analytics Engine receives log records from API Gateways asynchronously, processes them, and produces analytics data. Workflow:
API Gateway
│
│ Asynchronous Log Records
▼
Analytics Engine
│
│ Analytics Data
▼
Storage Layer (Elasticsearch)
│
│ Visualization
▼
Dashboard / Reports
Log records are collected asynchronously from API Gateways
Request, Response, and Error logs are collected in the background. API Gateway's performance is not affected.
Log records are processed and metrics are calculated
Data transformation and metric calculation are performed.
Analytics data is saved to storage layer
Log and metric data are stored in Elasticsearch.
Dashboards and reports are created
Users can visualize metrics.
Analytics Engine provides valuable information about API performance and usage by processing log records from API Gateways asynchronously. Thanks to asynchronous logging, API Gateway's response time is not affected and high performance is maintained.
Data Storage and Retention
API Traffic Logs and ILM
API traffic logs are stored in Elasticsearch and Index Lifecycle Management (ILM) is directly supported. ILM policies can be managed through Apinizer. Additionally, logs can be sent to different targets through connectors:
API traffic logs are stored in Elasticsearch
- Real-time log storage
- Search and analysis capabilities
- Scalable storage
- Direct viewing and management on Apinizer screen
- Kibana visualization support
Elasticsearch ILM policies are managed from Apinizer
- Index lifecycle stages (hot, warm, cold, delete)
- Automatic index rollover
- Retention policies
Apinizer Analytics and Reporting
Apinizer provides special analytics and reporting features for API traffic logs stored in Elasticsearch:
- Analytics Dashboard: Direct viewing on Apinizer screen
- Reporting: Creating and managing custom reports
- Real-Time Metrics: Live traffic and performance metrics
- Filtering and Querying: Advanced filtering and querying capabilities
You can view, analyze, and create reports for API traffic logs from the Apinizer screen. All analytics and reporting operations can be managed through Apinizer.
Kibana Integration
If desired, logs in Elasticsearch can be visualized with Kibana:
- Kibana Dashboard: Creating dashboards on Kibana
- Visualization: Kibana's visualization tools
- Advanced Analysis: Kibana's analysis capabilities
Thanks to Kibana integration, logs in Elasticsearch can be used with Kibana's powerful visualization and analysis tools.
Log Sending with Connector
API traffic logs can be sent to different targets through connectors:
- Kafka: Sending logs to message queue
- Syslog: Sending logs to syslog servers
- SIEM Tools: Integration with security information and event management tools
- Relational Database: Writing log records to relational databases
Thanks to connectors, API traffic logs can be sent to multiple targets in parallel. This way, logs are both stored in Elasticsearch and transferred to other systems.
Apinizer Application Logs
Automatic cleanup tasks can be configured for logs created by various components of the Apinizer platform:
Component Logs
- Manager Logs: Software logs of Manager application
- Gateway Logs: Software logs of Gateway application
- Cache Logs: Software logs of Cache application
Audit and Security Logs
- Audit Records: User change records
- ACL Audit Records: API Proxy and group access records
- Token Requests: Token request records
- API Manager User Token Blacklist: Token records kept when logout is performed
Monitoring and Operation Logs
- Alert Logs: Alarm records
- Anomaly Detector Logs: Anomaly detection logs
- Uptime Monitor Logs: Uptime monitoring logs
- Task Flow Logs: Task Flow operation logs
- Report Logs: Report operation logs
- API Proxy Trace Logs: Logs kept when trace feature is enabled
User and Metric Data
- User Notifications: User notifications
- Environment Metrics: Environment (Namespace) metrics
Application log cleanup tasks can be run hourly or daily. Records before the specified date are automatically deleted. For detailed information, see the Application Log Cleanup Tasks page.
Unsent API Traffic Logs
When API traffic logs cannot be sent to the connector, a failover connector is used. If Apinizer configuration database is selected as the failover connector, accumulated data can be transferred to main connectors:
Unsent API traffic logs can be transferred manually. If an error is encountered during the operation, it is stopped halfway without data loss and can be restarted later. For detailed information, see the Transfer Unsent API Traffic Logs page.
Prometheus and Grafana Integration
Analytics Engine metrics can be integrated with Prometheus and Grafana:
Apinizer metrics are exported to Prometheus
- Gateway metrics
- Cache metrics
- Custom PromQL queries
Prometheus data is visualized in Grafana
- Sample dashboards
- Customizable graphs
- Real-time metrics
For detailed information about Prometheus and Grafana integration, see the Integrating Apinizer Metrics with Prometheus and Grafana page.