Gateway Performance Tuning
Overview
Gateway performance depends on several factors:
- Hardware resources: CPU core count, memory amount
- Traffic profile: Concurrent request count, request size, SSE/streaming usage
- Policy complexity: Number and weight of applied policies (e.g., CPU-intensive policies like Content Filtering)
- Backend response time: Latency profile of the target service
Tuning approach — priority order:
- Automatic profile — Sufficient for most environments, no intervention needed
- Tier recommendations — Ready-made parameter sets based on resource size (this page)
- Manual tuning — Fine-tuning based on load test results
Tier-Based Recommended Settings
The following tables show recommended Gateway Worker settings for different hardware profiles. These values should be validated with pre-production load tests.
Tier 1 — 2 Core / 2 GB RAM
Suitable for low-traffic environments, development, and PoC scenarios.
| Parameter | Recommended Value |
|---|---|
tuneIoThreads | 2 |
tuneWorkerThreads | 512 |
tuneWorkerMaxThreads | 1024 |
tuneRoutingConnectionPoolMaxConnectionTotal | 2048 |
tuneAsyncExecutorCorePoolSize | 512 |
tuneAsyncExecutorMaxPoolSize | 1024 |
tuneAsyncExecutorQueueCapacity | 1000 |
Tier 1 Environment Variable Block Example:
tuneIoThreads=2
tuneWorkerThreads=512
tuneWorkerMaxThreads=1024
tuneRoutingConnectionPoolMaxConnectionTotal=2048
Tier 2 — 4 Core / 4 GB RAM
Recommended for medium-scale production environments. A balanced starting point for most deployments.
| Parameter | Recommended Value |
|---|---|
tuneIoThreads | 4 |
tuneWorkerThreads | 1024 |
tuneWorkerMaxThreads | 2048 |
tuneRoutingConnectionPoolMaxConnectionTotal | 4096 |
tuneAsyncExecutorCorePoolSize | 1024 |
tuneAsyncExecutorMaxPoolSize | 2048 |
tuneAsyncExecutorQueueCapacity | 1000 |
Tier 2 Environment Variable Block Example:
tuneIoThreads=4
tuneWorkerThreads=1024
tuneWorkerMaxThreads=2048
tuneRoutingConnectionPoolMaxConnectionTotal=4096
Tier 3 — 8 Core / 8 GB RAM
Recommended for high-traffic production environments.
| Parameter | Recommended Value |
|---|---|
tuneIoThreads | 8 |
tuneWorkerThreads | 2048 |
tuneWorkerMaxThreads | 4096 |
tuneRoutingConnectionPoolMaxConnectionTotal | 8192 |
tuneAsyncExecutorCorePoolSize | 2048 |
tuneAsyncExecutorMaxPoolSize | 4096 |
tuneAsyncExecutorQueueCapacity | 2000 |
Tier 3 Environment Variable Block Example:
tuneIoThreads=8
tuneWorkerThreads=2048
tuneWorkerMaxThreads=4096
tuneRoutingConnectionPoolMaxConnectionTotal=8192
JVM Parameters: Heap, direct memory, and metaspace values are set automatically based on the container memory limit (no need to add JAVA_OPTS). Heap settings in JAVA_OPTS are not applied unless jvmOverride=true is set. For the tier tables and manual GC configuration, see JVM Garbage Collector Tuning.
Thread Tuning
Worker Threads
Worker threads are the core thread pool that processes incoming HTTP requests.
| Parameter | Default | Description |
|---|---|---|
tuneWorkerThreads | CPU core count × 8 | Minimum Worker thread count — threads created at server startup |
tuneWorkerMaxThreads | CPU core count × 16 | Maximum Worker thread count — upper limit the pool can grow to under load |
If the environment variable is not set, the Gateway uses the defaults above based on the CPU core count visible to the pod.
Recommended Thread Values by CPU
| CPU Cores | tuneWorkerThreads | tuneWorkerMaxThreads |
|---|---|---|
| 1 | 512 | 1024 |
| 2 | 1024 | 2048 |
| 4 | 2048 | 4096 |
| 8 | 4096 | 8192 |
General rule: tuneWorkerThreads ≈ CPU × 512, tuneWorkerMaxThreads ≈ CPU × 1024. These values are based on Undertow's (the HTTP server used by Gateway) thread management model.
Virtual Threads
| Parameter | Default | Description |
|---|---|---|
tuneRoutingVirtualThreadsEnabled | true | Runs backend routing calls on virtual threads. When set to false, routing calls run on the worker thread that accepted the request; in addition, the Async Executor settings below start to apply. |
tuneLoggingVirtualThreadsEnabled | true | Runs log writes to connectors (Elasticsearch bulk, webhook, etc.) on virtual threads. |
IO Threads
IO threads are low-level threads that handle network I/O operations (socket read/write).
| Parameter | Default | Description |
|---|---|---|
tuneIoThreads | CPU core count | IO Thread count |
IO thread count is typically kept equal to the CPU core count. Increasing it provides no benefit in most scenarios; it may cause context switching overhead.
Async Executor Thread Pool
Asynchronous operations such as RestApi Policy, Script Policy, logging, and traffic mirroring use this separate thread pool.
| Parameter | Default | Description |
|---|---|---|
tuneAsyncExecutorCorePoolSize | Same as tuneWorkerThreads | Number of threads kept in the pool |
tuneAsyncExecutorMaxPoolSize | Same as tuneWorkerMaxThreads | Maximum thread count (see the note below) |
tuneAsyncExecutorQueueCapacity | tuneMaxQueueSize if > 0, otherwise 1000 | Queue capacity (see the note below) |
These settings are ignored while virtual threads are enabled (the default). With tuneRoutingVirtualThreadsEnabled=true, the async executor pool is started with a fixed 64 core threads (the maximum is set to 256, but as explained below the pool never grows beyond its core size); the tuneAsyncExecutorCorePoolSize and tuneAsyncExecutorMaxPoolSize values are ignored. These parameters apply only when tuneRoutingVirtualThreadsEnabled=false.
In addition, because the async executor is a scheduling-capable pool, its queue is always unbounded and the pool never grows beyond its core size. In practice the effective value is tuneAsyncExecutorCorePoolSize; tuneAsyncExecutorMaxPoolSize and tuneAsyncExecutorQueueCapacity have no effect at runtime.
Connection Pool Tuning
Routing Connection Pool
Manages HTTP connections to backend services. These values are critical for high concurrent request volumes.
| Parameter | Default | Description |
|---|---|---|
tuneRoutingConnectionPoolMaxConnectionPerHost | 1024 | Maximum connections to a single backend host |
tuneRoutingConnectionPoolMaxConnectionTotal | 2048 | Total maximum connections to all backend hosts |
When to increase:
- When 503 errors or connection timeout logs are observed
- When backend services respond slowly and connections are exhausted
- When concurrent request count exceeds current pool limits
Cache Connection Pool
Manages connections to cache services.
| Parameter | Default | Description |
|---|---|---|
tuneCacheConnectionPoolMaxConnectionTotal | 256 | Cache Connection Pool total maximum connection count |
Elasticsearch Connection Pool
Manages the connections used to write API traffic logs to Elasticsearch.
| Parameter | Default | Description |
|---|---|---|
tuneElasticsearchConnectionPoolMaxConnectionPerHost | 256 | Maximum connections per Elasticsearch host |
API Call Connection Pool
Manages external API calls made by RestApi Policy and similar policies.
| Parameter | Default | Description |
|---|---|---|
tuneApiCallConnectionPoolMaxConnectionPerHost | 256 | Maximum connections per host |
tuneApiCallConnectionPoolMaxConnectionTotal | 4096 | Total maximum connection count |
Timeout Tuning
| Parameter | Default | Description |
|---|---|---|
tuneReadTimeout | 30000 ms (30 sec) | Client data read timeout. Connection is closed if client stops sending data |
tuneStreamingReadTimeout | 0 ms (unlimited) | Client-side read timeout for SSE/LLM streaming connections |
tuneNoRequestTimeout | 60000 ms (60 sec) | No-request timeout after connection is established |
Relationship Between Timeout Parameters
Client connection → tuneNoRequestTimeout (waiting for request)
→ Request received → tuneReadTimeout (reading data)
→ Streaming? → tuneStreamingReadTimeout
tuneNoRequestTimeout: Connection opened but no HTTP request sent — cleans up idle connectionstuneReadTimeout: Client stopped sending data during request processing — cleans up stuck requeststuneStreamingReadTimeout: For long-lived connections like SSE, Server-Sent Events, and LLM streaming, the normaltuneReadTimeoutis insufficient
SSE/LLM Streaming Scenarios:
In streaming connections, clients don't send data for extended periods, so the normal tuneReadTimeout may prematurely close the connection. tuneStreamingReadTimeout assigns a dedicated timeout value for streaming connections. The default value of 0 (unlimited) is suitable for most scenarios; however, you may set an appropriate upper limit for your environment to prevent resource leaks.
Benchmark Reference
For tier-based performance comparisons and detailed benchmark results, see Capacity Planning.
Benchmark results were measured under ideal conditions (fast backend, minimal network latency, simple policy chain). Always perform load tests based on your own traffic patterns for production environments.
Related Pages
Automatic memory tiers, GC selection, jvmOverride, and heap configuration
Hardware sizing, RAM calculation, and benchmark results
UI-based configuration and parameter reference for gateway environments
Monitoring thread usage over time and alert mechanisms