Ana içeriğe geç

Memory Leaks and OOM (Out of Memory) Errors

Problem Symptoms​

Warning

Memory leaks and OOM errors usually manifest themselves with the following symptoms:

  • Pods continuously restarting
  • OutOfMemoryError log records
  • Slowly increasing memory usage
  • Decrease in system performance
  • Pods transitioning to OOMKilled state in Kubernetes

Problem Causes​

Memory leaks and OOM errors can usually be caused by the following factors:

  • Insufficient Heap Settings: JVM heap size may be insufficient
  • Memory Leaks: Unused objects not being cleaned by garbage collection
  • Large Data Processing: Processing large message bodies
  • Cache Sizes: Cache growing unlimited
  • Connection Pool Issues: Connections not being closed
  • Thread Pool Issues: Threads not being properly cleaned
  • Metaspace Ceiling: The Metaspace upper limit, determined by the container memory limit, filling up (especially on low-memory Workers running the OpenTelemetry Java agent)

Detection Methods​

1. Log Analysis​

Search for OOM errors in log files:

kubectl logs <pod-name> | grep -i "OutOfMemoryError"
kubectl logs <pod-name> | grep -i "java.lang.OutOfMemoryError"

2. Monitoring Memory Metrics​

Monitor memory usage with Prometheus metrics:

  • jvm_memory_used_bytes
  • jvm_memory_max_bytes
  • container_memory_usage_bytes

3. Heap and Thread Dump Analysis​

Apinizer images are JRE-based (Eclipse Temurin 25 JRE); JDK tools such as jmap and jstack are not available inside the container. Heap and thread dumps are taken through the Worker and Cache diagnostics endpoints (/apinizer/diagnostics/heapdump, /apinizer/diagnostics/threaddump). A heap dump is taken only from the pod the request reaches; it is not broadcast to other pods.

For command examples and authorization details, see the Diagnostic Endpoint Addresses page.

Solution Recommendations​

1. Optimizing JVM Heap Settings​

Apinizer modules set heap, direct memory, and metaspace values automatically based on the container memory limit (1 GB tiers).

Note

Recommendations:

  • By default, Worker, Manager, Integration, and Portal use G1GC; Cache uses ZGC at 2 GB and above. Worker heap is between 50% and 65% depending on the memory tier.
  • Settings in JAVA_OPTS that the tier also sets, such as -XX:MaxRAMPercentage, are not applied unless jvmOverride=true is set; the tier values win.
  • If you need your own heap value, set it together with jvmOverride=true. When ZGC is selected, MaxRAMPercentage values above 70% are automatically lowered to 70% (except in Cache).
  • See JVM Garbage Collector Tuning for the detailed tier tables and GC options.
env:
  - name: JAVA_OPTS
    value: "-XX:MaxRAMPercentage=70.0"
  - name: jvmOverride
    value: "true"

2. Increasing Memory Limits​

Increase memory limits in Kubernetes deployment:

resources:
  limits:
    memory: "4Gi"
  requests:
    memory: "2Gi"
OpenTelemetry Java agent and Metaspace

If you see java.lang.OutOfMemoryError: Metaspace in the logs, check the container memory limit and the Metaspace ceiling. The Metaspace ceiling is set in tiers based on the memory limit (1 Gi → 128m, 2 Gi → 192m, 4 Gi → 256m), and heap settings do not grow this area. On Workers running the OpenTelemetry Java agent, the memory limit must be at least 2 Gi; raising only -XX:MaxMetaspaceSize without increasing the pod limit increases the OOMKilled risk. See OpenTelemetry integration (Part 1) for details.

3. Optimizing Cache Configuration​

Set cache TTL and size limits:

  • Optimize cache TTL values
  • Determine cache size limits
  • Clean unnecessary cache entries

4. Optimizing Connection Pool Settings​

Check connection pool settings:

  • Limit maximum connection count
  • Set connection timeout values
  • Regularly clean idle connections

5. Thread Pool Management​

Optimize thread pool settings:

  • Limit maximum thread count
  • Set thread timeout values
  • Ensure threads are properly cleaned

6. Optimizing Large Message Processing​

When processing large message bodies:

  • Use streaming
  • Determine message size limits
  • Avoid unnecessary data copying

Preventive Measures​

1. Regular Monitoring​

  • Regularly monitor memory usage
  • Get early warning by setting up alerts
  • Detect problems in advance by performing trend analysis

2. Load Testing​

  • Perform regular load tests
  • Detect memory leaks early
  • Perform capacity planning

3. Code Review​

  • Review code that may cause memory leaks
  • Check resource management
  • Follow best practices
En son tarihinde güncellendi