Pod Diagnostics

Limits

Procedure

1. Log on to the Cloud Monitor console, select the target workspace, and in the left-side navigation pane, click All Features.

2. On the All Features page, click CloudLens for ECS.

3. At the top of the page, click System Diagnostics (SysOM).

4. In the Diagnostics Mode section, select Pod Diagnostics. Select the cluster, Type of diagnosis, and Pod Namespace, and then click Run Diagnostics.

5. In the Diagnostic Records section, click View Report.

Use Cases

Case 1: High WorkingSet anomaly alert

Scenario overview

In Kubernetes, real-time container memory usage (Pod Memory) is measured by WorkingSet size (WSS). WSS is a key metric for scheduling decisions such as node eviction, and users often set alert thresholds on it. However, WSS alerts can produce false positives — for example, WSS may appear high while most usage comes from the active file cache, which can be reclaimed without affecting available memory.

Problem description

The customer received frequent ARMS alerts indicating that WSS usage exceeded 80%, yet the service ran normally — memory allocation behaved as expected and no OOM events occurred.

Memory monitoring shows that cache contributed the most to the WSS increase. According to the Kubernetes documentation, WSS equals cgroup memory usage minus the inactive file cache. If WSS is mostly cache, the active file cache is high, indicating frequent file reads.

Diagnostics analysis

For this scenario, you can run the Pod Diagnostics – Memory Overview feature on the target pod and review the report:

The memory distribution report shows the active memory share. Combined with heap memory usage, this value matches the WSS shown in monitoring.

The file cache list reveals that the largest cache contributor is a log file. Three processes have this file open: one writes the log, and two read and parse it.

The active file cache holds parts of files that are accessed frequently — a file read only once has its cache marked inactive. Here, two processes repeatedly read and parsed the same log file, keeping all its cache active. Because WSS includes the active file cache, working memory appeared high and triggered the alert, even though the service was running normally.

Case 2: OOM caused by burst memory allocation, but monitoring shows low memory usage

Scenario overview

During traffic spikes, a pod may require a sudden increase in memory. If allocation quickly reaches the pod’s memory limit, an OOM event occurs. Monitoring often shows low usage at that time because it samples every 15, 30, or 60 seconds and misses short-lived bursts.

Problem description

The customer observed an OOMKilled event in the ACS Event Center, meaning the pod exceeded its memory limit and the kernel’s oom-killer terminated the highest-memory process. To check memory usage at the time of the event, the customer reviewed the pod monitoring data.

Pod monitoring shows a memory request of 4 GiB. Between 17:05 and 17:30, utilization stayed below 15% with no sign of high usage.

WorkingSet usage also remained low and the cache did not grow — contradicting the OOM event and suggesting a burst allocation faster than the sampling interval. The question is which process caused the OOM.

Diagnostics analysis

For this scenario, you can run the Pod Diagnostics – OOM Diagnostics feature on the target pod and review the report:

The report confirms an OOM event around 17:15. OOMKilled events appear with a small delay after the actual event. The triggering process was a Python application — the container was running 20 Python processes using 3.9 GiB of memory, close to the limit.

The detailed report shows that most memory was anonymous (mainly heap from business processes). The task list at the time of the OOM shows the top Python process used about 3.43 GiB, while other processes used only 50–60 MiB each.

This confirms that a sudden memory request from the Python application caused the OOM event.

Case 3: Repeated OOM in a container, with incomplete memory recovery after each OOM

Scenario overview

Shared memory enables efficient inter-container communication or data sharing — for example, in ACS you can use it by mounting an EmptyDir volume. Because shared memory is independent of processes, it is not released when a process is OOM-killed. Without cleanup, this leads to shared memory leaks.

Problem description

The customer saw an OOMKilled event in the ACS Event Center and checked the pod monitoring data to investigate.

Pod monitoring shows six clear memory spikes likely linked to the OOM events.

[Mon Sep 22 06:48:02 2025] fngi invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=-20
[Mon Sep 22 06:49:05 2025] fngi invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=-20
[Mon Sep 22 06:55:15 2025] fngi invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=-20
[Mon Sep 22 07:03:39 2025] fngi invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=-20
[Mon Sep 22 07:06:35 2025] fngi invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=-20
[Mon Sep 22 07:33:35 2025] fngi invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=-20
[Mon Sep 22 07:46:48 2025] fngi invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=-20

The container’s dmesg logs show seven OOM events. Note that the container’s internal time is UTC+8.

The following patterns emerge:

1. Not every OOM event reached the memory limit. One OOM event had no matching spike in monitoring, which indicates a burst memory request.

2. After each OOM event, memory usage dropped but trended upward overall. This suggests an incomplete release and a possible memory leak.

Diagnostics analysis

Analyze the latest OOM

First, you can run the Pod Diagnostics – OOM Diagnostics feature on the target pod and review the report:

The report shows that the latest OOM event occurred because the shared memory used 3 GiB and the application used 1 GiB, which reached the memory limit.

Analyze historical OOM events

Because OOM events occurred multiple times and the latest involved high shared memory plus 1 GiB from the application, analyzing earlier events helps identify root causes.

When you run OOM Diagnostics, you can specify a time to analyze a historical OOM event. The tool analyzes the OOM event that occurred closest to the specified time.

The report shows that the OOM at 15:06 was caused mainly by high application memory usage, with little shared memory involved.

Analyze shared memory using Memory Overview

The first two reports show that burst memory requests caused periodic OOM events and that shared memory leaked — after each OOM, memory did not fully recover. Use Memory Overview to check shared memory usage.

The Memory Overview report shows many small shared memory files tied to the application. Manually deleting these leaked files freed memory, as confirmed by monitoring data.

This confirms that burst memory requests caused the OOM events, each of which leaked small shared memory files. Memory was released only after these files were manually deleted.