Pod diagnosis best practices
The pod diagnosis feature helps you diagnose pods deployed in ACK or ACS clusters. It currently covers memory-related scenarios, such as memory panoramic analysis, out-of-memory (OOM) diagnosis, and Java application memory diagnosis. This topic describes the best practices for using the pod diagnosis feature in the Alinux console and provides three case studies.
Scope
This feature is available for ACK and ACS instances in all regions in the Chinese mainland except for the Qingdao region. The supported cluster types are:
ACK managed clusters
ACK Serverless clusters
ACS clusters
The following operating systems are also required:
Architecture | Operating system |
x86 architecture |
|
ARM architecture | Not supported. |
Procedure
If you use a RAM user, ensure that your Alibaba Cloud account has granted the permissions to manage RAM users by attaching theAliyunECSReadOnlyAccessAliyunSysomFullAccess
Diagnose in the Alinux console
Go to the Operating System Console > System Diagnosis > Pod Diagnosis page.
Select the cluster, diagnosis type, diagnosis item, namespace, and the name of the pod that you want to diagnose.
Click Execute Diagnosis. Then, in the Diagnosis Records section, click View Report.
After the diagnosis is complete, the diagnosis status of the corresponding record in the Diagnosis Records table changes to Completed, and the Diagnosis Result column indicates whether an exception was detected.
Access diagnosis from the ACS console
Log on to the ACS console. In the left sidebar, choose .
Select Pod Memory Panoramic Diagnosis or Pod OOM Diagnosis as needed.
Case study 1: Anomaly alert for high WorkingSet usage
Scenario description
In Kubernetes, the real-time memory usage of a container is measured by its working set (WSS). Kubernetes uses the WSS to make scheduling decisions about memory resources, such as node eviction. Users often set a monitoring threshold for WSS to trigger alerts. However, WSS alerts can sometimes be false positives. This can happen when WSS indicates high usage, but most of that usage is active file cache. The total available memory is still sufficient because a large amount of page cache can be reclaimed.
Problem description
A customer received frequent alerts from a rule configured in Application Real-Time Monitoring Service (ARMS). The alerts indicated that WSS usage exceeded 80%. However, their services were not significantly affected. Memory could be requested and used normally, and no out-of-memory (OOM) events occurred.
Memory monitoring data showed that the increase in WSS was mainly caused by cache. According to the official Kubernetes documentation, WSS is the working set of a container in a pod. It is calculated as the cgroup memory usage of the container minus the inactive file cache. Therefore, if most of the WSS is cache, it means that the active file cache usage is high, which indicates that a file is being read frequently.
Diagnostic analysis
For this scenario, you can use the Memory Panoramic Analysis feature of Pod Diagnosis in the Alibaba Cloud Linux console to diagnose the target pod. The diagnosis results are as follows.
The memory distribution result shows that the active memory is 3.52 GB. This amount, plus the heap memory usage, is roughly consistent with the monitored WSS.
In the Container File Cache Usage Ranking list, the /var/log/ntgh_data.log file occupies 3.96 GB of cache, far more than any other file. Its associated tasks are ntgh_processor(3494), ntgh_logp(3495), and ntgh_logp(3496).
A closer look at the file cache usage list reveals that a log file occupies the largest portion of the container cache. Three processes have this log file open. One process generates the logs, while the other two read and parse the log file.
The active file cache contains frequently accessed data. If a file is read only once, its cache becomes inactive. If most of a file's cache is active, it is likely that multiple processes are repeatedly reading the same file content. In this case, a log file was generated inside the container. Two processes repeatedly read and parsed this file, causing its cache to become active. Because WSS includes active file cache, the container's working set usage appeared high and triggered an alert, even though the service was running normally.
Case study 2: OOM caused by a sudden memory request, but monitoring shows memory was not full
Scenario description
In some scenarios, a pod may need a sudden burst of memory when service traffic spikes. If the memory request is large and quickly reaches the pod's limit, it can trigger an OOM event. When the OOM event occurs, monitoring data often shows that the memory was not fully used. This happens because monitoring data is sampled at intervals, such as every 15, 30, or 60 seconds, and a sudden memory request can be missed between samples.
Problem description
The alert record shows that the target pod is named fina-migrate-599b684c85-6dlfs, the severity is Warning, the event occurred 2 times in the event history, and the event time is 2025-09-18 17:19:14.
A customer observed an OOMKilled event for a pod in the ACS Event Center. This event indicated that the pod's memory usage exceeded its limit, which triggered the system's oom-killer to terminate the process with the highest memory usage. To see the actual memory usage when the event occurred, the customer checked the pod's monitoring data.
The pod monitoring data shows that the target container's memory request was 4 GB. Between 17:05 and 17:30, the monitored memory utilization of the container remained below 15%, with no high usage observed.
The observed WSS usage also remained low, and the cache did not grow. This led to a counterintuitive situation: an OOM event occurred, but monitoring did not show memory exhaustion. This usually happens when a process makes a sudden memory request, and the time between the request and the OOM event is shorter than the monitoring sampling period. The question is how to determine what happened at the time of the OOM event and identify which process used too much memory.
Diagnostic analysis
For this scenario, you can use the OOM Diagnosis feature of Pod Diagnosis in the Alibaba Cloud Linux console to diagnose the target pod. The diagnosis results are as follows.
The diagnosis result shows that the target pod experienced an OOM event around 17:15. (The OOMKilled event is generated with a delay, a few minutes after the actual OOM event occurs). The process that triggered the OOM event was a Python application. At that time, 20 Python processes were running inside the container, with a total memory usage of 3.9 GB, which was close to the container's memory limit.
The detailed report shows that at the time of the OOM event, the main memory usage was normal anonymous memory, which was mostly the application's heap memory. The task list from that time shows that the top Python process had requested about 3.43 GB of memory, while the other Python processes were using only 50 to 60 MB each.
This analysis confirms that a sudden memory request from a Python application process in the target pod caused the OOM event.
Case study 3: Continuous OOM events, and memory is not fully recovered after each event
Scenario description
In Kubernetes, shared memory is often used for efficient cross-container communication or data sharing. In ACS, for example, you can use shared memory by mounting an EmptyDir volume. This memory is independent of any process. If a process is killed by an OOM event, the shared memory it was using is not released. This can lead to a memory leak if the shared memory is not properly cleaned up.
Problem description
A customer first observed an OOMKilled event for a pod in the ACS Event Center. For further analysis, they checked the pod's monitoring data.
The alert record shows that the target pod is named fgni-64cdc65b87-x89xv, the severity is Warning, the event occurred 7 times in the event history, and the event time is 2025-09-22 15:46:48.
The pod monitoring data shows six distinct memory spikes, which are likely related to OOM events.
A further look at the container's dmesg logs shows seven OOM events during this period. (Due to a time zone difference, add 8 hours to the container's internal time to get the actual time).
The preceding information reveals the following:
Not every OOM event reached the memory limit. One OOM event did not correspond to a spike in the memory monitoring graph, which indicates a sudden memory request.
After each OOM event, memory usage dropped, but the overall trend was upward. This suggests that memory was not fully released and that a leak was occurring.
Diagnostic analysis
Diagnose the latest OOM event
For this scenario, you can first use the OOM Diagnosis feature of Pod Diagnosis in the Alibaba Cloud Linux console to diagnose the OOM event for the target pod. The diagnosis results are as follows.
The diagnosis report shows that the OOM event occurred at 2025-09-22 15:46:48. The task that triggered the OOM event was fngi(5877), which belonged to the fgni-64cdc65b87-x89xv pod and the fgni container. The report recommends that you use File Cache Diagnosis to further analyze the file cache usage of shared memory.
The task list at the time of the OOM event shows that the fngi process (PID 5877) had the highest user-mode memory usage (RSS) at 1,022.44 MB. Other processes, such as nginx (about 6 to 9 MB) and bash (4.03 MB), used much less memory.
The diagnosis result shows that the latest OOM event on the target pod was caused by 3 GB of shared memory usage and 1 GB of application memory usage, which together reached the memory limit.
Diagnose historical OOM events
Multiple OOM events occurred. The latest one was caused by excessive shared memory usage combined with 1 GB of application memory. To better analyze the root cause, you can diagnose historical OOM events.
On the Pod Diagnosis configuration page, set Diagnosis Mode to Pod Diagnosis, Diagnosis Type to Memory Diagnosis, and Diagnosis Item to OOM Diagnosis. In Diagnosis Time, specify the target time, such as 2025-09-22 15:10:00, and then click Run Diagnosis.
When you start an OOM diagnosis, you can specify a time to diagnose a historical event. The tool analyzes the OOM event closest to the specified time. The result is as follows:
The diagnosis result shows that the OOM event occurred at 15:06:35 and was triggered by the fngi task (PID 4368, Pod: fgni-64cdc65b87-x89xv). The total unreclaimable memory of the container reached the 4.00 GB limit, of which normal anonymous memory (RSS) accounted for 3.49 GB and shared memory (shmem) accounted for only 524.65 MB.
The diagnosis result shows that the OOM event at 15:06 was mainly caused by high memory usage from the application. At that time, shared memory usage was not high.
Use memory panoramic analysis to check shared memory
The first two diagnosis results show that the application process had sudden memory requests that caused periodic OOM events. They also show a shared memory leak, because memory was not fully released after each event. Next, you can use Memory Panoramic Analysis to analyze the shared memory usage.
The Memory Panoramic Analysis result table shows multiple TMPFS cache files in shared memory. All the file paths are located in the kubelet pods directory, and the file names match the fgni_* pattern. The cache size of each file is about 29 MB, with a maximum of 29.95 MB, and the associated task column is empty.
The Memory Panoramic Analysis result shows many small, application-related shared memory files in the container. After these leaked files were manually deleted, monitoring data confirmed that the memory was released.
This analysis confirms that sudden memory requests from an application process caused OOM events in the target pod. These OOM events were accompanied by leaks of many small shared memory files. The memory was released only after the leaked files were manually deleted.