The pod diagnosis feature helps you diagnose pods deployed in ACK or ACS clusters. It currently covers memory-related scenarios, such as memory panoramic analysis, out-of-memory (OOM) diagnosis, and Java application memory diagnosis. This topic describes the best practices for using the pod diagnosis feature in the Alinux console and provides three case studies.
Scope
This feature is available for ACK and ACS instances in all regions in the Chinese mainland except for the Qingdao region. The supported cluster types are:
ACK managed clusters
ACK Serverless clusters
ACS clusters
The following operating systems are also required:
Architecture |
Operating system |
x86 architecture |
|
ARM architecture |
Not supported. |
Procedure
If you use a RAM user, ensure that your Alibaba Cloud account has granted the permissions to manage RAM users by attaching theAliyunECSReadOnlyAccessAliyunSysomFullAccess
Diagnose in the Alinux console
Go to the Operating System Console > System Diagnosis > Pod Diagnosis page.
Select the cluster, diagnosis type, diagnosis item, namespace, and the name of the pod that you want to diagnose.
Click Execute Diagnosis. Then, in the Diagnosis Records section, click View Report.
Access diagnosis from the ACS console
Log on to the ACS console. In the left sidebar, choose .
Select Pod Memory Panoramic Diagnosis or Pod OOM Diagnosis as needed.
Case study 1: Anomaly alert for high WorkingSet usage
Scenario description
In Kubernetes, the real-time memory usage of a container is measured by its working set (WSS). Kubernetes uses the WSS to make scheduling decisions about memory resources, such as node eviction. Users often set a monitoring threshold for WSS to trigger alerts. However, WSS alerts can sometimes be false positives. This can happen when WSS indicates high usage, but most of that usage is active file cache. The total available memory is still sufficient because a large amount of page cache can be reclaimed.
Problem description
A customer received frequent alerts from a rule configured in Application Real-Time Monitoring Service (ARMS). The alerts indicated that WSS usage exceeded 80%. However, their services were not significantly affected. Memory could be requested and used normally, and no out-of-memory (OOM) events occurred.
Memory monitoring data showed that the increase in WSS was mainly caused by cache. According to the official Kubernetes documentation, WSS is the working set of a container in a pod. It is calculated as the cgroup memory usage of the container minus the inactive file cache. Therefore, if most of the WSS is cache, it means that the active file cache usage is high, which indicates that a file is being read frequently.
Diagnostic analysis
For this scenario, you can use the Memory Panoramic Analysis feature of Pod Diagnosis in the Alibaba Cloud Linux console to diagnose the target pod. The diagnosis result is shown in the following figure.
The memory distribution result shows that the active memory is 3.52 GB. This amount, plus the heap memory usage, is roughly consistent with the monitored WSS.
A closer look at the file cache usage list reveals that a log file occupies the largest portion of the container cache. Three processes have this log file open. One process generates the logs, while the other two read and parse the log file.
The active file cache contains frequently accessed data. If a file is read only once, its cache becomes inactive. If most of a file's cache is active, it is likely that multiple processes are repeatedly reading the same file content. In this case, a log file was generated inside the container. Two processes repeatedly read and parsed this file, causing its cache to become active. Because WSS includes active file cache, the container's working set usage appeared high and triggered an alert, even though the service was running normally.
Case study 2: OOM caused by a sudden memory request, but monitoring shows memory was not full
Scenario description
In some scenarios, a pod may need a sudden burst of memory when service traffic spikes. If the memory request is large and quickly reaches the pod's limit, it can trigger an OOM event. When the OOM event occurs, monitoring data often shows that the memory was not fully used. This happens because monitoring data is sampled at intervals, such as every 15, 30, or 60 seconds, and a sudden memory request can be missed between samples.
Problem description
A customer observed an OOMKilled event for a pod in the ACS Event Center. This event indicated that the pod's memory usage exceeded its limit, which triggered the system's oom-killer to terminate the process with the highest memory usage. To see the actual memory usage when the event occurred, the customer checked the pod's monitoring data.
The pod monitoring data shows that the target container's memory request was 4 GB. Between 17:05 and 17:30, the monitored memory utilization of the container remained below 15%, with no high usage observed.
The observed WSS usage also remained low, and the cache did not grow. This led to a counterintuitive situation: an OOM event occurred, but monitoring did not show memory exhaustion. This usually happens when a process makes a sudden memory request, and the time between the request and the OOM event is shorter than the monitoring sampling period. The question is how to determine what happened at the time of the OOM event and identify which process used too much memory.
Diagnostic analysis
For this scenario, you can use the OOM Diagnosis feature of Pod Diagnosis in the Alibaba Cloud Linux console to diagnose the target pod. The diagnosis result is shown in the following figure.
The diagnosis result shows that the target pod experienced an OOM event around 17:15. (The OOMKilled event is generated with a delay, a few minutes after the actual OOM event occurs). The process that triggered the OOM event was a Python application. At that time, 20 Python processes were running inside the container, with a total memory usage of 3.9 GB, which was close to the container's memory limit.
The detailed report shows that at the time of the OOM event, the main memory usage was normal anonymous memory, which was mostly the application's heap memory. The task list from that time shows that the top Python process had requested about 3.43 GB of memory, while the other Python processes were using only 50 to 60 MB each.
This analysis confirms that a sudden memory request from a Python application process in the target pod caused the OOM event.
Case study 3: Continuous OOM events, and memory is not fully recovered after each event
Scenario description
In Kubernetes, shared memory is often used for efficient cross-container communication or data sharing. In ACS, for example, you can use shared memory by mounting an EmptyDir volume. This memory is independent of any process. If a process is killed by an OOM event, the shared memory it was using is not released. This can lead to a memory leak if the shared memory is not properly cleaned up.
Problem description
A customer first observed an OOMKilled event for a pod in the ACS Event Center. For further analysis, they checked the pod's monitoring data.
The pod monitoring data shows six distinct memory spikes, which are likely related to OOM events.
A further look at the container's dmesg logs shows seven OOM events during this period. (Due to a time zone difference, add 8 hours to the container's internal time to get the actual time).
The preceding information reveals the following:
Not every OOM event reached the memory limit. One OOM event did not correspond to a spike in the memory monitoring graph, which indicates a sudden memory request.
After each OOM event, memory usage dropped, but the overall trend was upward. This suggests that memory was not fully released and that a leak was occurring.
Diagnostic analysis
Diagnose the latest OOM event
For this scenario, you can first use the OOM Diagnosis feature of Pod Diagnosis in the Alibaba Cloud Linux console to diagnose the OOM event for the target pod. The diagnosis results are shown in the following figures.
The diagnosis result shows that the latest OOM event on the target pod was caused by 3 GB of shared memory usage and 1 GB of application memory usage, which together reached the memory limit.
Diagnose historical OOM events
Multiple OOM events occurred. The latest one was caused by excessive shared memory usage combined with 1 GB of application memory. To better analyze the root cause, you can diagnose historical OOM events.
As shown in the figure, when you start an OOM diagnosis, you can specify a time to diagnose a historical event. The tool analyzes the OOM event closest to the specified time. The result is as follows:
The diagnosis result shows that the OOM event at 15:06 was mainly caused by high memory usage from the application. At that time, shared memory usage was not high.
Use memory panoramic analysis to check shared memory
The first two diagnosis results show that the application process had sudden memory requests that caused periodic OOM events. They also show a shared memory leak, because memory was not fully released after each event. Next, you can use Memory Panoramic Analysis to analyze the shared memory usage.
The Memory Panoramic Analysis result shows many small, application-related shared memory files in the container. After these leaked files were manually deleted, monitoring data confirmed that the memory was released.
This analysis confirms that sudden memory requests from an application process caused OOM events in the target pod. These OOM events were accompanied by leaks of many small shared memory files. The memory was released only after the leaked files were manually deleted.