The Panoramic Memory Analysis feature helps you diagnose high memory usage when the cause is unclear. It scans your system's current memory status and provides a detailed breakdown of memory consumption. The generated report visualizes the distribution of system and application memory using pie charts and ranks the top 30 consumers of application memory, file cache, and shared memory cache. This topic describes how to use the Panoramic Memory Analysis feature.
Limitations
Region limitations
This feature is currently available only in the Chinese mainland and China (Hong Kong).
Operating system limitations
Architecture
Operating system
x86 architecture
Rocky Linux 9.5
Rocky Linux 9.1
Ubuntu 20.04
Alibaba Cloud Linux 3 Container Optimized Edition
Rocky Linux 8.8
Ubuntu 22.04
Alibaba Cloud Linux 3 Pro
Alibaba Cloud Linux 2/3
CentOS 7.6 or later, or CentOS 8
Anolis OS 7/8
Ubuntu 24.04
ARM architecture
Alibaba Cloud Linux 3 Pro
Alibaba Cloud Linux 3
JDK limitations
This feature supports OpenJDK 1.8 or later.
Prerequisites
If you use a RAM User, make sure that the
AliyunECSReadOnlyAccessandAliyunSysomFullAccesssystem policies have been granted to the RAM User by your Alibaba Cloud account.The SysOM service is enabled.
The first time you log on to the SysOM console, click Enable Service.
Procedure
Go to the SysOM console.
In the left-side navigation pane, click System Diagnosis. In the upper-left corner of the page, select the region where your target instance is located.
Diagnose instance
From the diagnostic type drop-down list, select Memory Diagnosis. From the diagnostic item drop-down list, select Panoramic Memory Analysis. Select the target instance ID and click Execute Diagnosis.
NoteIf SysOM is not installed, clicking Execute Diagnosis automatically installs it on the target ECS instance and then starts the diagnosis.
Diagnose Java in instance
For diagnostic mode, select Node Diagnosis. From the diagnostic type drop-down list, select Memory Diagnosis. From the diagnostic item drop-down list, select Panoramic Memory Analysis.
Select the instance ID of the target instance.
Enable or disable application memory profiling. If you enable it, set the profiling duration in minutes. If you do not set a duration or set it to 0, profiling is not performed.
To profile a specific process, enter its process ID (PID). If you leave this field blank, the system profiles the Java process with the highest memory usage by default.
Click Execute Diagnosis.
ImportantTo enable application memory profiling, the instance or the instance where the Pod resides must be under management.
Enabling application memory profiling incurs a performance and memory overhead. Memory overhead ranges from 50 MB to 300 MB, depending on the symbol size.
Diagnose Java in Pod
For diagnostic mode, select Pod Diagnosis. Select the cluster type and cluster ID. From the diagnostic type drop-down list, select Memory Diagnosis. From the diagnostic item drop-down list, select Panoramic Memory Analysis.
Select the target Pod.
Enable or disable application memory profiling. If you enable it, set the profiling duration in minutes. If you do not set a duration or set it to 0, profiling is not performed.
To profile a specific process, enter its PID. If you leave this field blank, the system profiles the Java process with the highest memory usage by default.
Click Execute Diagnosis.
ImportantTo enable application memory profiling, the instance or the instance where the Pod resides must be under management.
Enabling application memory profiling incurs a performance and memory overhead. Memory overhead ranges from 50 MB to 300 MB, depending on the symbol size.
In the Diagnostic Records section, click View Report.
Diagnostic report
Basic information
This section contains basic information about the diagnosis, including the instance ID (or resource ID), diagnostic item, diagnostic report ID, and the diagnosis start time.
Diagnostic conclusion
If any memory usage anomalies are detected, they are displayed in this section.
Diagnostic suggestion
If your system's memory usage is abnormal, this section provides recommendations for further investigation.
Diagnostic details
Memory usage analysis
Three pie charts show the distribution of total memory usage, kernel-space memory usage, and user-space memory usage.
Application memory usage ranking
This ranking helps you identify Java processes with high memory consumption in the instance or Pod.
File cache usage ranking
Lists the top 30 files that consume the most file cache on your system. This includes the file name, cached size (Cached), and associated task (if any).
Shared memory cache usage ranking
Lists the top 30 files that consume the most shared memory cache on your system. This includes the file name, cached size (Cached), shared memory type, and associated task (if any).
View Java application memory distribution
Java process information overview & Java process Pod/container information
Displays basic information about the Java process, including its PID, JVM memory usage (from the JVM's perspective), Java process memory usage (the actual memory occupied by the process), anonymous memory usage, and file-backed memory usage.
If you are diagnosing a Java process in a Pod, the report also shows the RSS and working set memory usage of the Pod and its container.
Java process memory details
Three pie charts detail the Java heap memory usage (JVM perspective), Java non-heap memory usage (JVM perspective), and the actual Java process memory usage. This helps you identify the largest contributors to memory consumption, both from the JVM's perspective and in terms of the actual memory footprint.
Top 10 in-heap object memory usage
Lists the top 10 classes by in-heap memory consumption, showing the class name, number of instances, and the total size occupied by those instances.
Ranking of class loaders by number of loaded classes (Top 10)
Lists the top 10 classloaders by the number of classes loaded, showing the classloader name, the number of loaded classes, and the number of classloader instances.
Java process actual memory usage increment during profiling & Java non-heap memory usage increment during profiling
If you enabled application memory profiling, this section shows the size change for each component of the Java process's memory during the profiling period. For example, the figure shows that at the end of profiling, the process's actual memory usage (RSS) increased by 258 MB compared to the start. From the JVM's perspective, total memory increased by 590 MB and heap memory increased by 206 MB, but the actual heap memory increased by 276 MB, and non-heap memory increased by 384 MB. The non-heap memory increment details show that metaspace increased by 355 MB. This indicates that the total RSS increase of 258 MB resulted from memory allocation for metaspace and the heap, combined with the release of JNI memory.
Java (heap/non-heap) memory allocation flame graph
If you enabled application memory allocation profiling, the report shows flame graphs for heap and non-heap memory allocation in the Java process. The flame graph displays all call paths that allocate memory. Note: Because data is collected through sampling, the memory sizes in the flame graph do not represent the actual allocated sizes.
JNI (Java Native Interface) memory allocation flame graph
If you enabled application memory profiling, the report shows a flame graph for JNI (Java Native Interface) memory allocation. The flame graph displays all call paths that allocate JNI memory. Because data is collected through sampling, the memory sizes in the flame graph do not represent the actual allocated sizes. Enabling application memory profiling incurs a performance and memory overhead. Memory overhead ranges from 50 MB to 300 MB, depending on the symbol size.
To view this flame graph, enable JNI (Java Native Interface) memory allocation profiling when you start the diagnosis. The data appears in the report after the specified profiling duration elapses.
JNI (Java Native Interface) memory leak flame graph
If you enabled JNI (Java Native Interface) profiling, the report shows a flame graph for potential JNI memory leaks. The flame graph displays all allocation call paths that may have JNI memory leaks. Because data is collected through sampling, the memory sizes in the flame graph do not represent the actual allocated sizes. Enabling application memory profiling incurs a performance and memory overhead. Memory overhead ranges from 50 MB to 300 MB, depending on the symbol size.
To view this flame graph, enable JNI (Java Native Interface) memory allocation profiling when you start the diagnosis. The data appears in the report after the specified profiling duration elapses.
Use cases
Containerization has become a best practice for building modern IT architectures. Cloud-native container-based deployment is now an industry standard that combines operational efficiency with cost control. While containerization abstracts away the underlying infrastructure and cloud resources, it can also introduce a lack of transparency in the container engine layer, leading to issues such as memory black holes, high memory usage, memory leaks, jitter, andcgroup leaks. Panoramic Memory Analysis helps make system memory maintainable, testable, and traceable by attributing cache and shared memory within the system and containers to specific file names.
High working set memory in Kubernetes containers
Kubernetes monitors and manages container memory usage based on the working set. When a container's memory usage exceeds its defined limit or the node experiences memory pressure, Kubernetes uses the working set to decide whether to evict or terminate the container.
working set = anonymous memory + active_file
Anonymous memory is typically allocated by using methods such asnew,malloc, ormmap, whileactive_file memory is introduced when a process reads or writes files. Application use of this type of memory is often opaque, which can easily lead to problems.
Symptoms
A container monitoring system detects that the working set memory of a Pod in a Kubernetes cluster is continuously increasing, but the application's own memory usage has not risen significantly. This makes it difficult to pinpoint the cause of the increased working set memory.
Analysis
For this scenario, use the Panoramic Memory Analysis feature in the SysOM console to diagnose the ECS instance where the Pod is running. The diagnostic results are shown in the following figures.
Based on the Diagnostic suggestion, the problem is caused by the business application creating and writing log files in the/var/log directory within the container, which generates a file cache. The resulting high working set memory causes the Pod to hit its memory limit. This triggers direct memory reclamation and blocks the business process.
Solution
Based on the Diagnostic conclusion, you can resolve the high memory issue in the following ways:
Run the following command to manually clear the cache:
sudo sh -c 'echo 1 > /proc/sys/vm/drop_caches'If the cached files are not essential, you can manually delete them to free up cache space.
Use the memory QoS feature of your ACK cluster.
High shared memory usage
Symptoms
A user runs thefree -h command on a long-running machine and notices that the available memory is low, whilebuff/cache is high. Runningsudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' reclaims some cache, but a large amount ofcache remains.
Analysis
For this scenario, use the Panoramic Memory Analysis feature in the SysOM console to diagnose the target ECS instance. The diagnostic results are shown in the following figures.
The Diagnostic conclusion provides a concise analysis.
Following the guidance in the Diagnostic suggestion and checking the shared memory cache usage ranking, you can see the top 30 files that consume shared memory cache. Because the top few files are only 160 KB each, this confirms the diagnostic conclusion that there is a leak of many small shared memory files in the system.
Solution
This analysis confirms that the application creates shared memory files in the specified directory but fails to release them in a timely manner. You need to assess whether there is a memory leak based on your business requirements and delete the leaked shared memory files.
Mismatch between Pod and JVM memory usage
Symptoms
A container monitoring system detects that the RSS memory of a Java business Pod in a Kubernetes cluster is continuously increasing, which eventually leads to an OOM error.
However, JVM monitoring shows that the heap and non-heap memory usage is nearly 100 MB less than the RSS memory reported by container monitoring. This leaves a 100 MB discrepancy to account for.
Analysis
For this scenario, use the Panoramic Memory Analysis feature in the SysOM console to diagnose the business Pod. The diagnostic results are shown in the following figure:
From the diagnostic conclusion and diagnostic suggestion, you can determine that a JNI (Java Native Interface) memory leak exists. Because JNI memory usage is typically not tracked by the JVM, this is one of the main reasons why RSS memory is higher than the memory reported by the JVM.
Solution
Run the diagnosis again, this time with JNI memory profiling enabled. After the diagnosis is complete, wait for the specified profiling duration, and then view the report.
The JNI memory allocation and leak flame graphs show that the code created a large number of java.util.zip.Deflater objects without releasing them correctly, which caused a JNI memory leak. Optimizing the code to fix the leak resolved the issue.