This document describes how to configure and use the continuous gpu profiling feature in the operating system console to analyze performance hotspots of AI applications on CPUs and GPUs. This feature visualizes the function call stack and time distribution for a process, helping you identify performance bottlenecks and optimize AI task efficiency.
Configure and enable continuous GPU profiling
Step 1: Create a configuration
Go to the operating system console.
Install the SysOM component. For installation instructions, see component management.
If the SysOM component is already installed, you must upgrade it to version 3.9.0 or later.
In the left-side navigation pane, click component management to create a new configuration.
Enter a configuration name and select the Enable continuous GPU profiling checkbox. In this example, the configuration name is
gpu continuous profiling configuration.
Step 2: Activate the configuration
Use a management plan in the operating system console to enable this feature for the target GPU instances, set the SysOM component configuration to the continuous GPU profiling configuration created in the previous step, and then click Submit to successfully enable continuous GPU profiling.
After you enable continuous GPU profiling, the memory limit of the SysOM Agent changes from the default 300 MB to 2 GB.
Step 3: Continuous GPU profiling
Navigate to the continuous GPU profiling page under gpu performance and diagnostics. Select the instance under test running your AI application, the PID of the application's process, and the desired time range from the creation time selector. Then, click Start Analysis.
After the analysis is complete, the results are displayed as follows:
CPU/GPU heatmap
In this chart, each column represents a one-second interval and contains 50 squares, where each square represents a 20 ms period. The color intensity of a square indicates the number of sampling events during that period. A darker color indicates a higher load. You can drag either end of the interactive timeline at the bottom to compare and view the data for different time ranges.
CPU flame graph
This graph shows function call relationships and hotspots, similar to the graph in the process hotspot tracking feature.
GPU flame graph
This graph displays the GPU call stack information related to Python processes.
