FAQ & known issues

FAQ

1. Reduce report size

For optimal performance, keep collected reports under 200 MB. You can use the following methods to reduce the report size:

• Try to keep the report file size under 200 MB.

• Disable CPU sampling using the-s none option.

• Enable delayed collection startup and control the collection duration using the-y and-d options.

• Trigger collection based on events by specifying the trigger method with the-c option.

• Use the--trace-rotation option with the--duration option to enable report file rotation.

• When profiling applications launched with tools like mpirun, use the--process-rank-include option to collect trace data from only a subset of rank processes.

To open a specific portion of a large report, select File > Open Advanced... in the Asight Systems GUI and configure the following:

• Limit the loaded data to a percentage of the report (for example, load only the first 50% of the trace data).

• Exclude specified types of trace data from being parsed (for example, ignore CPU scheduling information).

2. Collect multiple reports

To collect multiple reports after an application starts, use the following methods:

Launch the application withasys launch, and then useasys start/stop to control the collection. For example:

asys launch -t hggc cuda_test
asys start
asys stop
asys start
asys stop

If your application is instrumented with multiple capture ranges, you can use the--capture-range-end repeat-shutdown:N option to trigger collection multiple times. For example, to generate three report files based on capture range events:

asys profile --capture-range-end repeat-shutdown:3 python test_linear.py 

Use theasys profile subcommand's--duration and--trace-rotation options to periodically generate reports through report file rotation. For example, to generate a report every 30 seconds and keep the three most recent files:

asys profile --duration 30 --trace-rotation 3 --output test_report_%t python test_linear.py 

Use theasys attach subcommand to attach to the target application multiple times, collecting a separate report for each session. For example, to repeatedly collect trace data from the process with process ID (PID) 94644, run the following command multiple times:

asys attach -o report_%i 94644

3. Trigger trace collection with events

asys supports event-triggered collection. For example:

asys profile -c hgtx -p msg python test_linear.py 

The collection can be triggered by NVTX or profilerStart/Stop event markers. The following examples show how to add these event markers:

Add NVTX ranges in a PyTorch application:

import torch.cuda.nvtx as nvtx

nvtx.range_push("msg")
...
nvtx.range_pop()

Add NVTX markers in a C/C++ application:

#include <nvtx3/nvToolsExt.h>

nvtxRangePushA("msg");
...
nvtxRangePop();

Add profilerStart/Stop markers in a PyTorch application:

torch.cuda.cudart().cudaProfilerStart()
...
torch.cuda.cudart().cudaProfilerStop()

4. Analyze CPU suspension with OSRT

You can collect OSRT information to identify the causes of CPU thread suspension.

To enable OSRT tracing, add osrt to the -t option.

• Use the--osrt-threshold option to specify the collection threshold.

• Use the--osrt-backtrace-threshold option to specify the call stack backtrace threshold.

5. Profile multi-node and multi-PPU applications

asys provides two methods to collect trace data for applications that run on multiple PPUs across multiple compute nodes, often launched with tools like mpirun or deepspeed. The following examples use mpirun:

• Use multi-node trace collection mode (recommended):

  • Start the mpirun application by using asys and use the --multi-node-mode mpirun option to enable the multi-node trace collection mode. Each node generates a report file.

• Do not use multi-node collection mode:

  • Single-node mode: When an mpirun application is started by using asys, trace data for all processes on the current node is recorded in a single report file, and no trace data is collected from other nodes.

  • Multi-node method: When asys is launched by using mpirun, the trace data from each application (which typically corresponds to one Rank or one PPU) on each node is recorded into a single report file.

The following examples show how to use the single-node and multi-node methods. To collect trace information from a single node and consolidate it into one report:

asys profile mpirun –np 4 cuda_test

To collect data from multiple nodes and generate a separate report for each application, use the-o option with%h (host name) and%p (process ID) to avoid naming conflicts. You can also use the-o option to generate report names from environment variables. For applications launched with mpirun, you can include the Rank ID in the report name using-o with%q{OMPI_COMM_WORLD_RANK}.

mpirun -np 4 asys profile -o report_%h_%p cuda_test

When collecting trace data in single-node mode, all trace data is recorded in a single file, which can become very large. To collect data from only a subset of processes, use the--process-rank-include option to specify the Rank IDs of the target processes. Separate multiple Rank IDs with a comma (,). For example, to collect trace data from only processes with Rank IDs 0 and 2:

asys profile --process-rank-include 0,2 mpirun –np 4 cuda_test

You can use the --process-ppu-include option to specify a comma-separated list of PPU device IDs from which to collect trace data. For example, to collect trace data only for processes that use PPU device IDs 0 and 2:

asys profile --process-ppu-include 0,2 mpirun –np 4 cuda_test

6. Profile applications that use fork

If you notice that trace data from a child process is missing, the application might be calling the fork function without a subsequent exec call. By default, asys does not collect trace data between a fork and an exec. To enable tracing of child processes in this scenario, use the--trace-fork-before-exec option:

asys profile --trace-fork-before-exec true cuda_test

7. Missing device memory usage information

An application might allocate device memory only during its initialization phase and perform no further allocation or deallocation operations at runtime. If device memory usage information is missing, check your asys options:

• If you are using the-c (--capture-range) or-y (--delay) options, try removing them to ensure collection starts with the application and captures the initialization phase.

• If you are usingasys start orasys attach to profile an already running application, switch toasys profile to capture trace data from the application's startup.

8. Install the SSH service

To install the SSH service on Ubuntu, follow these steps:

# install openssh service
sudo apt-get update
sudo apt-get install openssh-server

# edit ssh service config to modify SSH port / environment / authentication
# vi /etc/ssh/sshd_config

# restart ssh service
sudo service ssh restart

To install the SSH service on CentOS, follow these steps:

# install openssh service
sudo yum install openssh-server

# edit ssh service config to modify SSH port / environment / authentication
# vi /etc/ssh/sshd_config

# restart ssh service
sudo systemctl restart sshd.service

When using Docker, you need to forward the SSH port from the container to a host port. For example, to forward the container's port 22 to port 50022 on the host, add the following options to your docker run command:

--expose=22 -p 50022:22

9. Clean up the asys runtime environment

If asys behaves unexpectedly, try the following steps to clean up its runtime environment:

• Runasys sessions list to check for any long-running sessions. If found, stop them using theasys shutdown --session xxx command.

• Stop all asys background services by runningps -aux | grep traced | grep -v grep | awk '{print $2}' | xargs kill, and then try running asys again.

• Reinstall the PPU-SDK.

10. Application does not stop after collection

When you use the--duration or--capture-range option, asys sends a SIGTERM signal to stop the target application when collection ends. If the application has a SIGTERM handler that does not exit promptly, the application may not terminate. To resolve this, try the following:

• Specify the--kill 9 option. This forces asys to send a SIGKILL signal, which cannot be intercepted, to terminate the application.

11. Incomplete trace data for TensorFlow Eager mode

If asys collects incomplete trace data for TensorFlow applications running in Eager mode, add the--flush-on-context-synchronize true option. This forces asys to save data every time an HGGC context or stream is synchronized. Enabling this option increases profiling overhead.

asys profile --flush-on-context-synchronize true cuda_test

12. Enable Alt-key panning on Linux

On some Linux desktop environments, the Alt + mouse wheel combination is reserved for a global zoom feature. To free up the Alt key for panning in the Timeline View, disable the global zoom feature.

In a terminal, run the following command:

xfconf-query -c xfwm4 -p /general/zoom_desktop -s false

You can verify that the value is set to false by runningxfconf-query -c xfwm4 -p /general/zoom_desktop.

13. Heap usage vs. process memory footprint

asys provides the following options to collect data on a process's CPU-side memory usage:

• --heap-memory-usage: Collects the dynamically allocated memory usage of a process, such as memory dynamically allocated by using malloc / new.

• --host-memory-sampling: Samples the actual physical memory usage of a process. This is the memory usage shown in the RES column for the process when you run the Linux top command.

The dynamically allocated memory usage and actual physical memory usage of a process collected by asys are affected by the following factors, which can cause significant variations in memory usage:

• Due to the C library's memory cache optimization mechanism and memory fragmentation, memory may be held by a process instead of being reclaimed by the operating system, causing the actual physical memory usage to be significantly higher than the dynamically allocated memory usage.

• Due to the operating system's memory paging mechanism and lazy memory allocation strategy, memory that is allocated by using methods such as malloc but has not been accessed is not assigned actual physical memory. This causes the actual physical memory usage to be significantly lower than the dynamically allocated memory usage.

14. Missing trace data for Ray applications

The default collection mode of asys does not support collecting trace data for Ray framework applications, so you must use the --multi-node-mode ray option to enable the multi-node trace collection mode, which causes each node to generate a report file:

asys profile --multi-node-mode ray -f true -o qwen_vllm bash eval_llm_infer.sh

Known issues

• PPU-SDK v1.0.0: The duration of PtoP memcpy operations collected by asys may be inaccurate.

• PPU-SDK v1.0.0: The PCIe Read Throughput and PCIe Write Throughput metrics only account for transactions initiated by the PPU.

• PPU-SDK v1.1.0: The reported PPU memory usage for hgMemAllocFromPoolAsync may be higher than the actual value.

• PPU-SDK v1.1.0: Collecting CPU backtraces with asys may cause the PPU KMD driver to crash.

• PPU-SDK v1.2.0: asys may drop some HGGC memset trace data during collection.

• PPU-SDK v1.2.0: When profiling applications that use pthread_yield, OSRT tracing may cause asys to exit unexpectedly.

• PPU-SDK v1.2.0: The results for ppu_gaps and ppu_time_util from asys analyze are inaccurate.

• PPU-SDK v1.2.0: The collected trace data for HtoH (host-to-host) memcpy operations is inaccurate.

• PPU-SDK v1.2.0: When capture range is enabled, the acu command generated from an HGGC Kernel in the Timeline View is incorrect.

• PPU-SDK v1.2.0: The ICN utilization rate metric collected by asys is lower than the actual value.

• PPU-SDK v1.2.0: The PCIe utilization rate metric collected by asys is lower than the actual value.

• PPU-SDK v1.2.0: asys may crash when profiling an application that subscribes to hgpti.

• PPU-SDK v1.2.0: Profiling an HGGC graph that contains HtoH memcpy operations may produce incorrect results. As a workaround, set the environment variable: export HGGC_GRAPH_FORCE_DYNAMIC=1.

• PPU-SDK v1.3.0: asys only supports running on systems with a page size of 4 KB.

• PPU-SDK v1.3.0: The PPU operating frequency collected by asys is inaccurate.

• PPU-SDK v1.4.0: asys attach does not currently support Ubuntu 22.04 or later versions.

• PPU-SDK v1.5.0: asys does not support collecting NVTX traces from applications that use version 0.2.12 of the Python NVTX library.

• PPU-SDK v1.6.0: Sampling Python application call stacks may cause a Python resource leak, leading to PPU memory exhaustion. To work around this, disable the sampling feature using the --python-sampling false option.

• PPU SDK v1.6.0: When asys enables the multi-node trace collection mode for the Ray framework, the program exits with an error if runtime_env is used to configure env_vars.

• If you interrupt collection with Ctrl+C, data from the end of the collection period might not be saved to the report file.

• asys does not support collecting CPU call stack information on operating systems with a Linux kernel version lower than 4.1.

• Profiling with asys may significantly increase the execution time of HGGC stream synchronization.

• You must compile the application with the -pthread option. Otherwise, asys may crash during runtime.

• Using asys attach may cause some applications to hang or crash.

• asys attach cannot collect trace information for legacy HGGC graphs that were instantiated before the attach operation.

• The RDMA network card data throughput collected by asys is lower than the actual value.

• asys can collect CPU thread switch information only in environments where the ftrace clock is set to boottime.

• When asys profiles an application that has the PyTorch Profiler enabled, the PyTorch Profiler cannot collect any data.