FAQ and known issues

FAQ

1. Reduce report size

Keep the collected report size under 200 MB. You can reduce the report size in the following ways:

• Limit the report file size to under 200 MB.

• Disable CPU sampling using the-s none option.

• Delay the start of collection and limit the duration with the-y and-d options.

• Trigger collection with events: use the-c option to specify the trigger type.

• Use the--trace-rotation option with the--duration option to perform rotational collection of report files.

• When profiling applications launched with tools like mpirun, use the--process-rank-include option to collect trace data only from specific rank processes.

For large, existing report files, you can selectively view their content in the Asight Systems GUI. Go to File > Open Advanced... to specify what to load:

• Load only a specific portion of the report, such as the first 50% of the trace data.

• Skip parsing certain types of trace data, such as CPU scheduling information.

2. Collect multiple reports

You can collect multiple reports after an application has started in the following ways:

Launch an application with asys launch, and then use asys start and asys stop to collect multiple reports. For example:

asys launch -t hggc cuda_test
asys start
asys stop
asys start
asys stop

If your application is instrumented with multiple capture ranges, you can use the--capture-range-end repeat-shutdown:N option to trigger collection multiple times. For example, to generate three report files using capture range events:

asys profile --capture-range-end repeat-shutdown:3 python test_linear.py 

You can use the--duration and--trace-rotation options with the asys profile subcommand to periodically generate report files using rotational collection. For example, to generate a new report file every 30 seconds and retain the three most recent files:

asys profile --duration 30 --trace-rotation 3 --output test_report_%t python test_linear.py 

You can use the asys attach subcommand to attach to a target application multiple times and collect multiple reports. For example, to repeatedly collect trace data from the process with PID 94644, run the following command multiple times:

asys attach -o report_%i 94644

3. Trigger trace collection with events

asys supports event-triggered collection. For example:

asys profile -c hgtx -p msg python test_linear.py 

Events can be marked with NVTX or profilerStart/Stop. The following examples show how to add event markers:

Add NVTX range information in PyTorch:

import torch.cuda.nvtx as nvtx

nvtx.range_push("msg")
...
nvtx.range_pop()

Add NVTX markers in a C/C++ application:

#include <nvtx3/nvToolsExt.h>

nvtxRangePushA("msg");
...
nvtxRangePop();

Add profilerStart/Stop markers in PyTorch:

torch.cuda.cudart().cudaProfilerStart()
...
torch.cuda.cudart().cudaProfilerStop()

4. Collect OSRT information

You can collect OSRT information to identify why CPU threads are suspended.

To collect OSRT traces, add osrt to the -t option.

• Use the--osrt-threshold option to specify the collection threshold.

• Use the--osrt-backtrace-threshold option to specify the call stack backtrace threshold.

5. Profile multi-node and multi-card applications

Applications that run on multiple PPUs across multiple compute nodes are often launched using tools such as mpirun or deepspeed. You can use asys to collect trace data from these applications in two ways. The following examples use an mpirun application:

• Single-node mode: Use asys to launch the mpirun application. All trace data for processes on the local node is recorded to a single report file, and no data is collected from other nodes.

• Multi-node mode: Use the mpirun application to launch asys. The trace data for each application instance on each node, which typically corresponds to a rank or a PPU, is recorded to a separate report file.

To collect trace information from a single node and consolidate it into one report file:

asys profile mpirun –np 4 cuda_test

To collect data from multiple nodes and generate a separate report file for each process, use the-o option with%h and%p to create unique report names based on the host name and PID. This prevents name conflicts. For applications launched with mpirun, you can also use an environment variable in the-o option, such as-o with%q{OMPI_COMM_WORLD_RANK} to include the rank ID in the report name.

mpirun -np 4 asys profile -o report_%h_%p cuda_test

In single-node mode, data volume can be large because asys saves data from all processes into a single report file. To collect trace data from only a subset of processes, use the--process-rank-include option to specify the rank IDs of the target processes. Use a comma (,) to separate multiple rank IDs. For example, to collect trace data only from processes with rank IDs 0 and 2:

asys profile --process-rank-include 0,2 mpirun –np 4 cuda_test

When you specify the--process-rank-include option, asys checks the following environment variables by default to detect the rank ID of a process:

OMPI_COMM_WORLD_RANK
SLURM_PROCID
RANK
PMI_RANK

If you need to use a different environment variable to get the rank ID of a process, specify the variable name using the--process-rank-env option. For example, to detect the rank ID using thePROCESS_RANK_ID environment variable:

asys profile --process-rank-include 0,2 --process-rank-env=PROCESS_RANK_ID mpirun –np 4 cuda_test

Note: When using the--process-rank-include option, asys will not collect trace data from any process whose rank ID cannot be detected.

6. Profile applications that use fork

If you find that trace data for a child process is missing, your application may be calling fork without a subsequent exec. By default, asys does not collect data in this scenario. To trace the child process after the fork, enable this feature using the--trace-fork-before-exec option:

asys profile --trace-fork-before-exec true cuda_test

7. Device memory usage information is not collected

An application might allocate device memory only during its initialization phase, with no further allocation or deallocation operations during its runtime. Check your asys options:

• If you are using the-c or--capture-range option, or the-y or--delay option, try removing them to capture trace data from the application startup phase.

• If you are using asys start or asys attach to collect data from a running application, try switching to asys profile to capture trace data that includes the application's initialization phase.

8. Install the SSH service

To install the SSH service on Ubuntu:

# install openssh service
sudo apt-get update
sudo apt-get install openssh-server

# edit ssh service config to modify SSH port / environment / authentication
# vi /etc/ssh/sshd_config

# restart ssh service
sudo service ssh restart

To install the SSH service on CentOS:

# install openssh service
sudo yum install openssh-server

# edit ssh service config to modify SSH port / environment / authentication
# vi /etc/ssh/sshd_config

# restart ssh service
sudo systemctl restart sshd.service

If you use Docker, you must forward the container's SSH port to a host port. For example, to forward the default SSH port 22 to port 50022 on the host, add the following option to your docker run command:

--expose=22 -p 50022:22

9. Clean up the asys runtime environment

If you encounter issues with asys, try the following steps to clean up its runtime environment:

• Runasys sessions list to check for long-running sessions. Stop a session with theasys shutdown --session xxx command.

• Runps -aux | grep traced | grep -v grep | awk '{print $2}' | xargs kill to stop all asys background services, and then try running asys again.

• Reinstall the PPU-SDK.

10. Application does not stop after collection

When you specify the--duration or--capture-range option, asys sends a SIGTERM signal to stop the application when collection finishes. If your application traps SIGTERM but does not exit promptly, it may fail to stop. Try the following:

• Specify the--kill 9 option. This forces asys to send the unblockable SIGKILL signal to terminate the application.

11. Incomplete traces in TensorFlow Eager mode

If you find that asys collects incomplete trace data for TensorFlow applications in Eager mode, add the--flush-on-context-synchronize true option. This enables asys to save data every time the HGGC context or stream synchronizes, which increases collection overhead:

asys profile --flush-on-context-synchronize true cuda_test

12. Enable the Alt+pan shortcut

On Linux, if the global zoom feature uses the Alt + mouse wheel shortcut, disable global zoom to free up the Alt key:

In a terminal, run the following command:

xfconf-query -c xfwm4 -p /general/zoom_desktop -s false

You can runxfconf-query -c xfwm4 -p /general/zoom_desktop to verify that its value is set to false.

Known issues

• PPU SDK v1.0.0: The duration of P2P memcpy operations collected by asys may be inaccurate.

• PPU SDK v1.0.0: The PCIe Read Throughput and PCIe Write Throughput metrics only count transactions initiated by the PPU.

• PPU SDK v1.1.0: The reported PPU memory usage for hgMemAllocFromPoolAsync may be higher than the actual value.

• PPU SDK v1.1.0: Collecting CPU backtraces with asys may cause the PPU KMD driver to crash.

• PPU SDK v1.2.0: asys may lose some HGGC memset trace data during collection.

• PPU SDK v1.2.0: When collecting OSRT traces, asys may unexpectedly exit if the application uses pthread_yield.

• PPU SDK v1.2.0: The results from asys analyze for ppu_gaps and ppu_time_util are inaccurate.

• PPU SDK v1.2.0: The trace data collected by asys for HtoH memcpy is inaccurate.

• PPU SDK v1.2.0: When capture range is enabled, the acu command generated from an HGGC Kernel in the Timeline View may be incorrect.

• PPU SDK v1.2.0: The ICN utilization metric collected by asys is lower than the actual value.

• PPU SDK v1.2.0: The PCIe utilization metric collected by asys is lower than the actual value.

• PPU SDK v1.2.0: asys may crash when profiling applications that subscribe to hgpti.

• PPU SDK v1.2.0: When profiling an HGGC graph that includes HtoH memcpy operations, asys may produce incorrect results. To work around this issue, set the environment variable by running export HGGC_GRAPH_FORCE_DYNAMIC=1.

• PPU SDK v1.3.0: asys can only run on systems with a page size of 4 KB.

• PPU SDK v1.3.0: The PPU operating frequency collected by asys is inaccurate.

• If you interrupt collection by pressing Ctrl+C, the report file might be missing traces from the final stages of the process.

• asys does not support collecting CPU call stack information on Linux kernel versions earlier than 4.1.

• asys collection may significantly increase the duration of HGGC stream synchronization.

• Applications must be compiled with the -pthread option; otherwise, asys may crash.

• Using the asys attach command may cause some applications to hang or crash.

• The asys attach command does not support collecting trace information for legacy HGGC graphs that were instantiated before attaching.

• The RDMA NIC data throughput collected by asys is lower than the actual value.

• asys only supports collecting CPU thread switch information in environments where the ftrace clock is set to boottime.