FAQ and known issues

FAQ

1. Reducing report size

We recommend keeping collected reports under 200 MB. You can use the following methods to reduce the report size:

• Keep report files to < 200 MB.

• Disable CPU sampling using the-s none option.

• Delay the start of collection and control the sampling duration using the-y and-d options.

• Trigger collection by event using the-c option to specify the trigger method.

• Use the--trace-rotation option with the--duration option to perform report file rotation.

• When profiling applications launched with tools like mpirun, use the--process-rank-include option to collect trace data from only specific ranks.

For large reports, you can partially load them in the Asight Systems GUI by selecting File > Open Advanced.... This allows you to:

• Load only a portion of the report (e.g., the first 50% of the trace data).

• Skip parsing specific types of trace data (e.g., CPU scheduling trace data).

2. Collecting multiple reports after application startup

After an application starts, you can collect multiple reports using the following methods:

You can use asys launch to launch an application and then use asys start/stop to start data collection. For example:

asys launch -t hggc cuda_test
asys start
asys stop
asys start
asys stop

If your application is instrumented with multiple capture ranges, you can trigger collection multiple times with--capture-range-end repeat-shutdown:N. For example, to generate three report files triggered by capture range events:

asys profile --capture-range-end repeat-shutdown:3 python test_linear.py 

You can use the--duration and--trace-rotation options of theasys profile subcommand to periodically generate reports via report file rotation. For example, to generate a new report file every 30 seconds and keep the three most recent ones:

asys profile --duration 30 --trace-rotation 3 --output test_report_%t python test_linear.py 

You can run theasys attach subcommand multiple times on a target application to collect multiple reports. For example, to repeatedly collect trace data from the process with PID 94644, run the following command multiple times:

asys attach -o report_%i 94644

3. Triggering trace collection with events

asys supports event-triggered collection. For example:

asys profile -c hgtx -p msg python test_linear.py 

Supported event markers include NVTX and profilerStart/Stop. The following examples show how to add event markers:

Add NVTX range information in a PyTorch application:

import torch.cuda.nvtx as nvtx

nvtx.range_push("msg")
...
nvtx.range_pop()

Add NVTX markers in a C/C++ application:

#include <nvtx3/nvToolsExt.h>

nvtxRangePushA("msg");
...
nvtxRangePop();

Add profilerStart/Stop markers in a PyTorch application:

torch.cuda.cudart().cudaProfilerStart()
...
torch.cuda.cudart().cudaProfilerStop()

4. Collecting OSRT information

You can identify the cause of CPU thread suspension by collecting OSRT information.

To specify OSRT tracing, add osrt to the -t trace option.

• Use the--osrt-threshold option to specify the collection threshold.

• Use the--osrt-backtrace-threshold option to specify the call stack backtrace threshold.

5. Profiling multi-node, multi-device applications

For applications that run on multiple PPUs across multiple compute nodes and are often launched using tools like mpirun or deepspeed, you can collect trace data in two ways. The following examples use an mpirun application:

• Single-node mode: Use asys to launch the mpirun application. Trace data from all processes on the current node is recorded in a single report file. No trace data is collected from other nodes.

• Multi-node mode: Use mpirun to launch asys. Trace data for each application instance on each node, which typically corresponds to one rank or one PPU, is recorded in a separate report file.

The following examples show how to use single-node and multi-node modes. To collect trace information from a single node and consolidate it into one report file:

asys profile mpirun –np 4 cuda_test

To collect data from multiple nodes with a separate report file for each application, use the-o option with%h (host name) and%p (PID) to generate unique report names and avoid conflicts. You can also use the-o option to generate report names from environment variables. For example, with applications launched by mpirun, the-o option can use%q{OMPI_COMM_WORLD_RANK} to include the rank ID in the report name.

mpirun -np 4 asys profile -o report_%h_%p cuda_test

When collecting trace data in single-node mode, the trace data from all processes is recorded in a single file, which can become very large. To collect trace data from only a subset of processes, use the--process-rank-include option to specify the target rank IDs, separated by commas (","). For example, to collect trace data only from ranks 0 and 2:

asys profile --process-rank-include 0,2 mpirun –np 4 cuda_test

When you specify the--process-rank-include option, asys checks the following environment variables by default to detect the process rank ID:

OMPI_COMM_WORLD_RANK
SLURM_PROCID
RANK
PMI_RANK

To use a different environment variable to get the rank ID, specify its name with the--process-rank-env option. For example, to use thePROCESS_RANK_ID environment variable to detect the rank ID:

asys profile --process-rank-include 0,2 --process-rank-env=PROCESS_RANK_ID mpirun –np 4 cuda_test

Note: If you use the--process-rank-include option to specify which ranks to collect, asys will not collect trace data from any process whose rank ID it cannot detect.

6. Profiling applications that use fork

If trace data is missing for an application's child processes, the cause may be a fork call that is not followed by an exec call to start another program. By default, asys does not collect trace data after a fork and before an exec. To enable trace collection for child processes in this scenario, use the--trace-fork-before-exec option:

asys profile --trace-fork-before-exec true cuda_test

7. Missing device memory usage information

If no device memory usage is collected, the application might be allocating it only during initialization. In that case, check your asys options:

• If you are using the-c,--capture-range,-y, or--delay options, try removing them to capture trace data from the application startup phase.

• If you are using asys start or asys attach to collect data while the application is already running, switch to asys profile to capture trace data that includes the application's initialization phase.

8. Installing the SSH service

To install the SSH service on Ubuntu, follow these steps:

# install openssh service
sudo apt-get update
sudo apt-get install openssh-server

# edit ssh service config to modify SSH port / environment / authentication
# vi /etc/ssh/sshd_config

# restart ssh service
sudo service ssh restart

To install the SSH service on CentOS, follow these steps:

# install openssh service
sudo yum install openssh-server

# edit ssh service config to modify SSH port / environment / authentication
# vi /etc/ssh/sshd_config

# restart ssh service
sudo systemctl restart sshd.service

When using Docker, you need to forward the container's SSH port to a host port. For example, to forward the container's port 22 to the host's port 50022, add the following option to your docker run command:

--expose=22 -p 50022:22

9. Cleaning up the asys runtime environment

If asys behaves unexpectedly, try the following steps to clean up its runtime environment:

• Runasys sessions list to check for any long-running sessions. You can stop a session with theasys shutdown --session xxx command.

• Stop theasys background services by runningps -aux | grep traced | grep -v grep | awk '{print $2}' | xargs kill, then try running asys again.

• Reinstall the PPU-SDK.

10. Application does not stop after collection

When you use the--duration or--capture-range option, asys sends a SIGTERM signal by default to stop the application when collection completes. If your application handles SIGTERM but does not terminate promptly, it may continue running after collection finishes. Try the following:

• Specify the--kill 9 option. This makes asys send the non-interceptable SIGKILL signal to terminate the application.

11. Incomplete trace data for TensorFlow Eager mode

If you find that trace data collected by asys for a TensorFlow Eager mode application is incomplete, add the--flush-on-context-synchronize true option. This enables asys to save data upon each HGGC context or stream synchronization. Note that this option increases collection overhead.

asys profile --flush-on-context-synchronize true cuda_test

12. Enabling Alt-pan shortcut on Linux

On Linux platforms, if the Alt + scroll wheel combination is already used by the global zoom feature, you can disable global zoom to free up the Alt key.

In a terminal, run the following command:

xfconf-query -c xfwm4 -p /general/zoom_desktop -s false

You can usexfconf-query -c xfwm4 -p /general/zoom_desktop to verify that its value is false.

13. Heap and process memory usage discrepancy

asys supports the following options to collect the CPU-side memory usage of a process:

• --heap-memory-usage: Collects the process's usage ofdynamically allocated memory, such as memory allocated viamalloc ornew.

• --host-memory-sampling: Samples the process'sphysical memory usage, which is similar to the resident set size (theRES column) shown by the Linux top command.

The amounts ofdynamically allocated memory andphysical memory that asys collects can differ significantly for the following reasons:

• Due to C library memory caching optimizations and memory fragmentation, the process may hold memory instead of returning it to the operating system. This can cause thephysical memory usage to be significantly higher than thedynamically allocated memory usage.

• Due to the operating system's memory paging mechanism and lazy allocation strategies, memory allocated via methods likemalloc but not yet accessed does not consume actual physical memory. This can cause thephysical memory usage to be significantly lower than thedynamically allocated memory usage.

Known issues

• PPU-SDK v1.0.0: The duration of P2P memcpy operations collected by asys may be inaccurate.

• PPU-SDK v1.0.0: The PCIe read throughput and PCIe write throughput metrics only account for transactions initiated by the PPU.

• PPU-SDK v1.1.0: The reported PPU memory usage for hgMemAllocFromPoolAsync may be higher than the actual value.

• PPU-SDK v1.1.0: Collecting CPU backtraces with asys may cause the PPU KMD driver to crash.

• PPU-SDK v1.2.0: asys may lose some HGGC memset trace data during collection.

• PPU-SDK v1.2.0: asys may fail and exit when collecting OSRT traces for applications that use pthread_yield.

• PPU-SDK v1.2.0: Analysis results for ppu_gaps and ppu_time_util from asys analyze are inaccurate.

• PPU-SDK v1.2.0: Trace data for H2H memcpy operations collected by asys is inaccurate.

• PPU-SDK v1.2.0: When capture range is enabled, the acu command generated from an HGGC Kernel in the Timeline View is incorrect.

• PPU-SDK v1.2.0: The ICN utilization metric collected by asys is lower than the actual value.

• PPU-SDK v1.2.0: The PCIe utilization metric collected by asys is lower than the actual value.

• PPU-SDK v1.2.0: asys may crash when profiling applications that subscribe to hgpti.

• PPU-SDK v1.2.0: Profiling an HGGC graph that includes H2H memcpy operations may produce incorrect results. As a workaround, set the environment variable by running export HGGC_GRAPH_FORCE_DYNAMIC=1.

• PPU-SDK v1.3.0: asys is only supported on systems with a page size of 4 KB.

• PPU-SDK v1.3.0: The PPU operating frequency collected by asys is inaccurate.

• PPU-SDK v1.4.0: asys attach does not currently support Ubuntu 22.04 or later OS versions.

• If you interrupt collection with Ctrl+C, asys may not write the final trace data to the report file.

• asys does not support collecting CPU call stack information on operating systems with a Linux kernel version earlier than 4.1.

• The asys collection process may significantly increase the duration of HGGC stream synchronization.

• Applications must be compiled with the-pthread option; otherwise, asys may crash.

• asys attach may cause some applications to hang or crash.

• asys attach does not support collecting trace information for legacy HGGC graphs that were instantiated before the attach operation.

• The RDMA NIC data throughput collected by asys is lower than the actual value.

• asys can collect CPU thread switching information only if the ftrace clock is set to the boottime clock.