Trace target application

1. PPU activity tracing

asys supports collecting trace information for both HGGC and CUDA. You can use the -t option to specify which items to trace for your CUDA application. If you do not specify any trace items, asys collects hggc,hgtx,acdnn,acblas by default. The following table lists the PPU-related trace items that the --trace option supports.

The scope of tracing and data collection for HGGC/CUDA by the hggc trace item is controlled by the--hggc-trace-set option, which enables specific sub-trace items and allows you to specify multiple subsets separated by a,, for example,--hggc-trace-set=kernel-activity,kernel-api:

To exclude specific APIs from the hggc trace, set the ASIGHT_HGGC_TRACE_BLACKLIST environment variable to a comma-separated list of API names. For example: export ASIGHT_HGGC_TRACE_BLACKLIST=cudaEventQuery,cuEventQuery.

1.1 Device PPU activity

1.1.1 PPU activity

Required asys collection option: --trace hggc

After you open a report in the GUI tool, it displays the PPU workload in a hierarchical view, as shown in the following figure:

To improve readability, the tool can render a single timeline across multiple rows. For example, in the figure above, the add_kernel activity appears in its own row and also in the Kernels and Stream rows. If a context contains multiple streams, an [All Stream] row appears, which aggregates information from all streams.

Tips:

• If a device has only one context, the tool hides the context node and directly displays its child stream nodes. Similarly, if a context has only one stream, the tool also hides the stream node.

• The PPU device index shown in the view is the physical index and is independent of the CUDA_VISIBLE_DEVICES environment variable.

The name of each PPU node is prefixed with a percentage indicating its duration relative to its parent node. For a stream node, the calculation is as follows:

stream time percentage = 100.0 * stream duration / context duration
stream duration = Total duration of all events in the stream
context duration = Total duration of all events in the context

Therefore, 50.3% Stream 8 indicates that the total duration of all events in this stream accounts for 50.3% of the total duration of all events in its parent context.

Tips: The percentage calculation for Kernel HGTX nodes under the Kernels row is an exception. The denominator for this percentage is the Stream node, not its direct parent node.

Tips: After you apply a filter, the tool recalculates and reorders the PPU node time percentages based on the filtered range.

The PPU Device row aggregates all activities on the device and displays them in two layers. The upper layer shows kernel executions, and the lower layer shows memory operations. The view uses different colors to distinguish memory operation types, such as HtoD, DtoH, DtoD, Pinned, and Pageable, as shown in the following figure:

1.1.2 PPU graph

Required asys collection option: --trace hggc

The Timeline View supports displaying graph timelines separately to facilitate analysis of graph execution. If the report contains graph information, a graph node appears under the stream node with the following hierarchy:

• Graph Group

  • Graph

    • Graph Exec

For graph timelines, their Graph ID and Graph Exec ID are displayed, as shown in the following figure:

The device row also displays aggregated graph information.

1.1.3 PPU video

Required asys collection option: --trace hgvideo

asys also supports collecting PPU Video Activity data and displays the corresponding timelines to help analyze video processing performance, as shown in the following figure:

A video node appears under the stream node, showing all Video Activity within that stream. Similar to the kernel timeline, the device node also aggregates this information. When you select a time range, the tooltip shows the percentage of time spent on Video Activity, as shown in the following figure:

1.2 Host API calls

1.2.1 HGGC API

Required asys collection option: --trace hggc

Under each thread node, the timeline displays HGGC API calls. You can view the dispatch time of kernels in HGGC and the launch time of memory operations:

1.2.2 ACDNN

Required asys collection option: --trace acdnn

After you enable collection, the ACDNN timeline appears under the thread node:

1.2.3 ACBLAS

Required asys collection option: --trace acblas

After you enable collection, the ACBLAS timeline appears under the thread node:

1.2.4 Video API

Required asys collection option: --trace hgvideo

When enabled, Video API calls, including Encode API, Decode API, and HGJPEG API, appear as follows:

1.3 Navigation between host APIs and device activities

Asight Systems displays the correlation between host API calls and device activities, and allows you to navigate between them:

In the figure above, when you click cudaLaunchKernel, the tool highlights the corresponding kernel executing on the device. You can use the Go to Host API or Go to Device Activity options in the context menu to navigate between them.

1.4 Switching HGGC launch API and kernel names

You can switch how launch API names are displayed by navigating to Tools > Options > Systems Profile and changing the HGGC API Name Mode setting:

• Show host API name

• Show launched kernel name

1.5 Dependencies between PPU activities

Kernels on different streams on a PPU can establish execution dependencies through cudaEventRecord and cudaStreamWaitEvent. The Timeline View can visualize these dependencies. You can display a dependency from the context menu of a PPU activity timeline or a related Event API timeline:

For timelines with dependencies created by event-related APIs, the Timeline View uses an orange curve to represent the dependency, as shown in the following figure:

The view grays out timelines unrelated to the dependency. When you hover over the curve, a tooltip displays details about the dependency. Clicking the curve highlights all timelines related to that dependency path. A green curve indicates dependencies between kernels on the same stream and dependencies on the default stream:

For scenarios where an activity depends on multiple events across multiple streams, the tool displays the dependency as follows:

• Shows the last dependent event on the depended-upon stream.

• Does not show dependent events on the same stream as the queried PPU activity.

By default, the view shows only actual effective dependencies. For example, when you check the dependencies for a kernel, the timeline shows dependencies that existed between the API launch and the actual kernel execution on the PPU—that is, the dependencies that delayed the kernel's execution.

If you want to view all dependencies, including logical dependencies that were already resolved, navigate to Tools > Options > Systems Profile. In the settings window, set HGGC Dependency Display Mode to All to display all logical dependencies.

Tips: The tool calculates dependencies from the trace data in the report. If multiple threads operate on an HGGC stream with contention, the displayed dependencies may be inaccurate.

1.6 Analyzing a single kernel

If a kernel's performance is subpar and requires in-depth analysis, select the kernel in the Timeline View. From the context menu, choose Analyze the Selected Kernel with Asight Compute. This action copies the command required to generate an Asight Compute report. Run this command in the Asight Compute command-line tool to generate the report, and then open it in the Asight Compute GUI tool for further analysis.

2. Tracking PPU metrics

asys periodically collects various runtime metrics from devices, such as resource utilization and I/O throughput, and visualizes these metrics over time in the Asight Systems GUI:

Metrics in the same category are grouped and support two aggregated display modes: Overlay and Stacked. Use the--ppu-metrics-device option to specify the devices to collect from. To collect metrics from all devices, use--ppu-metrics-device all. For example:

asys profile -t hggc --ppu-metrics-device all python test_linear.py

This command uses the default metric set to collect data from all devices. Use the--ppu-metrics-set option to specify the metric set to collect. For example, use--ppu-metrics-set throughput to collect the metric set related to I/O throughput:

asys profile -t hggc --ppu-metrics-device all --ppu-metrics-set throughput python test_linear.py

You can use the --ppu-metrics-device help option to query the list of available devices. You can specify multiple devices, separated by a ,. For example, --ppu-metrics-device 0,1 indicates that runtime metrics are collected for only devices 0 and 1:

root@02892cb56ba5:~# asys profile --ppu-metrics-device help
Possible --ppu-metrics-device values are:
    all: Select all supported PPUs
    none: Disable PPU Metrics [Default]
    device ID list: comma separated device ID list(eg.: 0,1)

Available PPU metrics sampling devices are:
    0: PPU, PCI Bus ID: 00000000:10:00.0
    1: PPU, PCI Bus ID: 00000000:11:00.0
    2: PPU, PCI Bus ID: 00000000:CE:00.0
    3: PPU, PCI Bus ID: 00000000:CF:00.0

Device metrics are collected periodically. The default sampling frequency is 1,000 samples per second (1 kHz). Use the--ppu-metrics-frequency option to set the sampling frequency. For example,--ppu-metrics-frequency 10000 sets the frequency to 10 kHz.

asys supports collecting a wide range of device metrics. However, capacity limitations of the PPU performance counters restrict the number of metrics that can be collected simultaneously. asys provides several predefined metric sets. To view the available metric sets, use the--ppu-metrics-set help option:

root@02892cb56ba5:~# asys profile --ppu-metrics-set help
Possible --ppu-metrics-set values are:
 ---------------- ---------------------------------------- --------------------------------------------
  Identifier       Description                              Enabled Metrics
 ---------------- ---------------------------------------- --------------------------------------------
  summary          Collect clock / memory / CU activity /   ce__cycles_elapsed.avg.per_second
                   Bus throughput information.              ce__cycles_active.avg.pct_of_peak_sustained_elapsed
                                                            gd__dispatch_count.avg.pct_of_peak_sustained_elapsed
                                                            ce__warps_active_accumulated.avg.pct_of_peak_sustained_elapsed
                                                            ce__warps_inactive_cu_active_accumulated.avg.pct_of_peak_sustained_elapsed
                                                            ce__warps_inactive_cu_idle_accumulated.avg.pct_of_peak_sustained_elapsed
                                                            cu__cycles_active.avg.pct_of_peak_sustained_elapsed
                                                            cu__ws_issue_active.avg.pct_of_peak_sustained_elapsed
                                                            cu__we_pipe_tensor_cycles_active.avg.pct_of_peak_sustained_elapsed
                                                            dram__exclude_hbm_bytes_read.sum.pct_of_peak_sustained_elapsed
                                                            dram__exclude_hbm_bytes_write.sum.pct_of_peak_sustained_elapsed
...
  throughput       Collect DRAM / PCIe / ICN link           kvd__transaction_hit_rate.pct
                   throughput information.                  ksd__transaction_hit_rate.pct
                                                            l2__transaction_hit_rate.pct
                                                            dram__exclude_hbm_bytes_read.sum.pct_of_peak_sustained_elapsed
                                                            dram__exclude_hbm_bytes_write.sum.pct_of_peak_sustained_elapsed
                                                            pcie__read_bytes.avg.pct_of_peak_sustained_elapsed
                                                            pcie__write_bytes.avg.pct_of_peak_sustained_elapsed
                                                            icnltx__bytes.sum.pct_of_peak_sustained_elapsed
                                                            icnlrx__bytes.sum.pct_of_peak_sustained_elapsed
...

asys also supports specifying a custom list of metrics. Use the--ppu-metrics-list help option to view the descriptions of all supported metrics:

root@02892cb56ba5:~# asys profile --ppu-metrics-list help
Possible --ppu-metrics-list values are:

ce__cycles_active.avg
    Display Name:
        ce__cycles_active.avg
    Unit: cycle
    Description:
        # of cycles active on CE across CEs
...
kvd__transaction_hit_rate.pct
    Display Name:
        kvd__transaction_hit_rate.pct
    Unit: %
    Description:
        (%) hit rate of KVD cacheable requests
...

Use the--ppu-metrics-list option to specify a custom list of metrics, with individual metrics separated by commas. asys collects the metrics specified with--ppu-metrics-list in addition to those from the metric set specified with--ppu-metrics-set. For example,--ppu-metrics-list ce__cycles_active.avg,kvd__transaction_hit_rate.pct specifies the collection of CE utilization and KVD cache hit rate:

asys profile -t hggc --ppu-metrics-device all --ppu-metrics-set none --ppu-metrics-list ce__cycles_active.avg,kvd__transaction_hit_rate.pct python test_linear.py

Due to the capacity limitations of the PPU performance counters, you must ensure that a custom list specified with--ppu-metrics-list does not exceed the device's capacity. You can use the--check-ppu-metrics-list option to validate a comma-separated list of metrics:

root@02892cb56ba5:~# asys profile --check-ppu-metrics-list ce__cycles_active.avg,kvd__transaction_hit_rate.pct,pu__we_average_warps_issue_stalled_compute_sfu_raw_per_issue_active.ratio
Check sampling metrics list result: PASS.
Metrics list is suitable for PPU sampling capacity.

Tips: Collecting device runtime metrics with asys may fail if another application occupies the PPU performance counter resources. Common reasons for this failure include:

• Another asys application is already collecting device runtime metrics.

• Another acu application is collecting trace data.

• DCGM is collecting profiling metrics. You can pause DCGM collection by runningdcgmi profile --pause.

• You can use PPU-SMI to query the busy status of the PPU performance counter and the HGML GPM service.

  • ppu-smi -q queries the operating status of the PPU performance counter.

  • ppu-smi gpm --get-sample-state Checks if the HGML GPM service is enabled. Enabling the service uses the PPU performance counter.

  • Run ppu-smi gpm -s 0 to pause the output from the HGML GPM service and its use of the PPU performance counter.

3. HGTX tracing

Asight Systems supports HGTX tracing. You can use HGTX to annotate timelines and create custom domains. It supports the push/pop and start/end APIs. The following example shows how to use HGTX:

#include <nvtx3/nvToolsExt.h>

for (int index = 0; index < 1000; ++index) {
    nvtxRangePushA("Loop");
    nvtxRangePushA("DoProcess");
    DoProcess();
    nvtxRangePop();
    ...
    nvtxRangePop();
}

This code creates two nested HGTX ranges: Loop and DoProcess. The inner DoProcess() function calls several HGGC APIs. Use the following command to collect the hgtx trace:

asys profile -t hgtx -o test_report sample

The GUI displays the collected report as follows:

The outer NVTX timeline is layered above the inner NVTX timeline. The Loop timeline encompasses the DoProcess timeline, which in turn encompasses the timeline of the innermost CUDA API calls. Asight supports collecting and displaying user-specified colors:

4. OS runtime tracing

By collecting OSRT information, you can view the reasons why CPU threads are suspended, such as pthread_mutex_lock or sleep. To collect OSRT, add osrt to the -t parameter of the trace option:

• Use the --osrt-threshold option to specify the collection threshold.

• Use the --osrt-backtrace-threshold option to specify the backtrace threshold.

For example:

asys profile -t osrt --osrt-backtrace-threshold 80000 python /test_script/ops/test_linear.py

View the OSRT timeline in the GUI:

5. CPU and thread activity tracing

Asight Systems lets you track thread scheduling on CPU cores. Use the asys tool's --sample or -s option to collect application execution data across all system CPUs. This feature also periodically samples CPU call stacks and aggregates the data to provide function execution time statistics.

For example, to collect CPU execution information and set the call stack sampling period:

asys profile -s process-tree -b dwarf --sample-period 2000000 python test_linear.py

• -s process-tree: Collects CPU scheduling traces for the application process and its child processes.

• -b dwarf: Enables collecting CPU execution call stack information. This is enabled by default and can be omitted.

• --sample-period 2000000: Sets the sampling period for each CPU call stack, in CPU execution cycles.

• --sample-backtrace-depth: Sets the call stack depth for CPU sampling. The default is 24.

Tips:

• Enabling CPU execution sampling can significantly increase the report size. The smaller the --sample-period value, the greater the performance impact on the application.

• The operating system may limit the sampling period specified by the --sample-period option, resulting in a lower sampling rate than you configured.

• If you specify a short sampling period with --sample-period, some sample points may be dropped due to throughput limitations.

5.1 CPU activity

Requires one of the following collection options:

• --sample process-tree: Collects the process tree.

• --sample system-wide: Collects all processes on the system.

When CPU sampling is enabled, you can view the CPU activity:

In the figure above, the CPU Group row shows the overall CPU activity. Each node below it represents the workload on a specific core. A filled area on the timeline indicates that the core was busy during that time period, while a blank area indicates it was idle. Dark blue indicates activity from the main process, and black indicates activity from child processes. Each CPU core node on the left has a color label that is used in the thread rows. The timeline tooltip shows the process and thread currently executing on the CPU:

From the context menu, select "Go to Thread Row" to navigate to the corresponding thread row:

Tips: CPU sampling has a high sampling rate, which can significantly increase the report file size. To disable CPU sampling, pass -s none to the asys command.

5.2 Thread scheduling

Requires one of the following collection options:

• --sample process-tree: Collects the process tree.

• --sample system-wide: Collects all processes on the system.

When CPU sampling is enabled, the thread timeline appears as follows:

The child nodes of a thread row show various API calls. The thread row itself displays the thread's execution on CPU cores in four sub-rows:

From top to bottom, these are:

• CPU utilization: Shows CPU resource usage.

• CPU core: Shows the core where the thread is scheduled. Its color corresponds to the color label of the CPU core node.

• Thread state, divided into three types:

  • Running

  • Not scheduled

  • Waiting

• Calling OSRT

• Backtrace sample point: Indicates a sample point. The tooltip provides detailed call stack information.

Hovering over an item displays more detailed information.

When CPU sampling is disabled but OSRT collection is enabled, Asight Systems estimates the thread state based on OSRT execution. Time slices spent in OSRT are considered CPU idle, while those outside OSRT are considered CPU busy, as shown below:

In this mode, the thread state is one of two estimated states:

• Possibly running

• Possibly waiting

When only backtrace sampling is enabled, the thread row displays sample points:

View function execution time statistics for the application in the Function View at the bottom of the GUI:

6. API call stack tracing

asys can collect call stack information for APIs such as HGGC, CUDA, OSRT, ACDNN, ACBLAS, CUDNN, and CUBLAS. You can configure it to collect call stacks only for APIs that exceed a runtime threshold and to set the maximum backtrace depth. Once call stack tracing is enabled, you can view the flame graph, icicle graph, and the Top Down, Bottom Up, and Flat tables for function calls in the Function View.

6.1 HGGC call stacks

Use the following command to enable HGGC tracing and collect call stack information:

asys profile -t hggc --hggcbacktrace all:1000 --hggc-backtrace-depth 24 python test_linear.py

• -t hggc: Enables HGGC tracing.

• --hggcbacktrace all:1000: Collects call stacks for all HGGC APIs whose runtime exceeds a threshold. The value after the colon (:) sets this threshold in nanoseconds (ns). In this example, the threshold is 1000 ns.

• --hggc-backtrace-depth 24: Sets the maximum backtrace depth for the HGGC call stack to 24 frames.

You can hover over the timeline to view the call stack:

6.2 OSRT call stacks

Use the following command to enable OSRT tracing and collect call stack information:

asys profile -t osrt --osrt-backtrace-threshold 80000 --osrt-backtrace-depth 24 python /test_script/ops/test_linear.py

• -t osrt: Enables OSRT tracing.

• --osrt-backtrace-threshold 80000: Sets the runtime threshold for collecting OSRT API call stacks to 80,000 nanoseconds (ns).

• --osrt-backtrace-depth 24: Sets the maximum backtrace depth for the OSRT call stack to 24 frames.

The call stack is shown below:

6.3 ACDNN and ACBLAS call stacks

Use the following command to enable ACDNN and ACBLAS tracing and collect call stack information:

asys profile -t acdnn,acblas --acompute-backtrace-threshold 80000 python /test_script/ops/test_linear.py

• -t acdnn,acblas: Enables ACDNN and ACBLAS tracing.

• --acompute-backtrace-threshold 80000: Sets the runtime threshold for collecting ACDNN and ACBLAS API call stacks to 80,000 nanoseconds (ns).

The call stack is shown below:

7. Tracking memory usage

7.1 Tracking PPU memory usage

asys can track an application's usage of device memory. It captures call stack information for device memory allocations and deallocations, and aggregates this information to provide device memory usage statistics. You can enable memory usage tracking with the --hggc-memory-usage option. For example:

asys profile -t hggc --hggc-memory-usage true python test_linear.py

When memory usage tracking is enabled, Asight Systems displays PPU memory usage on the timeline, allowing you to analyze memory consumption, as shown below:

The Device memory usage row shows the PPU memory consumption during program execution. Features include:

• Correlation with CUDA/HGGC API calls.

• call stack support.

• Display of memory allocation/deallocation events in the Events View.

If a PPU memory allocation fails, a small red dot appears in the Device memory usage row:

In the Device Memory View, you can view the Top Down, Bottom Up, and Flat tables.

Tips:

• You can control the call stack collection depth with the --hggc-backtrace-depth option. For example, --hggc-backtrace-depth 50 sets the maximum depth to 50 frames.

• This feature does not capture device memory deallocation by the HGGC destructor during process termination.

7.2 Sampling host memory usage

asys can periodically sample an application's system memory usage. You can enable system memory sampling with the --host-memory-sampling option and control the sampling frequency with the --host-memory-sampling-frequency option. For example:

asys profile -t hggc --host-memory-sampling true --host-memory-sampling-frequency 500 python test_linear.py

By sampling a process's system memory usage, you can view how system memory usage changes for each process in the Asight Systems GUI:

Two types of memory usage are displayed:

• Resident set size: The actual physical memory usage.

• Water mark: The peak physical memory usage during the collection period.

Tip: asys can collect system memory usage for an application's process tree (the root process and its descendants). However, if system-wide tracing is enabled (-s system-wide), the system memory collection feature is disabled.

7.3 Tracking heap memory usage

asys can track an application's heap memory usage, which is memory dynamically allocated on the CPU (e.g., memory allocated via malloc or new). It captures call stack information for these allocations and aggregates the data to provide usage statistics. Enable this feature with the --heap-memory-usage option. For example:

asys profile --heap-memory-usage true python test_linear.py

By tracking heap memory usage, you can observe how it changes over time for each process in the Asight Systems GUI:

Click a point on the Heap memory usage timeline to view flame graphs, icicle graphs, and other information about the process's heap memory usage up to that moment. This supports memory leak analysis and provides statistics on memory usage and allocation counts. For example:

You can use the --heap-sampling-interval-bytes option to adjust the average sampling interval bytes for dynamic memory allocations in asys. A smaller sampling interval improves collection accuracy for smaller dynamic memory allocations but significantly increases the performance overhead of asys trace collection.

You can use the --heap-backtrace-depth option to adjust the call stack collection depth for heap memory usage, and the --heap-report-interval option to adjust the aggregation interval for usage information. For example:

asys profile --heap-memory-usage true --heap-sampling-interval-bytes 16777216 --heap-backtrace-depth 50 --heap-report-interval 2 python test_linear.py

• --heap-memory-usage true: Enables heap memory usage tracking.

• --heap-sampling-interval-bytes 16777216: Sets the average sampling interval bytes to 16 MiB.

• --heap-report-interval 2: Aggregates and records heap memory usage information every 2 milliseconds.

Tips:

• You can use the --heap-sampling-interval-bytes option to set the average sampling interval bytes. To reduce the performance overhead of trace collection, asys may merge and report the usage of dynamically allocated memory that is smaller than this threshold.

• Setting --heap-sampling-interval-bytes=1 forces asys to capture every heap memory allocation individually. This significantly increases overhead and is not recommended for complex applications.

7.4 Analyzing pinned memory usage

asys can track an application's pinned memory usage. It captures call stack information for pinned memory allocations and deallocations, and aggregates this data to provide usage statistics. Pinned memory tracking is automatically enabled when you enable both device memory tracking with --hggc-memory-usage and system memory sampling with --host-memory-sampling. For example:

asys profile -t hggc --hggc-memory-usage true --host-memory-sampling true python test_linear.py

By tracking pinned memory usage, you can view its changes over time for each process in the Asight Systems GUI. You can also view aggregated device memory usage information for various APIs in the application, as shown below:

The figure above shows how pinned memory usage changes over time. The tooltip displays details for each memory change, including allocation/deallocation status and call stack information. Pinned memory events also appear in the Events View. In the Host Pinned Memory View, you can view flame graphs, icicle graphs, and the Top Down, Bottom Up, and Flat tables.

Tips:

• You can control the call stack collection depth with the --hggc-backtrace-depth option. The default depth is 24 frames. For example, --hggc-backtrace-depth 50 sets the maximum depth to 50 frames.

• You can track both device memory and pinned memory usage. To specify multiple values for an option, separate them with a ,. For example: --hggc-memory-usage device,pinned.

• This feature does not capture pinned memory deallocation by the HGGC destructor during process termination.

8. Python tracing

8.1 Collect python call stacks

8.1.1 Sample python call stacks (python sampling)

asys supports sampling the call stack of a running Python script, including function names, filenames, and line numbers. The Asight Systems GUI displays this call stack data:

Use the --python-sampling option to enable python call stack sampling, and use --python-sampling-frequency to control the sampling frequency. For example:

asys profile --python-sampling true --python-sampling-frequency 1000 python launch.py

• --python-sampling true enables python call stack sampling. This option is enabled by default.

• --python-sampling-frequency 1000 sets the sampling frequency. The valid range is 1 Hz to 1 kHz. The default is 500 samples per second (500 Hz).

• --python-backtrace-depth sets the call stack depth for python sampling. The default value is 24.

8.1.2 Capture python call stacks for specific events (python backtrace)

asys can capture the current python call stack when tracing specific events. asys then combines this call stack with the C call stack to display the complete call chain.

Enable this feature with --python-backtrace=hggc and control the call stack depth with --python-backtrace-depth. Currently, you can capture a python backtrace for the following events:

• hggc

  For example:

  asys profile --python-backtrace=hggc python launch.py

• device memory / pinned memory

  For example:

  asys profile --hggc-memory-usage=true --host-memory-sampling=true --python-backtrace=hggc python launch.py

Note:

• asys collects python call stacks only for Python 3.8 to 3.12.

• asys does not support python call stack collection for applications launched with an aliased Python interpreter (any name other than python or python3).

8.2 Trace python functions (python functions trace)

asys can collect execution information for python functions without modifying the source code and displays the information as HGTX ranges:

Use --python-functions-trace to specify which functions to trace with one of the following modes:

You can set a threshold for tracing python functions with the --python-functions-threshold option. asys only traces functions with a runtime duration exceeding this threshold. The value is in nanoseconds (ns), and the default is 80,000 ns. You can use the following three methods to specify functions in a JSON file:

• Specify only the module. This traces all functions within that module. You can specify a parent module. For example, specifying torch also traces all functions in its submodules, such as torch.nn.

  [
    {    
      "module": "torch.nn.functional"
    },
    {
      "module": "threading"
    }
  ]

• Specify a list of function names under a specific module (using the full module path). You can also set a custom domain (the default is 'Python Hgtx').

  [
    {    
      "module": "torch.nn.functional",
      "domain": "Torch Domain",
      "functions": ["sigmoid","log_softmax"]
    }
  ]

• Within a module's functions list, you can override attributes for individual functions. If a function object specifies its own module or domain, it overrides the parent's attributes for that function.

  [
    {    
      "module": "torch.nn.functional",
      "domain": "Parent Domain",
      "functions": ["sigmoid", {"function": "Adadelta.step", "module": "torch.optim.adadelta", "domain": "Child Domain"}]
    }
  ]

You can mix these methods in the JSON file. If there are conflicting rules, the last specified rule applies.

For example, to trace all functions in the torch module and set the runtime threshold to 1,000 ns:

asys profile --python-functions-trace module:torch --python-functions-threshold 1000 python xx.py

Note:

• Python functions trace supports Python 3.8 and later.

• In multithreaded applications, asys might delay data collection for newly created child threads.

• You cannot specify functions in the __main__ module.

• Function tracing may be inaccurate around the application's fork boundary.

9. PCCL activity tracking

asys tracks PCCL activity by collecting activity events. Use the -t pccl option to enable this feature. For example:

asys profile -t pccl hggc python test_pccl.py

The timeline is shown below:

The PCCL timeline groups events by channel, and then by pipeline within each channel.

10. RDMA NIC runtime metrics

asys periodically collects runtime metrics for RDMA NICs in the system, such as transmit/receive byte counts, average packet sizes, and congestion control data. You can track how these metrics change over time in the Asight Systems GUI:

Supported metrics include:

• network receive rate

• network send rate

• network receive packet size

• network send packet size

• network receive CNP rate

• network send CNP rate

• send wait

When you select a time range, a tooltip displays the average network transfer rate.

Use the --nic-metrics option to enable NIC runtime metrics collection. asys then collects runtime data for all NIC devices in the system and displays the active devices in the GUI. For example:

asys profile --nic-metrics true ./all_reduce_perf

• --nic-metrics true: Enables NIC runtime metrics collection.

Use the --nic-device-include option to specify regular expressions for filtering NIC devices by name. Separate multiple expressions with a comma. asys collects metrics from any NIC whose name matches one of the expressions. For example:

asys profile --nic-metrics true --nic-device-include "bond" ./all_reduce_perf

• --nic-device-include "bond": Collects metrics for NIC devices with names containing "bond".

asys profile --nic-metrics true --nic-device-include "bond,eth" ./all_reduce_perf

• --nic-device-include "bond,eth": Collects metrics for NIC devices with names containing either "bond" or "eth".

Use the --nic-metrics-frequency option to control the sampling frequency. The valid range is 1 Hz to 100 kHz, and the default is 1 kHz.

asys profile --nic-metrics true --nic-metrics-frequency 10000 ./all_reduce_perf

Note: Due to RDMA driver limitations, the actual sampling frequency might not reach your configured value. If you set a frequency above the maximum supported rate, asys collects data at that maximum rate.