Expert system analysis report

2.3.1 Grouped device memory usage

Description

Analyzes and summarizes unreleased PPU device memory allocations from an asysrep report. This feature groups memory usage by category, such as operator library or framework, and provides a summary and detailed information for each group.

Required asys collection option: --hggc-memory-usage true

Command-line usage: Use the -r <rule>:usage-mode=<mode> option to specify the data type.

The device_memory_usage_detail rule supports two data types:

• 'all': Exports traces for all memory allocations and deallocations.

• 'unreleased': Exports traces for leaked memory (default).

The device_memory_usage_summary rule supports four data types:

• 'alloc-count': Summarizes the total number of memory allocations.

• 'alloc-size': Summarizes the total size of memory allocations.

• 'unreleased-count': Summarizes the total number of leaked memory allocations.

• 'unreleased-size': Summarizes the total size of leaked memory (default).

Use the --callstack-group-config option to add grouping rules. Use the :match-first-group option to specify that an allocation is assigned only to the first matching group. By default, an allocation is assigned to all matching groups.

Use the --report option to specify a grouped statistics report. For example:

asys stats -r device_memory_usage_summary:usage-mode=unreleased-size:match-first-group \
--callstack-group-config "acompute=(libacblas|libacdnn)" \
--callstack-group-config "launch_kernel=libtorch/LaunchKernel" \
--callstack-group-config "loss=loss.py" \
report.asysrep

• -r device_memory_usage_summary: Specifies the device memory usage summary report.

• :usage-mode=unreleased-size: Specifies summarizing the size of leaked memory.

• :match-first-group: Specifies that an allocation belongs only to the first group it matches.

• --callstack-group-config "acompute=(libacblas|libacdnn)":

  • Creates a group named acompute.

  • Matching regular expression (libacblas|libacdnn): Matches memory allocations where the callstack contains the libacblas or libacdnn keyword.

• --callstack-group-config "launch_kernel=libtorch/LaunchKernel":

  • Creates a group named launch_kernel.

  • Matching regular expression libtorch/LaunchKernel: Matches memory allocations based on callstack hierarchy where a parent function's frame in the callstack contains the keyword libtorch and a child function's frame contains the keyword LaunchKernel.

    • Unmatched callstack frames can exist between the matching parent and child function frames.

• --callstack-group-config "loss=loss.py"

  • Creates a group named loss.

  • Matching regular expression loss.py: Matches memory allocations where the callstack contains the loss.py keyword.

    • Python callstacks are supported for matching.

The following example shows the grouped statistical results. The output is in CSV format and can be saved to a file using the --output option.

device_memory_usage_summary report:

PID,Device ID,Group List,Memory Usage,
1873,0,launch_kernel,213174,
1873,0,others,48674898730,
1873,0,acompute,1610624066,
1873,0,loss,28591

• The report provides a summary of unreleased memory for the acompute, launch_kernel, and loss groups, measured in bytes.

• Memory allocations that do not match any user-defined group are aggregated into the default others group.

The device_memory_usage_summary table contains the following columns:

Row# : Row number of the device memory usage
PID : Process identifier
Device ID : PPU device identifier
Group List : List of groups the memory allocation belongs to, separated by '/'
Memory Usage [bytes] : Device memory usage

The device_memory_usage_detail table contains the following columns:

Row# : Row number of the device memory usage
PID : Process identifier
Device ID : PPU device identifier
Group List : List of groups the memory allocation belongs to, separated by '/'
Time [ns] : Memory usage timestamp
TID : Thread identifier
Context ID : Context identifier
Memory : Memory identifier
Size [bytes] : Memory usage size
Access Flag : Memory access flag
Event ID : Memory usage event identifier

The device memory usage grouping feature provides two report types, which you can specify using the --report option:

• device_memory_usage_summary: Outputs a summary of unreleased memory for each process, device, and group.

• device_memory_usage_detail: Outputs each unreleased memory allocation record along with the group it belongs to.

For detailed information about the device memory usage grouping feature, including its description and output format, use the --help-reports option. For example, running asys stats --help-report device_memory_usage_summary displays the following output:

root@eb4c64fd3401:~# asys stats --help-report device_memory_usage_summary
device_memory_usage_summary -- Device Memory Usage Summary

    Options:
        match-first-group
            Optional argument. When used with --callstack-group-config:
            If given, only matching the first callstack group.
            Default is matching all callstack group.
        usage-mode=<mode>
            Possible values are 'alloc-size', 'unreleased-size', 'alloc-count' or 'unreleased-count'.
            Specify the memory usage mode.
            If 'alloc-size', statistic overall allocated device memory usage.
            If 'unreleased-size', statistic allocated but not freed device memory usage.
            If 'alloc-count', statistic overall allocated device memory count.
            If 'unreleased-count', statistic allocated but not freed device memory count.
            Default is 'unreleased-size'.
        Use --filter-time / --filter-hgtx to specify report time range.
        Use --callstack-group-config to create report group configuration.
        Try 'asys stats --help' for more information.
        
    Output:
        Row# : Row number of the device memory usage
        PID : Process identifier
        Device ID : PPU device identifier
        Group List : Memory usage belonged group list, splited by '/'
        Memory Usage [bytes] : Device memory usage

    Group and statistic device memory usage of specified mode,
    If 'alloc-size' mode,  display allocated device memory usage summary.
    If 'alloc-count' mode, display allocated device memory count summary.
    If 'unreleased-size' mode, display allocated but not freed device memory usage summary.
    If 'unreleased-count' mode, display allocated but not freed device memory count summary.

When a memory allocation record matches multiple groups, the report shows all the groups it belongs to, separated by a forward slash (/). For example:

acompute/launch_kernel

• The memory allocation belongs to both the acompute and launch_kernel groups.

• If the grouping policy is set to match the first group, the allocation is assigned only to the first group that meets the criteria.

• The order of the groups is determined by the order in which they are created using the --callstack-group-config option.

Use the --callstack-group-config option to create groups and specify matching rules with regular expressions. The format is group_name=frame_filters, where frame_filters specifies one or more regular expressions for frames in the format frame_regex/frame_regex.... The frame regular expressions are separated by a forward slash (/) and are ordered from parent function to child function. The group matching rules are as follows:

• A frame matches if its regular expression partially matches the library name or function signature in the callstack.

• A group matches if the hierarchy of matched frames in the callstack corresponds to the one specified in frame_filters.

The following example uses the --callstack-group-config option multiple times to create several groups, each with a unique rule for matching frames:

--callstack-group-config "torch=libtorch" --callstack-group-config "acompute=(libacblas|libacdnn)" --callstack-group-config "buffer_init=_to_copy/empty_strided"

Tips:

1. If you do not add grouping rules with the --callstack-group-config option, the system applies the default built-in rules.

2. If you use multiple --callstack-group-config options with the same group name, any memory allocation record that matches any of these rules is assigned to that group.

GUI guide

• Settings

  • HGTX/NVTX range and/or domain to filter by: Sets the analysis interval. The format for specifying an HGTX/NVTX range is range[@domain][/index]. For more information, see Set the statistics range.

  • Usage mode: Specifies the type of statistical data.

    • The Device Memory Usage Detail rule supports two data types:

      • Unreleased Alloc Trace: Exports traces for leaked memory (default).

      • All Alloc and Free Trace: Exports traces for all memory allocations and deallocations.

    • The Device Memory Usage Summary rule supports four data types:

      • Unreleased Alloc Size: Summarizes the size of leaked memory (default).

      • Total alloc size: Summarizes the total size of memory allocations.

      • Unreleased Alloc Count: Summarizes the number of leaked memory allocations.

      • Total alloc count: Summarizes the total number of memory allocations.

  • Group config: Creates groups and specifies matching rules using regular expressions. The format is group_name=frame_filters, where frame_filters can specify one or more regular expressions for frames in the format frame_regex1/frame_regex2/.... The frame regular expressions are separated by a forward slash (/) and are ordered from parent function to child function.

  • Only match the first group in order: When selected, an allocation is assigned only to the first matching group in top-to-bottom order. By default, this option is not selected, which means allocations are assigned to all matching groups.

• Context menu features

  • The context menu for the Device Memory Usage Detail rule lets you highlight or zoom to the selected memory trace in the Timeline View.

Statistics logic

This feature summarizes unreleased PPU device memory allocations on a per-process, per-PPU basis.

The system matches each unreleased memory allocation against the grouping rules based on its callstack information at the time of allocation, including frame order and matched keywords.

A memory allocation record may match multiple groups. The output shows all groups to which the record belongs, separated by a forward slash (/).

The tool summarizes memory usage for each group combination and sorts the results in descending order by total memory usage.

2.3.2 HGGC API summary

Description

Summarizes the execution time of HGGC API calls from an .asysrep report and sorts the results by total time in descending order.

Required asys collection option: --trace hggc

Command-line usage:

asys stats -r hggc_api_sum report.asysrep

Run asys stats --help-report hggc_api_sum for detailed help.

Example report output:

hggc_api_sum report:

Row#,Time (%),Total Time (ns),Num Calls,Avg (ns),Med (ns),Min (ns),Max (ns),StdDev (ns),Name,
1,84.9,393100090,342,1149415,98746,3981,11001565,2011747,"hgMemcpyHtoDAsync_v2",
2,4.3,19851112,1297,15305,5351,2893,3860175,147778,"hgLaunchKernel",
3,2.9,13492044,1486,9079,2861,613,3707197,108998,"hgEventQuery",
4.2.9,13487638,1495,9021,6027,3191,222379,14733,"hggcLaunchKernel",
...

The following list defines each column in the hggc_api_sum table:

Note: The "Time" column shows a function's "Total Time" as a percentage of the sum of the "Total Time" values for all functions in the report. It is not a percentage of the application's total wall time or CPU time.

Row# : Row number of the HGGC API summary
Time (%) : Percentage of "Total Time"
Total Time (ns) : Total execution time for this function
Num Calls : Number of calls to this function
Avg (ns) : Average execution time of this function
Med (ns) : Median execution time of this function
Min (ns) : Minimum execution time of this function
Max (ns) : Maximum execution time of this function
StdDev (ns) : Standard deviation of the function's execution times
Name : Name of the function

GUI guide

• Settings

  • HGTX/NVTX range and/or domain to filter by: Specifies the range for which statistics are gathered. The format for specifying an HGTX/NVTX range is: range[@domain][/index]. For more information, see Set the statistics range.

Statistics logic

The system groups calls by HGGC API name, sums their execution times, and sorts the aggregated results by total time in descending order.

2.3.3 HGGC API trace

Exports HGGC API trace data from an asysrep report, sorted in ascending order by API duration.

Required asys collection option: --trace hggc

Command line usage:

asys stats -r hggc_api_trace report.asysrep

Run asys stats --help-report hggc_api_trace for detailed help.

The following is an example.

The hggc_api_trace report:

Row#,Start (ns),Duration (ns),Name,CorrID,Pid,Tid,Thread Name,
1,41550,12737,"cudaProfilerStart",0,104699,104699,"python",
2,938476,28340,"cudaLaunchKernel",130170,104699,104699,"python",
3,999414,9382,"cudaLaunchKernel",130171,104699,104699,"python",
...

The hggc_api_trace report includes the following columns:

Row# : Row number of the HGGC API trace
Start [ns] : Timestamp when the API call started
Duration [ns] : Duration of the API call
Name : API function name
CorrID : Correlation used to map to other HGGC traces
Pid : Process ID of the process that made the call
Tid : Thread ID of the thread that made the call
Thread Name : Name of the thread that made the call

GUI

• Settings

  • HGTX/NVTX range and/or domain to filter by: Sets the statistical range. The format for specifying an HGTX/NVTX range is: range[@domain][/index]. For details, see Set the statistical range.

• Context menu features

  • Lets you highlight or zoom to the selected HGGC API in the Timeline View.

Export rule

Each HGGC API call is exported as a single row.

2.3.4 HGGC kernel execution trace

Description

Exports trace data for HGGC kernels from an asysrep report, tracking the execution from the API launch to the actual kernel execution. The output is sorted in ascending order by the start time of the launch API.

Required asys collection option: --trace hggc

Command-line usage:

asys stats -r hggc_kern_exec_trace report.asysrep

Run asys stats --help-report hggc_kern_exec_trace for detailed help. Supported options include:

• base: Exports the short name of the kernel (function name only, without parameters).

• mangled: Exports the mangled name of the kernel.

By default, the exported HGGC kernel name is the demangled name.

Example report output:

hggc_kern_exec_trace report

Row#,API Start (ns),API Dur (ns),Queue Start (ns),Queue Dur (ns),Kernel Start (ns),Kernel Dur (ns),Total Dur (ns),PID,TID,DevId,API Function,GridXYZ,BlockXYZ,Kernel Name,
1,504935256,20348911,525284167,21237024,526172280,357034784,378271808,631660,631660,0,"hggcLaunchKernel","2 1 1","512 1 1","pcclKernel_AllReduce_RING_LL_Sum_int8_t(pcclWorkElem)",
2,845563829,20332834,865896663,20934656,866498485,17898735,38833391,631660,631660,1,"hggcLaunchKernel","2 1 1","512 1 1","pcclKernel_AllReduce_RING_LL_Sum_int8_t(pcclWorkElem)",
3,884469702,40488,884510190,627234,885096936,16944871,17572105,631660,631660,0,"hggcLaunchKernel","2 1 1","512 1 1","pcclKernel_AllReduce_RING_LL_Sum_int8_t(pcclWorkElem)",
...

The hggc_kern_exec_trace report contains the following columns:

Row# : Row number of the kernel trace
API Start [ns] : Start timestamp of HGGC API launch call
API Dur [ns] : Duration of HGGC API launch call
Queue Start [ns] : Start timestamp of queue wait time, if it exists
Queue Dur [ns] : Duration of queue wait time, if it exists
Kernel Start [ns] : Start timestamp of HGGC kernel
Kernel Dur [ns] : Duration of HGGC kernel
Total Dur [ns] : Duration from API start to kernel end
PID : process ID that made the kernel launch call
TID : thread ID that made the kernel launch call
DevId : HGGC device ID that executed the kernel (which PPU)
API Function : Name of the HGGC API call used to launch the kernel
GridXYZ : grid dimensions for the kernel launch call
BlockXYZ : block dimensions for the kernel launch call
Kernel Name : Name of the HGGC kernel

GUI guide

• Settings

  • HGTX/NVTX range and/or domain to filter by: Specifies the filter range. The format for specifying an HGTX/NVTX range is: range[@domain][/index]. For more information, see Setting the Filter Range.

  • Kernel name mode: Specifies the display mode for the kernel name. Three modes are available:

    • Base: The short name of the kernel (function name only, without parameters).

    • Mangled: The mangled name of the kernel.

    • Demangled: The demangled name of the kernel (default).

• Context menu features

  • Lets you highlight or zoom to the selected HGGC API in the Timeline View.

  • Lets you highlight or zoom to the selected Device Activity in the Timeline View.

Calculation rules

Each kernel execution is exported as a single row, sorted in ascending order by the start time of the corresponding launch API. In scenarios where a single HGGC API is associated with multiple kernels, such as in an HGGC Graph, a separate row is exported for each kernel. The queue wait time columns (Queue Start and Queue Dur) are calculated as follows:

• A kernel is considered to have a wait time if its actual execution starts after the corresponding HGGC API call finishes.

• The wait time is calculated as: actual kernel start time - API call end time.

• If a kernel begins execution before the corresponding HGGC API call finishes, the Queue Start and Queue Dur columns are marked with a hyphen (-) to indicate an invalid value.

2.3.5 HGGC kernel grid and block

Description

Aggregates and summarizes data on HGGC kernel launches from an asysrep report, including grid and block dimensions. The output provides statistics for each unique combination of kernel name, grid size, and block size.

Required asys collection options: --trace hgtx,hggc

Command-line usage:

asys stats -r hggc_ppu_kern_gb_sum report.asysrep

Run asys stats --help-report hggc_ppu_kern_gb_sum for detailed help. Supported options include:

• hgtx-name: Prefixes the kernel name with the name of the HGTX range closest to the kernel launch, separated by a /.

• base: Aggregates and outputs statistics using the kernel's base name (function name only, without parameters).

• mangled: Aggregates and outputs statistics using the kernel's mangled name.

• device: Specifies a list of PPU device IDs to include in the statistics. Separate multiple IDs with a /. If this option is not specified, data from all PPU devices is included by default.

Example report output:

Row#,Time (%),Total Time (ns),Instances,Avg (ns),Med (ns),Min (ns),Max (ns),StdDev (ns),GridXYZ,BlockXYZ,Name,
1,6.7,16337781,65,251350,260801,115520,277681,39767,"960 1 64","256 1 1","[prof_range]: iter 9/_ZN5acdnn4cuda9transposeILNS0_8LoopModeE0ELi32ELi8ELb0ELb0EN7",
2,3.1,7415267,20,370763,370601,366482,374162,1936,"2048 1 1","512 1 1","[prof_range]: iter 8/_ZL35batch_norm_bwd_single_vector_accessILb0EN5acdnn16identity",
3,3.0,7348069,20,367403,366401,361281,375921,3885,"122880 1 1","128 1 1","_ZN2at6native29vectorized_elementwise_kernelILi4EZZZNS0_12",

The following describes the columns in the hggc_ppu_kern_gb_sum table:

Note: The 'Time' column shows the percentage of a kernel's 'Total Time' relative to the total time of all kernels in the report. It is not a percentage of the application's wall time or CPU execution time.

Row# : Row number of the kernel summary
Time (%) : Percentage of 'Total Time'
Total Time (ns) : Total time used by all executions of this kernel
Instances : Number of calls to this kernel
Avg (ns) : Average execution time of this kernel
Med (ns) : Median execution time of this kernel
Min (ns) : Minimum execution time of this kernel
Max (ns) : Maximum execution time of this kernel
StdDev (ns) : Standard deviation of this kernel's execution time
GridXYZ : Grid dimensions for the kernel launch
BlockXYZ : Block dimensions for the kernel launch
Name : Name of the kernel

GUI user guide

• Settings

  • HGTX/NVTX range and/or domain to filter by: Defines the statistics range. The format for specifying an HGTX/NVTX range is: range[@domain][/index]. For more information, see Set the statistics range.

  • Kernel name mode: Specifies the display mode for the kernel name.

    • Base: Displays the kernel's base name (function name only, without parameters).

    • Mangled: Displays the kernel's mangled name.

    • Demangled: Displays the kernel's demangled name (default).

  • PPU devices: Filters the results by the selected PPU devices. The default is 'All', which includes data from all PPU devices.

  • Add HGTX name as a prefix: When enabled, this option prefixes the kernel name with the name of the HGTX range closest to the kernel launch, separated by a /. This option is disabled by default.

Aggregation rules

Aggregation method:

• Execution data, such as total time and number of instances, is aggregated for kernels with the same kernel name, grid size, and block size.

• If the hgtx-name option is enabled, the kernel name includes the prefixed HGTX range name.

A single row is exported for each unique combination of kernel name, grid size, and block size. The output is sorted in descending order by the 'Total Time' column.

2.3.6 HGGC kernel grid and block trace

Description

Exports trace data for HGGC kernel executions from an asysrep report, including grid and block dimensions. The output is sorted in ascending order by execution start time.

Required asys collection option: --trace hgtx,hggc

Command-line usage:

asys stats -r hggc_ppu_kern_gb_trace report.asysrep

Run asys stats --help-report hggc_ppu_kern_gb_trace for detailed help. Supported options include:

• hgtx-name: Prefixes the kernel name with the name of the HGTX range closest to the kernel launch, separated by a forward slash (/).

• base: Aggregates and outputs statistics using the kernel's short name, which includes only the function name without parameters.

• mangled: Aggregates and outputs statistics using the kernel's mangled name.

Example output:

Row#,Start (ns),Duration (ns),PID,Device ID,Context ID,Stream ID,GridXYZ,BlockXYZ,Name,
1,970508,6240,104699,0,1,1,"3 1 1","128 1 1","[prof_range]: iter 5/unrolled_elementwise_kernel",
2,1014188,44480,104699,0,1,1,"15360 1 1","128 1 1","[prof_range]: iter 5/unrolled_elementwise_kernel",
3,1158469,98480,104699,0,1,1,"960 1 64","256 1 1","[prof_range]: iter 5/transpose",
4,1257269,1880,104699,0,1,1,"1 1 32","256 1 1","[prof_range]: iter 5/transpose",

The hggc_ppu_kern_gb_trace table contains the following columns:

Row# : Row number of the kernel trace
Start [ns] : Timestamp of start time
Duration [ns] : Length of event
PID : Process identifier
Device ID : PPU device identifier
Context ID : Context identifier
Stream ID : Stream identifier
GridXYZ : Grid dimensions for kernel launch call
BlockXYZ : Block dimensions for kernel launch call
Name : Name of the kernel

GUI guide

• Settings

  • HGTX/NVTX range and/or domain to filter by: Sets the statistics range. The format for specifying an HGTX/NVTX range is: range[@domain][/index]. For more information, see Set the statistics range.

  • Kernel Name Mode: Specifies the display mode for the kernel name:

    • Base: Exports the kernel's short name, which includes only the function name without parameters.

    • Mangled: Exports the kernel's mangled name.

    • Demangled: Exports the kernel's demangled name (default).

  • Add HGTX name as a prefix: When enabled, prefixes the kernel name with the name of the HGTX range closest to the kernel launch, separated by a /. This option is not selected by default.

• Context menu features

  • Allows you to highlight or zoom to the selected Device Activity in the Timeline View.

Statistics rules

Each kernel execution is exported as a single row, and the output is sorted in ascending order by execution start time.

2.3.7 HGGC PPU kernel summary

Description

Summarizes the execution time of HGGC PPU kernels from an asysrep report and sorts the results by total time in descending order.

Required asys collection option: --trace hggc

Command line usage:

asys stats -r hggc_ppu_kern_sum report.asysrep

Run asys stats --help-report hggc_ppu_kern_sum for detailed help. Supported options include:

• hgtx-name: Prefixes the kernel name with the name of the HGTX range closest to the kernel launch, using a forward slash (/) as a separator.

• base: Aggregates and outputs statistics using the kernel's short name (function name only, without parameters).

• mangled: Aggregates and outputs statistics using the kernel's mangled name.

• device: Specifies a list of PPU device IDs to include in the statistics. Separate multiple IDs with a /. If you omit this option, the tool aggregates data from all PPU devices by default.

Example report output:

hggc_ppu_kern_sum report:

Row#,Time (%),Total Time (ns),Instances,Avg (ns),Med (ns),Min (ns),Max (ns),StdDev (ns),Name,
1,81.6,826473830,28,29516922,17075623,16812011,357034784,64193935,"_Z39pcclKernel_AllReduce_RING_LL_Sum_int8_t12ncclWorkElem",
2,13.9,141273640,4,35318410,34326818,31248932,41371071,4288359,"_Z9deltaKernIaLi256EEvPvS0_mPd",
3,2.3,23263744,4,5815936,5866699,4743763,6786583,1087875,"_Z14InitDataKernelIaEvPT_mii",
4,2.2,22033972,4,5508493,5461251,4528464,6583005,1031905,"_Z20InitDataReduceKernelIaXadL_Z9ncclOpSumIaET_S1_S1_EEEvPS1_mmii",

The columns in the hggc_ppu_kern_sum table are as follows:

Note: The Time column shows the percentage of a kernel's Total Time relative to the combined total time of all kernels in the report. It is not a percentage of the application's total wall time or CPU time.

Row# : Row number of the kernel summary
Time [%] : Percentage of total execution time
Total Time [ns] : Total time used by all executions of this kernel
Instances : Number of calls to this kernel
Avg [ns] : Average execution time of this kernel
Med [ns] : Median execution time of this kernel
Min [ns] : Minimum execution time of this kernel
Max [ns] : Maximum execution time of this kernel
StdDev [ns] : Standard deviation of this kernel's execution time
Name : Kernel name

GUI usage

• Settings

  • HGTX/NVTX range and/or domain to filter by: Sets the statistics range. The format for specifying an HGTX/NVTX range is: range[@domain][/index]. For details, see Set the statistics range.

  • Kernel Name Mode: Specifies the display mode for the kernel name. You can choose from three modes:

    • Base: Displays the kernel's short name (function name only, without parameters).

    • Mangled: Displays the kernel's mangled name.

    • Demangled: Displays the kernel's demangled name (default).

  • PPU Devices: Filters the results by the selected PPU devices. The default is "All", which includes data from all PPU devices.

Statistics logic

The report aggregates statistics for each HGGC PPU kernel by name. The name used for aggregation depends on whether the base or mangled option is specified. By default, the report uses the demangled kernel name, which includes the function parameter list. The results are sorted by total time in descending order.

2.3.8 HGGC PPU trace

Description

Exports trace data from an asysrep report for PPU executions of HGGC kernels, Memcpy, and Memset. The output is sorted in ascending order by execution start time.

Required asys collection option: --trace hggc

Command-line usage:

asys stats -r hggc_ppu_trace report.asysrep

Run asys stats --help-report hggc_ppu_trace for detailed help. Supported options include:

• hgtx-name: Prefixes the kernel name with the name of the HGTX range closest to the kernel launch, separated by a forward slash (/).

• base: Exports the kernel's short name (function name only, without parameters).

• mangled: Exports the kernel's mangled name.

By default, the exported HGGC kernel name is the demangled name. In the example report output, inapplicable cells contain the invalid value "-".

Sample output:

Row#,Start (ns),Duration (ns),CorrId,GrdX,GrdY,GrdZ,BlkX,BlkY,BlkZ,Reg/Trd,StcSMem (bytes),DymSMem (bytes),Bytes (bytes),Throughput (B/s),SrcMemKd,DstMemKd,Device,Ctx,Strm,Name,
1,193282492,3009,5,"-","-","-","-","-","-","-","-","-",256,85078016,"Device","-","",1,1,"Memset",
2,193329286,1332,6,"-","-","-","-","-","-","-","-","-",256,192192000,"Pageable","Device","",1,1,"Memcpy HtoD (device)",
3,193380562,3415,7,"-","-","-","-","-","-","-","-","-",256,74963200,"Pageable","Device","",1,1,"Memcpy HtoD (device)",
4,231829389,4562345,8,64,1,1,1,1,1,32,0,0,"-","-","-","-","",1,1,"add(int*, int*, int*)",

The following describes the columns in the hggc_ppu_trace table:

Row# : Row number of the PPU trace
Start [ns] : Start timestamp
Duration [ns] : Event duration
CorrId : Correlation ID
GrdX : Grid X values
GrdY : Grid Y values
GrdZ : Grid Z values
BlkX : Block X values
BlkY : Block Y values
BlkZ : Block Z values
Reg/Trd : Registers per thread
StcSMem [bytes] : Size of Static Shared Memory
DymSMem [bytes] : Size of Dynamic Shared Memory
Bytes [bytes] : Size of memory operation
Throughput [B/s] : Memory throughput
SrcMemKd : Memcpy source memory kind or memset memory kind
DstMemKd : Memcpy destination memory kind
Device : PPU device name and ID
Ctx : Context ID
Strm : Stream ID
Name : Trace event name

GUI Usage Guide

• Settings

  • HGTX/NVTX range and/or domain to filter by: Sets the filter range. The format is: range[@domain][/index]. For more information, see Set the filter range.

  • Kernel Name Mode: Specifies the display mode for the kernel name. The available modes are:

    • Base: Exports the kernel's short name (function name only, without parameters).

    • Mangled: Exports the kernel's mangled name.

    • Demangled: Exports the kernel's demangled name (default).

• Context menu features

  • You can highlight or zoom to the selected kernel or memory operation in the Timeline View.

Statistical Rules

Each row represents a single kernel, Memcpy, or Memset execution. The report sorts the output in ascending order by execution start time.

Because the output columns contain information for kernels, Memcpy, and Memset, inapplicable cells in a row are marked with the invalid value "-".

2.3.9 HGTX-associated kernel summary

Description

Aggregates and summarizes trace data from an asysrep report to show the relationship between HGTX ranges and their associated HGGC kernels. The report provides statistics for each unique combination of HGTX range name and kernel name.

Required asys collection option: --trace hgtx,hggc

Command-line usage:

asys stats -r hgtx_kern_sum report.asysrep

Run asys stats --help-report hgtx_kern_sum for detailed help. Supported options include:

• base: Exports the kernel's short name (function name only, without parameters).

• mangled: Exports the kernel's mangled name.

• standalone: Includes HGGC kernels that are not associated with any HGTX range.

• device: Specifies a list of PPU device IDs to include in the statistics. Separate multiple IDs with a /. If this option is omitted, data from all PPU devices is aggregated.

Example report output:

hgtx_kern_sum report:

Row#,HGTX Range,Style,PID,TID,HGTX Inst,Kern Inst,Total Time (ns),Avg (ns),Med (ns),Min (ns),Max (ns),StdDev (ns),Kernel Name,
1,"DoProcess","PushPop",3588556,3588556,2,2,8036961,4018480,4018480,3474616,4562345,543864,"add(int*, int*, int*)",
2,"Loop1","PushPop",3588556,3588556,1,1,4562345,4562345,4562345,4562345,4562345,0,"add(int*, int*, int*)",
3,"Loop2","PushPop",3588556,3588556,1,1,3474616,3474616,3474616,3474616,3474616,0,"add(int*, int*, int*)",
4,"profile","PushPop",3588556,3588556,1,2,8036961,4018480,4018480,3474616,4562345,543864,"add(int*, int*, int*)",

The hgtx_kern_sum report includes the following columns:

Row#: Row number of the HGTX range kernel summary
HGTX range: Name of the range
Style: Range style; Start/End or Push/Pop
PID: process ID for this set of ranges and kernels
TID: thread ID for this set of ranges and kernels
HGTX Inst: Number of HGTX range instances
Kern Inst: Number of HGGC kernel instances
Total Time (ns): Total execution time for all kernel instances in this row
Avg (ns): Average kernel execution time
Med (ns): Median execution time of this kernel
Min (ns): Minimum kernel execution time
Max (ns): Maximum kernel execution time
StdDev (ns): Standard deviation of kernel execution time
Kernel Name: The kernel's name

GUI

• Settings

  • HGTX/NVTX range and/or domain to filter by: Sets the filter range. The format for specifying an HGTX/NVTX range is: range[@domain][/index]. For more information, see Set the filter range.

  • Kernel Name Mode: Specifies the display mode for the kernel name:

    • Base: Exports the kernel's short name (function name only, without parameters).

    • Mangled: Exports the kernel's mangled name.

    • Demangled: Exports the kernel's demangled name (default).

  • Include Standalone Kernel: When enabled, includes kernels that are not associated with any HGTX range. For these kernels, HGTX-related fields contain invalid values. This option is disabled by default.

  • PPU Devices: Filters results by PPU device. The default, "All", includes data from all PPU devices.

Statistics logic

An HGGC kernel is associated with an HGTX range if the HGGC API that launched the kernel was called from the same thread while the HGTX range was active.

Aggregation method:

• For each process and thread, the system aggregates execution data, such as execution time, HGTX instance count, and kernel instance count, for kernels that share the same HGTX range name and kernel name.

The results are organized with one row for each unique combination of thread, HGTX range name, and HGGC kernel name. If the standalone option is enabled, rows for HGGC kernels without an associated HGTX range show a hyphen (-) in HGTX-related columns to indicate an invalid value. The results are sorted as follows:

• Primary sort: Sorted ascending by HGTX range name, process ID, and thread ID.

• Secondary sort: For kernels within the same thread and HGTX range, results are sorted descending by the 'Total Time' column.

2.3.10 HGTX-associated kernel trace

Description

This report lists the associations between HGTX ranges and HGGC kernels. Each entry represents one association.

Required asys collection option: --trace hgtx,hggc

Command-line usage:

asys stats -r hgtx_kern_trace report.asysrep

Run asys stats --help-report hgtx_kern_trace for detailed help. Supported options include:

• base: Exports the kernel's short name, which includes only the function name without parameters.

• mangled: Exports the kernel's mangled name.

• standalone: Includes HGGC kernels that are not associated with any HGTX range in the output.

Example report output:

hgtx_kern_trace report:

Row#,Range name,Style,PID,TID,Device ID,HGTX range ID,Kernel exec ID,Kernel Start (ns),Kernel duration (ns),API Start (ns),API duration (ns),GridXYZ,BlockXYZ,Kernel name,
1,"profile","PushPop",3588556,3588556,0,6,20,257324264,4562345,218889819,37829255,"64 1 1","1 1 1","add(int*, int*, int*)",
2,"profile","PushPop",3588556,3588556,0,6,40,291234962,3474616,290293060,17365,"64 1 1","1 1 1","add(int*, int*, int*)",
3,"Loop1","PushPop",3588556,3588556,0,7,20,257324264,4562345,218889819,37829255,"64 1 1","1 1 1","add(int*, int*, int*)",
...

hgtx_kern_trace table column descriptions:

Row# : Row number of the HGTX range kernel trace
Range name : Name of the HGTX range
Style : Range style; Start/End or Push/Pop
PID : Process identifier
TID : Thread identifier
Device ID : PPU device identifier
HGTX range ID : HGTX range identifier
Kernel exec ID : Kernel execution identifier
Kernel Start [ns] : Start timestamp of HGGC kernel
Kernel duration [ns] : Duration of HGGC kernel
API Start [ns] : Start timestamp of API call
API duration [ns] : Duration of API call
GridXYZ : Grid dimensions for kernel launch call
BlockXYZ : Block dimensions for kernel launch call
Kernel name : Name of the kernel

GUI usage

• Settings

  • HGTX/NVTX range and/or domain to filter by: Sets the range for analysis. The format for specifying an HGTX/NVTX range is: range[@domain][/index]. For details, see Set the statistics range.

  • Kernel name mode: Specifies the display mode for the kernel name.

    • Base: Displays the kernel's short name (the function name only, without parameters).

    • Mangled: Displays the kernel's mangled name.

    • Demangled: Displays the kernel's demangled name (default).

  • Include standalone kernel: Includes kernels that are not associated with any HGTX range. For these kernels, HGTX-related fields show a hyphen ("-"). This option is disabled by default.

• Context menu features

  • Highlight or zoom to the selected HGTX range in the Timeline View.

  • Highlight or zoom to the selected Device Activity in the Timeline View.

Statistics rule

An HGGC kernel is associated with an HGTX range if the HGGC API call that launched the kernel occurred on the same thread during the HGTX range's duration.

Each row in the results represents one association between an HGTX range and an HGGC kernel. If the standalone option is enabled, the HGTX-related columns for unassociated HGGC kernels show a hyphen ("-").

Sort Order:

• HGGC kernels are sorted in ascending order by their PPU execution start time.

• Rows for HGTX ranges associated with the same HGGC kernel are sorted in ascending order by the HGTX range's start time.

HGTX to PPU projection summary

Description

Projects CPU-side HGTX ranges from an asysrep report onto the PPU timeline to calculate statistics about their active time on the PPU.

Required asys collection options: --trace hgtx,hggc

Command line usage:

asys stats -r hgtx_ppu_proj_sum report.asysrep

Run asys stats --help-report hgtx_ppu_proj_sum for more details. Supported options include:

• device: Specifies a list of PPU device IDs to include in the statistics. Separate multiple PPU device IDs with a /. By default, the report includes statistics for all PPU devices.

The hgtx_ppu_proj_sum table contains the following columns:

Row# : Row number of the HGTX PPU projection summary
Range : Name of the HGTX range
Style : Range style; Start/End or Push/Pop
Total Proj Time [ns] : Total projected time used by all instances of this range name
Total Range Time [ns] : Total original HGTX range time used by all instances of this range name
Range Instances : Number of instances of this range
Proj Avg [ns] : Average projected time for this range
Proj Med [ns] : Median projected time for this range
Proj Min [ns] : Minimum projected time for this range
Proj Max [ns] : Maximum projected time for this range
Proj StdDev [ns] : Standard deviation of projected times for this range
Total PPU Ops : Total number of PPU operations
Avg PPU Ops : Average number of PPU operations
Avg Range Lvl : Average range stack depth
Avg Num Child : Average number of children ranges

GUI guide

• Settings

  • HGTX/NVTX range and/or domain to filter by: Sets the statistics range. The format for specifying an HGTX/NVTX range is: range[@domain][/index]. For more information, see Set the statistics range.

  • PPU devices: Filters the results by the selected PPU devices. The default value is "All", which includes data from all PPU devices.

Statistics logic

An HGGC kernel is associated with an HGTX range if the HGGC API call that launched the kernel occurred on the same thread and within the HGTX range's duration.

The projection of an HGTX range from the CPU timeline to the PPU timeline is based on its active time on the PPU. This active time spans from the start of the earliest associated PPU activity to the end of the latest. PPU activities include HGGC kernels, memcpy, and memset.

Aggregation method:

• Data for HGTX ranges with the same name and same style is aggregated, including CPU-side duration and PPU-side projected duration.

The report sorts results first in descending order by Total Proj Time, and then in ascending order by HGTX range name.

2.3.12 HGTX PPU projection trace

Description

This report projects HGTX ranges from the CPU timeline to the PPU timeline, showing the active time of each range on the PPU. It also includes HGTX stack information.

Required asys collection options: --trace hgtx,hggc

Command-line usage:

asys stats -r hgtx_ppu_proj_trace report.asysrep

For detailed help, run asys stats --help-report hgtx_ppu_proj_trace.

Example report output:

Row#,Name,Projected Start (ns),Projected Duration (ns),Orig Start (ns),Orig Duration (ns),Style,PID,TID,NumPPUOps,Lvl,NumChild,RangeId,ParentId,RangeStack,
1,"profile",193282492,75939861,190531194,79865797,"PushPop",3588556,3588556,10,0,2,6,"-",":6",
2,"Loop1",193282492,43122023,190588819,74067469,"PushPop",3588556,3588556,5,1,1,7,6,":6:7",
3,"DoProcess",193282492,43122023,190594879,72971893,"PushPop",3588556,3588556,5,2,0,8,7,":6:7:8",
...

The hgtx_ppu_proj_trace table contains the following columns:

Row# : Row number of the HGTX PPU projection trace
Name : Name of the HGTX range
Projected Start [ns] : Projected range start timestamp
Projected Duration [ns] : Projected range duration
Orig Start [ns] : Original HGTX range start timestamp
Orig Duration [ns] : Original HGTX range duration
Style : Range style; Start/End or Push/Pop
PID : Process identifier
TID : Thread identifier
NumPPUOps : Number of enclosed PPU operations
Lvl : Stack level, starts at 0
NumChild : Number of child ranges
RangeId : Arbitrary ID for range
ParentId : Range ID of the enclosing range
RangeStack : Range IDs that make up the push/pop stack

Calculation Logic

The system associates an HGGC kernel with an HGTX range if the corresponding HGGC API call occurs on the same thread during that range.

The projection of a CPU-side HGTX range onto the PPU timeline defines its active time on the PPU. This active time is measured from the start of the earliest associated PPU operation to the end of the latest one. PPU operations include HGGC kernel executions, memcpy, and memset.

The report exports one row for each HGTX range and sorts the output in ascending order by 'Projected Start'.

2.3.13 HGTX range summary

Description

Summarizes the execution time of HGTX ranges from an asysrep report and sorts the results by total range time in descending order.

Required asys collection option:--trace hgtx

Command-line usage:

asys stats -r hgtx_sum report.asysrep

Run asys stats --help-report hgtx_sum for detailed help. Supported options include:

• rows=<limit>: Limits the number of HGTX ranges in the output.

• process=<pid_list>: Specifies the PIDs of processes to analyze, separated by /; if omitted, all processes are analyzed.

• thread=<tid_list>: Specifies the TIDs of threads to analyze, separated by /; if omitted, all threads are analyzed.

Example report output:

hgtx_sum report:

Row#,Time (%),Total Time (ns),Instances,Avg (ns),Med (ns),Min (ns),Max (ns),StdDev (ns),Style,Range,
1,25.7,181837947,1,181837947,181837947,181837947,181837947,0,"PushPop","[prof_range]: iter 6",
2,25.1,177622641,1,177622641,177622641,177622641,177622641,0,"PushPop","[prof_range]: iter 9",
3,25.1,177254919,1,177254919,177254919,177254919,177254919,0,"PushPop","[prof_range]: iter 7",
4,24.1,170843422,1,170843422,170843422,170843422,170843422,0,"PushPop","[prof_range]: iter 8",
...

The hgtx_sum report includes the following columns:

Note: The 'Time' column shows the percentage of the 'Total Time' for a range relative to the sum of total times for all ranges in the report. It is not a percentage of the application's total wall time or CPU time.

Row# : Row number of the HGTX range summary
Time [%] : Percentage of 'Total Time'
Total Time [ns] : Total time used by all instances of this range
Instances : Number of instances of this range
Avg [ns] : Average execution time of this range
Med [ns] : Median execution time of this range
Min [ns] : Minimum execution time of this range
Max [ns] : Maximum execution time of this range
StdDev [ns] : Standard deviation of the time of this range
Style : Range style; Start/End or Push/Pop
Range : Name of the range

GUI Guide

• Settings

  • Processes: Specify the processes to include. Default: All.

  • Maximum number of results: Set the maximum number of rows to display. Default: -1 (no limit).

  • HGTX/NVTX range and/or domain to filter by: Filter statistics by a specific range or domain. The format for specifying HGTX/NVTX is: range[@domain][/index]. For details, see Setting the statistics range.

Statistics Logic

This report aggregates statistics by HGTX range domain and name, summing the execution time for ranges with the same identity and displaying the name as domain:range when a domain is present, sorting the results in descending order by total time. When filtering, a thread is included if it matches the thread filter or if its parent process matches the process filter.

2.3.14 OSRT API

Description

This report summarizes the execution time of os runtime api calls and sorts the results by total time in descending order.

Required asys collection option: --trace osrt

Command line usage:

asys stats -r osrt_sum report.asysrep

Run asys stats --help-report osrt_sum for detailed help.

Example report output:

Row#,Time (%),Total Time (ns),Num Calls,Avg (ns),Med (ns),Min (ns),Max (ns),StdDev (ns),Name,
1,85.6,8901500720,255,34907845,4994783,1095,348667026,63811239,"pthread_cond_wait",
2,6.9,721563692,1105,652998,1055069,52083,1072141,485233,"nanosleep",
3,6.7,700892565,7,100127509,100123388,100121020,100143853,8411,"poll",
...

The following describes the columns in the osrt_sum table:

Note: The 'Time' column shows a function's 'Total Time' as a percentage of the total time for all functions listed in the report. It is not a percentage of the total application wall time or CPU execution time.

Row# : Row number of the OS runtime summary
Time [%] : Percentage of 'Total Time'
Total Time [ns] : Total time used by all executions of this function
Num Calls : Number of calls to this function
Avg [ns] : Average execution time of this function
Med [ns] : Median execution time of this function
Min [ns] : Minimum execution time of this function
Max [ns] : Maximum execution time of this function
StdDev [ns] : Standard deviation of the execution time of this function
Name : Name of the function

GUI guide

• Settings

  • HGTX/NVTX range and/or domain to filter by: Specifies the range for statistics. The format for specifying an HGTX/NVTX range is: range[@domain][/index]. For details, see Set the range.

Aggregation rule

The system groups calls by OSRT API name, sums their execution times, and sorts the results by total time in descending order.

2.3.15 PCCL stage summary

Description

Summarizes the duration of each PCCL transfer stage in an asysrep report. The stages are sorted in descending order by total time.

Required asys collection option: --trace pccl

Command-line usage:

asys stats --report pccl_stage_sum report.asysrep

Run asys stats --help-report pccl_stage_sum for detailed help.

Example report output:

Row#,PID,Device ID,TID,Channel ID,Channel Type,Name,Total Time,Instances,Avg,Med,Min,Max,Stdev,
1,631660,0,631712,1,RX,RecvWait,305528068,24,12730336,13761981,9669723,18509164,2407844,
2,631660,1,631711,1,TX,GPUWait,299988851,24,12499535,12786840,9625474,16580086,2217489,
3,631660,0,631712,1,TX,GPUWait,297674072,24,12403086,12579239,9570100,17151019,2208876,
...

The pccl_stage_sum report contains the following columns:

Row# : Row number
PID : Process identifier
Device ID : PPU device identifier
TID : Thread identifier
Channel ID : PCCL channel identifier
Channel Type : PCCL channel type
Name : Stage name
Total Time [ns] : Total stage duration
Instances : Number of occurrences
Avg [ns] : Average stage duration
Med [ns] : Median stage duration
Min [ns] : Minimum stage duration
Max [ns] : Maximum stage duration
Stdev [ns] : Standard deviation of the stage duration

GUI guide

• Settings

  • HGTX/NVTX range and/or domain to filter by: Sets the statistics range. The format is: range[@domain][/index]. For details, see Set the statistics range.

Aggregation logic

For each thread, channel, direction, and transfer stage, the system aggregates metrics such as duration (average, maximum, and minimum) and the number of instances. The results are sorted in descending order by total stage time.

2.3.16 HGGC PPU memory size

Description

Summarizes the data size for HGGC PPU memory operations, such as memcpy and memset, from an asysrep report. The results are sorted by total data size in descending order.

Required asys collection option: --trace hggc

Command-line usage:

asys stats --report hggc_ppu_mem_size_sum report.asysrep

Run asys stats --help-report hggc_ppu_mem_size_sum for details.

Example report output:

Row#,Total (bytes),Count,Avg (bytes),Med (bytes),Min (bytes),Max (bytes),StdDev (bytes),Operation,
1,9896352144,564,17546723,1024,12,556254464,74087070,"[HGGC memset]",
2,536876208,342,1569813,16,4,67108864,10157989,"[HGGC memcpy Host-to-Device]",
3,156,44,3,1,1,16,4,"[HGGC memcpy Device-to-Host]",
4,64,12,5,4,8,1,"[HGGC memcpy Device-to-Device]",

The hggc_ppu_mem_size_sum table contains the following columns:

Row#: Row number.
Total (bytes): Total memory used by the operation.
Count: Number of executions.
Avg (bytes): Average memory size.
Med (bytes): Median memory size.
Min (bytes): Minimum memory size.
Max (bytes): Maximum memory size.
StdDev (bytes): Standard deviation of the memory size.
Operation: Name of the memory operation.

GUI guide

• Settings

  • HGTX/NVTX range and/or domain to filter by: Sets the statistics range. The format for specifying an HGTX/NVTX range is: range[@domain][/index]. For more information, see Set the statistics range.

Statistics rule

The report aggregates data by memory operation type and sorts the results by total data size in descending order.

• The report categorizes memcpy operations by copy direction, such as [HGGC memcpy Host-to-Device] and [HGGC memcpy Device-to-Host].

HGGC PPU memory time summary

This report summarizes the execution time of HGGC PPU memory operations, such as memcpy and memset, from an asysrep report, sorting the results by total memory operation time in descending order.

Required asys collection option: --trace hggc

Command line usage:

asys stats --report hggc_ppu_mem_time_sum report.asysrep

Run asys stats --help-report hggc_ppu_mem_time_sum for detailed help.

Example report output:

Row#,Time (%),Total Time (ns),Count,Avg (ns),Med (ns),Min (ns),Max (ns),StdDev (ns),Operation,
1,69.3,19697314,342,57594,1240,1080,2413129,364424,"[HGGC memcpy Host-to-Device]",
2,30.6,8687991,564,15404,720,120,496801,67433,"[HGGC memset]",
3,0.1,20920,44,475,400,320,960,165,"[HGGC memcpy Device-to-Host]",
4,0.1,15600,12,1300,1240,920,1800,293,"[HGGC memcpy Device-to-Device]",

The columns in the hggc_ppu_mem_time_sum table are as follows:

Note: The "Time (%)" column is calculated based on the sum of the "Total Time (ns)" column. It represents the percentage of time for a specific memory operation type relative to the total time of all listed operations, not as a percentage of application wall time or CPU execution time.

Row# : Row number of the memory summary
Time [%] : Percentage of 'Total Time'
Total Time [ns] : Total execution time for this operation
Count : Number of executions
Avg [ns] : Average execution time of this operation
Med [ns] : Median execution time of this operation
Min [ns] : Minimum execution time of this operation
Max [ns] : Maximum execution time of this operation
StdDev [ns] : Standard deviation of the execution time for this operation
Operation : Name of the memory operation

GUI

• Settings

  • HGTX/NVTX range and/or domain to filter by: Sets the statistics range. The format for specifying an HGTX/NVTX range is: range[@domain][/index]. For details, see Set Statistics Range.

Aggregation method

Aggregates the execution time for each memory operation type and sorts the results by total duration in descending order.

• memcpy operations are categorized by copy direction, such as [HGGC memcpy Host-to-Device] and [HGGC memcpy Device-to-Host].