Acu CLI

The Asight Compute CLI (acu) is a command-line tool for analyzing the performance of target applications. It generates a report file that can be transferred to another system for parsing and visualization using a GUI tool.

Key features of acu:

• Collects performance data as an application runs on a PPU and aggregates the data into various metrics, such as:

  • Metrics related to theoretical peak performance

  • Compute workload

  • Memory workload

  • Warp scheduler statistics

  • Warp state and instruction stall reason statistics

  • Instruction statistics

  • ICN link metrics

• Supports specifying the set of metrics to collect at three levels of granularity: set, section, and metric.

• If the number of metrics to collect exceeds the PPU's capacity, acu automatically replays the application to gather all performance data.

  • Application Replay mode

  • Kernel Replay mode

  • Range Replay mode

  • Application Range Replay mode

• Lets you define the data collection scope and provides multiple kernel filtering methods.

  • Specify the PPU devices

  • Specify the Context IDs and Stream IDs

  • Specify the kernel names

  • Use cudaProfilerStart/Stop() and hggcProfilerStart/Stop() to specify the collection scope

  • Specify the number of kernels to collect

  • Specify the number of kernels to skip

• Supports Warp Sampling.

1. Collect performance data

1.1 Supported metrics

A metric set defines what performance data to collect. You can choose from three levels of granularity:

• metric: A measure of a specific aspect of PPU performance. It is the most granular unit for defining the data collection scope in acu.

• section: A collection of logically related metrics. acu includes several built-in sections, and you can define custom ones by creating or modifying a section file.

• set: A collection of one or more sections. It is the highest-level grouping for defining the data collection scope in acu.

1.1.1 Querying metric sets

Use the --list-sets command-line option to view details about the metrics sets that acu supports. The following table describes the output:

root@d29fb2dd227a:/# acu --list-sets
 ------------ ----------------------------------------------------------------------------- --------- ------------------- 
  Identifier   Sections                                                                      Enabled   Estimated Metrics  
 ------------ ----------------------------------------------------------------------------- --------- ------------------- 
  default      Occupancy,MemoryWorkloadAnalysis,SpeedOfLight,SpeedOfLight_TensorRooflineC-   yes       314                
               hart,ComputeWorkloadAnalysis,SpeedOfLight_RooflineChart,LaunchStats               
                                                                                                                            
  detailed     MemoryWorkloadAnalysis_Tables,SchedulerStats,Occupancy,SourceCounters,Memo-   no        520                
               ryWorkloadAnalysis,SpeedOfLight,WarpStateStats,SpeedOfLight_TensorRoofline-    
               Chart,InstructionStats,ComputeWorkloadAnalysis,SpeedOfLight_RooflineChart,-                                
               LaunchStats  
               
  full         Icnlink_Tables,MemoryWorkloadAnalysis_Tables,InternalDebug,SchedulerStats,-   no        821                
               Occupancy,Icnlink_Topology,SourceCounters,MemoryWorkloadAnalysis,Professio- 
               nal,SpeedOfLight,WarpStateStats,SpeedOfLight_TensorRooflineChart,MemoryWor-                                
               kloadAnalysis_Chart,InstructionStats,ComputeWorkloadAnalysis,SpeedOfLight_-                                
               RooflineChart,LaunchStats,Icnlink 
                                                                                                                                         
  systems      Systems                                                                       no        0   

• Identifier: The name of the metrics set. Use this name with the --set option to specify which metrics set to collect.

• Sections: The list of sections included in the set. Review this list to determine if the metrics set meets your collection requirements.

• Enabled: Indicates if the metrics set is enabled. By default, only the default set is enabled.

• Estimated Metrics: The number of metrics in the set. Use this number to estimate the relative overhead of the collection.

Note: Set details may vary between versions. Always refer to the query result as the source of truth.

1.1.2 Querying metric sections

Use the --list-sections command-line option to list the supported sections, their GUI display names, and the preset path for each section file. By default, the command displays information for the sections in the default set. The following table describes the output:

root@d29fb2dd227a:/# acu --list-sections
 ---------------------------------- ------------------------------------------------- --------- --------------------------------------------------
  Identifier                         Display Name                                      Enabled   FileName
 ---------------------------------- ------------------------------------------------- --------- --------------------------------------------------
  InstructionStats                   instruction statistics                            no        /usr/sections/InstructionStatistics.section
  MemoryWorkloadAnalysis             memory workload analysis                          yes       /usr/sections/MemoryWorkloadAnalysis.section
  MemoryWorkloadAnalysis_Chart       memory workload analysis chart                    no        /usr/sections/MemoryWorkloadAnalysis_Chart.sect-
                                                                                                 ion
  MemoryWorkloadAnalysis_Tables      memory workload analysis tables                   no        /usr/sections/MemoryWorkloadAnalysis_Tables.sec-
                                                                                                 tion
  SchedulerStats                     scheduler statistics                              no        /usr/sections/SchedulerStatistics.section
  SpeedOfLight_RooflineChart         PPU speed of light roofline chart                 yes       /usr/sections/SpeedOfLight_RooflineChart.section
  SpeedOfLight_TensorRooflineChart   PPU speed of light roofline chart (Tensor Core)   yes       /usr/sections/SpeedOfLight_TensorRooflineChart.-
                                                                                                 section
  ComputeWorkloadAnalysis            PPU compute workload analysis                     yes       /usr/sections/ComputeWorkloadAnalysis.section
  Icnlink                            Icnlink                                           no        /usr/sections/Icnlink.section
  InternalDebug                      PPU all metrics debugging                         no        /usr/sections/InternalDebug.section
  SpeedOfLight                       PPU speed of light throughput                     yes       /usr/sections/SpeedOfLight.section
  WarpStateStats                     warp state statistics                             no        /usr/sections/WarpStateStats.section

• Identifier: The identifier for the metrics section, used with the --section option to specify the collection scope.

• Display name: The name displayed for this section in Asight Compute.

• Enabled: Specifies whether this section is enabled.

• File name: The path to the section file.

Note: Section details may differ between versions. The query results are the source of truth.

You can also use the --set option to view section information for a specific set. A value of yes in the Enabled column indicates that the section is available in the specified set:

root@eb8b441561f7:~# acu --set=full --list-sections
 ---------------------------------- ------------------------------------------------- --------- --------------------------------------------------
  Identifier                         Display Name                                      Enabled   FileName
 ---------------------------------- ------------------------------------------------- --------- --------------------------------------------------
  InstructionStats                   instruction statistics                            yes       /usr/sections/InstructionStatistics.section
  MemoryWorkloadAnalysis_Chart       memory workload analysis chart                    yes       /usr/sections/MemoryWorkloadAnalysis_Chart.sect-
                                                                                                 ion
  MemoryWorkloadAnalysis_Tables      memory workload analysis tables                   yes       /usr/sections/MemoryWorkloadAnalysis_Tables.sec-
                                                                                                 tion
  ComputeWorkloadAnalysis            PPU compute workload analysis                     yes       /usr/sections/ComputeWorkloadAnalysis.section
  Icnlink                            ICNLink                                           no        /usr/sections/Icnlink.section
  InternalDebug                      debug PPU all metrics                             yes       /usr/sections/InternalDebug.section
  MemoryWorkloadAnalysis             memory workload analysis                          yes       /usr/sections/MemoryWorkloadAnalysis.section
  SchedulerStats                     scheduler statistics                              yes       /usr/sections/SchedulerStatistics.section
  SpeedOfLight                       PPU speed of light throughput                     yes       /usr/sections/SpeedOfLight.section
  SpeedOfLight_RooflineChart         PPU speed of light Roofline chart                 yes       /usr/sections/SpeedOfLight_RooflineChart.section
  SpeedOfLight_TensorRooflineChart   PPU speed of light Roofline chart (Tensor Core)   yes       /usr/sections/SpeedOfLight_TensorRooflineChart.-
                                                                                                 section
  WarpStateStats                     warp state statistics                             yes       /usr/sections/WarpStateStatistics.section

Note: The sections in a set can vary between versions; always use the actual query result as the definitive source.

Specifying the section file search path

acu lets you specify a search path for section files. When a search path is specified, acu returns all files in that path with the .section suffix. You can specify the search path in two ways:

• --section-folderSets the search path for section files. acu searches this folder for files with the .section suffix but does not search its subfolders.

• --section-folder-recursiveSets the search path for section files. acu searches this folder and all its subfolders for files with the .section suffix.

For example, view the section information for /usr/custom:

acu --list-sections --section-folder=/usr/custom/

1.1.3 Query metrics

Use the --list-metrics command-line option to list supported metrics. By default, it lists the metrics in the default set. You can also use --section or --set to list metrics from a specific section or set.

List metrics in a section

Use --section to query the metrics in a specified section.

root@d29fb2dd227a:/# acu --section SpeedOfLight --list-metrics
breakdown:cu__throughput.avg.pct_of_peak_sustained_elapsed
breakdown:ppu__compute_memory_throughput.avg.pct_of_peak_sustained_elapsed
ce__cycles_active.max
ce__cycles_elapsed.avg.per_second
ce__cycles_elapsed.max
cu__cycles_active.avg
cu__throughput.avg.pct_of_peak_sustained_elapsed
dram__cycles_elapsed.avg.per_second
l1__throughput.avg.pct_of_peak_sustained_active
l2__throughput.avg.pct_of_peak_sustained_elapsed
llc__throughput.avg.pct_of_peak_sustained_elapsed
ppu__compute_memory_throughput.avg.pct_of_peak_sustained_elapsed
ppu__dram_throughput.avg.pct_of_peak_sustained_elapsed
ppu__time_duration.sum

Querying metrics in a set

Use --set to query the metrics in a specified set.

1.1.4 Metric descriptions

Run acu --query-metrics to view metric descriptions. For example:

root@0549cf16bb85:~# acu --query-metrics
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Metric name                                                                                    Type        Unit           Metric description
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ce__cycles_active.avg                                                                          counter     cycle          Number of active cycles on CEs
 ce__cycles_active.max                                                                          counter     cycle          Number of active cycles on CEs
 ... 
 dram__bytes_read.sum                                                                           counter     byte           Number of bytes read from DRAM
 dram__bytes_read.sum.pct_of_peak_sustained_elapsed                                             counter     %              Number of bytes read from DRAM
 dram__bytes_read.sum.per_second                                                                counter     byte/second    Number of bytes read from DRAM
 ...
 l2__transaction_hit_rate                                                                       ratio       unitless       Hit rate for L2-cacheable requests, as a ratio.
 l2__transaction_hit_rate.pct                                                                   ratio       %              Hit rate for L2-cacheable requests, as a percentage.
 ...

• Metric name: The name of the metric.

• Type: The type of the metric. Three types are available:

  • Counter: A raw counter from the PPU.

  • Ratio: The ratio of two counters.

  • Throughput: The highest percentage of the PPU peak value reached by any counter in the group.

• Unit: The unit of the metric.

• Metric description: A description of the metric.

Tip: In the Asight Compute GUI, hover over a performance metric or open the Metric Details page to view its description.

1.2 Specify the metric set

Specify the scope of metric collection by choosing a granularity: set, section, or metric.

The combined effects of the three granularity levels are as follows:

• If none of the --set, --section, or --metrics options are specified, the default set of metrics is used.

• The --section and --metrics options override the --set option.

• When you use the --section and --metrics options together, the effective metric set is the union of both.

Tips: The finer the granularity, the more metrics are collected. If the number of metrics exceeds the PPU limit, the application may be replayed multiple times to collect all the data, increasing the acu profiling time.

1.2.1 Collection set

Use the --set option to enable a metric set. The basic rules are:

• You can enable only one set at a time.

• If you do not specify --set, --section, or --metrics, the system enables the default set.

To enable the detailed set, run the following command:

acu -o test_report -f --set="detailed" ./cuda_test 10

The default set does not include all metrics, such as those for memory throughput. To collect all metrics, enable the full set with the following command:

acu -o test_report -f --set="full" ./cuda_test 10

1.2.2 Specify section

Use the --section option to specify which metric section to enable. The basic rules are as follows:

• Each --section option specifies a section matching rule.

• There are two types of section matching rules:

  • Exact match: Specify a section identifier. Only the section with that identifier is enabled. You can view section identifiers with the --list-sections option.

  • Regular expression match: Specify a regular expression with the syntax regex:<expression>. All sections with an identifier that matches the <expression> are enabled.

• If you specify multiple --section matching rules, only the last specified rule takes effect.

For example, the regular expression "regex:.*WorkloadAnalysis" enables sections ending with WorkloadAnalysis, such as ComputeWorkloadAnalysis and MemoryWorkloadAnalysis.

acu --section="regex:.*WorkloadAnalysis" python test_linear.py

For example, if you specify SpeedOfLight followed by SchedulerStats as exact match rules, the system collects only sectionSchedulerStats.

acu --section SpeedOfLight --section SchedulerStats python test_linear.py

Tip: To prevent the Linux shell from interpreting the regular expression, wrap the parameter in double quotes when specifying a section with a regular expression, for example:--section="regex:.*Analysis.*"

1.2.3 Specify metrics

Use the --metrics option to specify which metrics to enable. The basic rules are as follows:

• Separate multiple metric names with a comma. Use --list-metrics to view the list of supported metric names.

• If you specify the --metrics option multiple times, only the last specification takes effect.

• If --section is also specified, both the metrics included in that section and those specified by --metrics are collected.

• The --metrics option supports two name matching methods:

  • Exact match, which includes the following three cases:

    • Specify the full metric name.

    • Specify a metric group by its full name. The syntax is group:<group name>. This collects all metrics within the specified metric group.

    • Specify a metric for breakdown. The syntax is breakdown:<metric name>. This collects and calculates all sub-metrics of the specified metric.

  • Regular expression match: The syntax is regex:<expression>. This collects all metrics whose names match the regular expression <expression>.

Tip:

1. Use the --list-metrics command-line option to query metric groups and breakdowns.

2. Wrap the parameter in double quotes to prevent the Linux shell from interpreting the regular expression. For example: --metrics="regex:dram.*".

For example, to collect the ce__cycles_active.max and dram__bytes.sum.per_second metrics:

acu --metrics=ce__cycles_active.max,dram__bytes.sum.per_second python test_linear.py

For example, to collect all metrics from the memory__chart group, use the following command:

acu --metrics="group:memory__chart" python test_linear.py

For example, the following command collects and calculates the values of all sub-metrics of cu__throughput.avg.pct_of_peak_sustained_elapsed and ppu__compute_memory_throughput.avg.pct_of_peak_sustained_elapsed without calculating the values of the metrics themselves:

acu --metrics=breakdown:cu__throughput.avg.pct_of_peak_sustained_elapsed,breakdown:ppu__compute_memory_throughput.avg.pct_of_peak_sustained_elapsed python test_linear.py

For example, to collect all metrics containing cycles_elapsed using a regular expression, such as ce__cycles_elapsed.avg.per_second and ce__cycles_elapsed.max, run the following command:

acu --metrics="regex:cycles_elapsed.*" python test_linear.py

For example, to collect all metrics that start with dram using a regular expression, such as dram__bytes.sum.peak_sustained and dram__bytes.sum.per_second, but not ppu__dram_throughput.avg.pct_of_peak_sustained_elapsed, run the following command:

acu --metrics="regex:^dram.*$" python test_linear.py

1.3 Specify replay mode

Hardware limits the number of performance counters a PPU can collect simultaneously. If collecting all metrics requires more counters than this limit, acu replays the kernel in multiple passes. Each pass uses a different counter configuration to collect all metric data.

acu supports multiple replay modes. By default, it collects data in kernel replay mode. You can specify the replay mode using the --replay-mode option:

Since each replay mode affects an application differently—by altering the kernel execution order or introducing performance overhead—choose the mode best suited to your implementation.

Example: Use kernel replay mode to collect performance data.

acu --replay-mode kernel python test_linear.py

1.3.1 Kernel replay mode

In kernel replay mode, each kernel runs multiple times per application run to collect all metrics. Before the first pass, the system creates a backup of the PPU device memory. The system then restores this backup before each subsequent pass.

The key features of Kernel Replay mode include:

• The replay completes in a single application execution.

• Performance data is collected at kernel granularity.

• The application's kernels are forced to execute serially.

• Backing up and restoring memory before each kernel execution increases the overall application execution time.

• Memory backup consumes device memory.

Because Kernel Replay forces kernel serialization during profiling, it is unsuitable for certain programs. The following describes its suitable and unsuitable scenarios.

1.3.2 Application Replay

The application replay mode collects all metrics by running the application repeatedly. Because of this repeated execution, this mode does not require memory backup and restore for each kernel. However, the application must behave consistently across all runs; for example, the kernel names, count, and execution order must remain the same.

The key features of application replay mode are:

• The application is run multiple times to collect all metric data.

• Performance data is collected at kernel granularity.

• Kernels in the application are forced to execute serially.

• If the application performs time-consuming host-side operations (such as initialization), multiple replays increase execution time.

Scenarios

Like Kernel Replay, Application Replay enforces kernel serialization during profiling, so it shares the same guidelines for applicable and inapplicable scenarios. For details, see the Kernel Replay documentation: applicable scenarios and inapplicable scenarios.

Because Application Replay distributes a kernel's metric data across multiple application runs, matching these kernels can be challenging. To address this, acu provides the --app-replay-mode and --app-replay-match command-line options.

Kernel matching mode

The acu --app-replay-mode option specifies whether to strictly check for consistent application behavior over multiple runs during kernel matching in Application Replay. Two modes are available:

Tip: If an application fails to generate a report in strict mode, its behavior is likely non-deterministic. Try relaxed mode, but the final report may lack performance data for some kernels.

Kernel matching policy

acu provides the --app-replay-match option to specify the kernel matching strategy used in Application Replay, which defines how kernel instances are matched. The following three matching strategies are available:

Example: Collecting performance data using application replay mode, and performing matching in loose mode with the grid matching strategy:

acu --replay-mode application --app-replay-mode relaxed --app-replay-match grid python test_linear.py

1.3.3 Range Replay mode

Within a single application execution, Range Replay mode collects all metric data by replaying a range (including kernel launches and HGGC API calls) multiple times. The resulting metric data corresponds to the entire range rather than a specific kernel. Range Replay captures the kernel launches and HGGC API calls within the range and performs the necessary save and restore operations for host and device memory.

Range Replay mode offers the following key features:

• Replay completes in a single application execution.

• Performance data is collected at range granularity, not kernel granularity.

• Multiple kernels within a range can execute concurrently without forced serialization.

• Range replay involves memory backup and restore operations, which increase application execution time.

• Only a subset of HGGC APIs are permitted within a range.

• Range replay replays only PPU device-side behavior, not host-side application behavior.

• The application uses hg(gc)ProfilerStart and hg(gc)ProfilerStop to define a range, and ensures that the acu can synchronize all contexts at the start of the range.

• A single range supports performance data collection from only one PPU device.

Range definition

To use range replay, you need to specify a range for performance analysis in your application. A range consists of all HGGC API calls and kernels launched from any CPU thread between a start marker and an end marker. Your application must insert appropriate synchronization between threads to capture the expected API calls. You can set the range using one of the following two options:

Profiler start/stop API

Use hg(gc)ProfilerStart to set the start marker and hg(gc)ProfilerStop to set the end marker, which is the default method for defining an Asight Compute range.

hggcProfilerStart();
/* Code to profile, including HGGC API and kernels. */
hggcProfilerStop();

HGX range

Use HGTX include expressions to define ranges. A range starts with the first HGGC API call and ends with the last API call matching an expression. If multiple expressions are specified, a match on any of them defines a range. This enables you to use multiple expressions to easily capture and analyze multiple ranges from a single application execution. The application must use the HGTX API to mark ranges, enabling expression matching.

hgtxRangeId_t r1 = hgtxRangeStartA("Range 1");
/* Insert code to be profiled, including calls to the HGGC API and kernels. */
hgtxRangeEnd(r1);

Scenarios

Although Range Replay also works in the three use cases for Kernel Replay, it is primarily designed to profile concurrently executing kernels (specifically, the first two cases listed under inapplicable scenarios for Kernel Replay).

1. Single process, single thread (dependent kernels are within the same range)

   

   Example: pccl-tests (1 process, 1 thread, 2 ranks).

   acu --set full --replay-mode range -o all_reduce_perf_range_p1_t1_g2 ./build/all_reduce_perf -g 2 -n 1 -w 0 -b 2M -e 2M -c 0

2. Single process, multiple threads (dependent kernels are within the same range)

   

   Example: pccl-tests (1 process, 2 threads, 1 rank).

   acu --set full --replay-mode range -f -o all_reduce_2_ranks_2M ./build/all_reduce_perf -g 1 -t 2 -n 1 -w 0 -b 2M -e 2M -c 0

Unsupported Scenarios

Since the range in Range Replay is process-level, ensure it includes all interdependent kernels that run on different threads. Otherwise, the application may hang.

Range Replay serializes ranges, which can cause an application to hang if a range depends on a kernel or another range in a separate process.

1. Single process, multiple threads (where interdependent kernels exist in separate ranges).

   

2. Multiple processes (a range in one process depends on a kernel in another process).

   

   Example: pccl-tests (2 processes, 1 thread per process, 1 rank per thread; only one rank defines the range).

   acu --set full --replay-mode range --target-processes all -o all_reduce_perf_range_p2_t1_g1 mpirun -np 2 --allow-run-as-root ./build/all_reduce_perf -g 1 -t 1 -n 1 -w 0 -b 2M -e 2M -c 0 -j 0

3. Multiple processes (a range in one process depends on a range in another process).

   

   Example: pccl-tests (2 processes, 1 thread per process, 1 rank per thread; both ranks define ranges).

   acu --set full --replay-mode range --target-processes all -o all_reduce_perf_range_p2_t1_g1 mpirun -np 2 --allow-run-as-root ./build/all_reduce_perf -g 1 -t 1 -n 1 -w 0 -b 2M -e 2M -c 0

1.3.4 Application scope replay

In Application Range Replay mode, all requested metrics are grouped into one or more passes. Like Range Replay, metrics apply to the entire selected range, not to individual kernels. The tool does not serialize workloads, such as kernels and graphs, allowing it to analyze the performance of workloads that require concurrent execution.

Unlike range replay, a range does not require explicit capture for repeated execution. Instead, the entire application runs repeatedly, and each run is a pass in which metrics are collected for every range. This approach eliminates the need to track and capture application state, such as saving and restoring memory, because the correct execution of the range is handled by the application itself.

The definition of range is the same as in Range Replay, and the range workload must execute deterministically.

Use cases

Application Range Replay handles scenario 2, which Range Replay cannot, provided that only one process defines the range.

Example: pccl-tests (2 processes, 1 thread per process, 1 rank per thread, with a range defined on a single rank).

acu --section Icnlink --replay-mode app-range --target-processes all mpirun -np 2 --allow-run-as-root ./build/all_gather_perf -g 1 -t 1 -n 1 -w 0 -b 512M -e 512M -c 0 -q 1 -s 1 -j 0

Limitations

A multi-process application will hang if there are dependencies between kernels in the ranges defined by its processes.

Additionally, if the execution order of these ranges is inconsistent across replays, performance data from the replays may not be correctly matched by range, preventing report generation.

1. multi-process with two ranges, where one range depends on the other.

   

2. multi-process with two independent ranges, where the execution order is not guaranteed.

   

3. multi-thread with two independent ranges, where the execution order is not guaranteed.

   

1.3.5 Graph profiling

acu profiles both entire graphs and individual kernel nodes. Use the --graph-profiling option to specify the mode.

Graph mode

In graph mode, Graph Profiling treats an HGGC graph as a single workload entity for performance analysis.

Use the --graph-profiling graph option to enable this feature.

The main use cases for this mode include:

• The graph contains kernel nodes that must execute concurrently.

• You need to analyze the performance of a graph with multiple kernel nodes more accurately.

Node mode

In node mode, Graph Profiling treats a single kernel node from an HGGC graph as the workload entity for performance analysis.

Use the --graph-profiling node option to enable this feature.

Use cases for this mode include:

• The performance of each kernel node in the graph needs to be analyzed accurately.

• The kernel nodes in the graph have no dependencies.

When you enable graph profiling, note the following:

• This feature requires --replay-mode kernel. The default is graph mode.

• If you are not concerned with graph performance, you can disable the graph profiling feature with --graph-profiling none.

• Certain performance metrics, such as instruction statistics, are unavailable in graph mode.

2. Control the profiling process

2.1 Specify profiling devices

Use the --devices option to specify the PPU devices to profile.

• Specify devices by their index, which starts from 0. An index is assigned when a PPU device powers on.

• Use a comma (,) to separate multiple PPU device IDs.

• If this option is not specified, all PPU devices are profiled by default.

Example: To profile kernels on PPU devices 1 and 2:

 acu --devices 1,2 python test_linear.py

Note: In Range Replay mode, the --devices option supports specifying only one PPU device.

2.2 Specify the profiling time range

Use the --profile-from-start option to control whether profiling starts at application launch. This option applies to Application Replay and Kernel Replay modes. The supported values are described below.

acu supports the following profiler start/stop API pairs:

• cudaProfilerStart/Stop

• hggcProfilerStart/Stop

• cuProfilerStart/Stop

• hgProfilerStart/Stop

Example: To start and stop profiling based on API calls, run the following command:

acu --profile-from-start off python test_linear.py

If the profiling ranges in the application are defined using cu(da)ProfilerStart() and cu(da)ProfilerStop() as shown in the following figure, acu profiles the first range (three kernels) and the second range (one kernel).

disable-profiler-start-stop

Use the --disable-profiler-start-stop option to ignore the profiler start/stop APIs.

• If you specify the --disable-profiler-start-stop option, the --profile-from-start option is ignored, and profiling starts when the application launches.

2.3 Filter kernels

acu provides several methods to filter which kernels are profiled.

2.3.1 Specify the kernel name base

Use the --kernel-name-base option to specify the name matching criteria for the --kernel-name and --kernel-id options. Supported values include function, mangled, and demangled. The default value is function.

• function: The function name without arguments or templates, for example, foo.

• demangled: The demangled function name, including arguments and templates, for example, foo(float*,int,int).

• mangled: The mangled function name generated by the compiler, for example, _S4_S4_9TileShapeS4_S4_iSC.

Example: To use the function name as the filtering base and collect performance data for all kernels whose function name contains foo, run the following command:

acu --kernel-name-base function --kernel-name regex:foo -o cuda_test ./cuda_test 1

2.3.2 Specify the kernel name

Use the --kernel-name option to set an expression for matching kernel names. The matching criteria depends on the value of --kernel-name-base. For more information, see the description of --kernel-name-base. You can specify the --kernel-name option in the following ways:

• By kernel name: Profiles kernels whose names exactly match the specified string.

• By regular expression: Use the syntax regex:<expression> to profile all kernels with names that match the regular expression.

Example: Match all kernels named Bar.

acu --kernel-name Bar ./cuda_test 1

Example: Match all kernels that contain the string "Bar", such as Bar and FooBar.

acu --kernel-name "regex:Bar" ./cuda_test 1

Example: Match all kernels that contain the string "foo" or "bar", such as foo, foobar, and _bar2.

acu --kernel-name "regex:foo|bar" ./cuda_test 1

2.3.3 Specify the kernel ID

Use the --kernel-id option to specify a kernel ID expression. Only kernels with matching IDs are profiled.

The kernel-id expression uses the following syntax: context-id:stream-id:[name-operator:]kernel-name:invocation-nr. Fields are separated by colons (:). Leave a field empty to disable its filter. The five fields are:

• context-id: Specifies the context ID of the kernel.

• stream-id: Specifies the stream ID of the kernel.

• name-operator: An optional descriptor for the kernel-name field. If omitted, do not include the trailing colon (:).

  • Currently, only regex is supported, which indicates that kernel-name is a regular expression.

• kernel-name: An expression for the kernel name. The matching criteria for the kernel name depends on the value of the --kernel-name-base option. For more information, see the description of --kernel-name-base.

  • If name-operator is regex, profiles all kernels whose names match the regular expression.

  • If name-operator is empty, profiles kernels whose names exactly match the specified name.

• invocation-nr: Specifies the invocation number of this kernel.

  • You can specify multiple invocations by using a regular expression.

  • The invocation count increments only for kernels that match all preceding criteria, such as context-id, stream-id, and kernel-name.

Example: To match the second invocation of the kernel named Foo, without filtering by context or stream ID:

acu --kernel-id ::Foo:2 ./cuda_test

Example: To match the third invocation of all kernels, and all invocations that end with 5:

acu --kernel-id "::::.*5|3" ./cuda_test

Example: To match all kernels that end with "foo":

acu --kernel-id "::regex:^.*foo$" ./cuda_test

Example: To match all kernels that do not start with "foo":

acu --kernel-id "::regex:^(?!foo)" ./cuda_test

Example: To match the fifth invocation of all kernels on context 1 and stream 3:

acu --kernel-id 1:3::5 ./cuda_test

2.3.4 Specify the number of kernels to profile

Use the --launch-count option to set the maximum number of kernels to profile. Only kernels that meet the filtering criteria in the --kernel-name and --kernel-id options are counted.

Example: To collect performance data for only two kernels and skip all subsequent ones:

acu --launch-count 2 ./cuda_test

2.3.5 Specify the number of kernels to skip

Use the --launch-skip option to set the number of kernels to skip before profiling begins. Only kernels that meet the filtering criteria in the --kernel-name and --kernel-id options are counted.

Note: This option is also supported in Range Replay mode, where it counts ranges instead of kernels.

Example: To skip one kernel and then collect performance data for all subsequent kernels:

acu --launch-skip 1 ./cuda_test

Example: To skip one kernel, collect performance data for the next two kernels, and then skip all remaining kernels:

acu --launch-skip 1 --launch-count 2 ./cuda_test

2.3.6 Specify kernels to skip before matching

Use the --launch-skip-before-match option to set the number of kernels to skip before profiling begins. All kernels are counted, regardless of whether they match the filters.

Example: To unconditionally skip two kernels and then profile all kernels whose names contain Foo:

acu --launch-skip-before-match 2 --kernel-name regex:Foo ./cuda_test

Example: To skip the first two kernels regardless of their names (black), then, among kernels whose names contain Foo, skip the next two (red) and profile the following two (green). All remaining kernels are not profiled.

acu --launch-skip-before-match 2 --launch-skip 2 --launch-count 2 --kernel-name regex:Foo ./cuda_test

2.3.7 Terminate application after profiling

Use the --kill option to specify whether to terminate the application after the launch-count limit is reached. The default is yes.

Example: To collect performance data for two kernels and then terminate the application:

acu --launch-count 2 --kill yes ./cuda_test

Example: To collect performance data for two kernels and allow the application to continue running until it finishes. Performance data for subsequent kernels is not collected.

acu --launch-count 2 --kill no ./cuda_test

2.4 Specify the report name

By default, acu generates a .acurep report file for analysis in the GUI. Use the -o option to specify a custom report name. You do not need to add the .acurep file extension, as acu appends it automatically. If you omit the -o option, acu prints the results to the console using --page=details by default.

2.5 Output to console as page or CSV

In addition to generating a report file, acu can print results to the console in different page formats, which correspond to the pages in the GUI report. To select a page, use the --page=<Page> option. Supported pages are details and raw. If you omit the --page option but set an output file with the -o option, no results are printed to the console.

acu also supports CSV output. Use the --csv option to print results in CSV format.

2.6 Import a report file

Use the --import option to import an existing report and print its contents to the console. You can use this option together with the --page <page> option.

2.7 HGTX filtering

--hgtx-include <configuration> --hgtx-exclude <configuration>

Use these command-line options to profile only kernels that match the specified HGTX range configurations.

The HGTX filtering feature requires the --hgtx option. You can specify the --hgtx-include and --hgtx-exclude options one or more times.

Two types of HGTX range configurations are available: HgtxRangeStart/End and HgtxRangePush/Pop. The configuration syntax for both types is described below.

• Start-End Ranges

  Quantifier	Description	Example
  ,	Delimiter between range names.	Range A,Range B Range B,Range A,Range C
  @	Specifies the domain name. If omitted, the default domain is assumed.	Domain A@Range A Domain B@Range B,Range Z

  acu --hgtx --hgtx-include "Domain A@Range A" hgtx_filtering_test

  Profiles kernels that are within Range A in the domain Domain A.

  acu --hgtx --hgtx-include "Range A,Range B" hgtx_filtering_test

  Profiles kernels that are within both Range A and Range B (AND condition).

  acu --hgtx --hgtx-include "Range A" --hgtx-include "Range B" hgtx_filtering_test

  Profiles kernels that are within either Range A or Range B (OR condition).

  acu --hgtx --hgtx-exclude "Range A" hgtx_filtering_test

  Profiles all kernels except for those within Range A.

  acu --hgtx --hgtx-include "Range B"--hgtx-exclude "Range A" hgtx_filtering_test

  Profiles kernels that are within Range B but not within Range A.

• Push-Pop Ranges

  Quantifier	Description	Example
  /	Delimiter between range names.	Range A/Range B Range A/*/Range B Range A/
  [	The range is at the bottom of the stack.	[Range A [Range A/+/Range Z
  ]	The range is at the top of the stack.	Range Z] Range C/*/Range Z]
  +	Matches exactly one range between the two other ranges.	Range B/+/Range D
  *	Matches zero or more ranges between the two other ranges.	Range B/*/Range Z
  @	Specifies the domain name. If omitted, the default domain is assumed.	Domain A@Range A Domain B@Range A/*/Range Z]

  acu --hgtx --hgtx-include "Domain A@Range A/" hgtx_filtering_test

  Profiles kernels that are within Range A in the domain Domain A.

  acu --hgtx --hgtx-include "[Range A" hgtx_filtering_test

  Profiles kernels that are within Range A when Range A is at the bottom of the stack.

  acu --hgtx --hgtx-include "Range A/*/Range B" hgtx_filtering_test

  Profiles kernels that are within both Range A and Range B, with zero or more ranges between them.

  acu --hgtx --hgtx-exclude "Range A/*/Range B" hgtx_filtering_test

  Profiles all kernels except for those that are within both Range A and Range B with zero or more ranges between them.

  acu --hgtx --hgtx-include "Range A/" --hgtx-exclude "Range B]" hgtx_filtering_test

  Profiles kernels that are within Range A but not in a range where Range B is at the top of the stack.

• Other configurations

  --hgtx-include DomainA@RangeA,DomainB@RangeB // Invalid configuration

  A single HGTX configuration can specify only one domain, even if it includes multiple ranges.

  --hgtx-include "Range A\[i\]"

  The quantifiers @, ,, [, ], /, *, and + in a name can be escaped with a backslash (\). Range A\[i\] refers to a range named Range A[i].

  --hgtx-include "Range A" // Start/End configuration

  --hgtx-inlcude "Range A/" // Push/Pop configuration

  --hgtx-inlcude "Range A]" // Push/Pop configuration

  If a name contains a literal backslash (\), you may need to use additional backslashes to escape it depending on your shell. Do not escape quantifiers.

  To include or exclude a single Push/Pop range, use a trailing /. Do not use [ or ].

  --hgtx-include "Range A/*/RangeB"

  The order in a Push/Pop configuration is important. In this example, Range A is below Range B in the stack.

2.8 Run acu with multiple instances

acu can run as multiple instances, either in the same terminal or Docker container. Each instance profiles independently and generates a separate report. This method has limited support for the mpirun command.

Because a single acu instance can profile only one PPU device, you must ensure that the target application for each acu instance has exclusive access to a dedicated PPU device when multiple instances are running. If acu detects that another application is using the profiling device, the following log message is displayed:

The application running on the device[0], may affect the metrics results of profiling.
   Running application list: xxxxxxxxxx 

If this message appears, profiling continues, but the data may be inaccurate because it could include metrics from other applications.

You can use the mpirun command for acu profiling in two ways:

1. Place mpirun after acu to launch it as the target application. The command format is as follows:

   acu -o your_report mpirun -np 8 <your_target_application>

   This method starts only one acu instance. However, mpirun may fork multiple application processes. In this mode, performance data from all forked processes is collected and merged into a single report file.

2. Place mpirun before acu to use it as the launcher. The command format is as follows:

   mpirun -np 8 acu -o mpi_repot_%p <your_target_application>

   This method starts eight acu instances. Each acu instance launches one <your_target_application> process for profiling, resulting in eight separate report files.

2.9 Cache control

In Kernel Replay mode, a kernel may be replayed multiple times to collect all requested metrics. Asight Compute's checkpoint feature saves and restores PPU device memory used by the kernel, but not data in the L1, L2, or LLC caches. These caches can retain data from a previous run. Similarly, hardware performance counter values collected during the first replay may depend on which kernels were executed before the profiled kernel launch. These factors can cause subsequent replays to perform differently than the initial run.

To ensure more stable and accurate hardware performance counter values, Asight Compute provides the --cache-control all parameter, which allows you to flush all PPU caches before each kernel replay. As a result, the kernel accesses a clean cache in each replay, behaving as if it were executed in complete isolation.

However, this mode has side effects. When profiling a specific kernel within a large application, collecting data for cache-focused metrics with this mode enabled can hinder performance analysis. In this case, consider using --cache-control none to disable the tool from flushing any hardware caches.

Similarly, in Application Replay mode, memory accessed by a kernel does not need to be saved and restored via checkpoints, so each kernel launch occurs only once in the application process's lifecycle. Therefore, you can also use --cache-control none to prevent the tool from flushing any hardware caches. This setting is appropriate unless the application requires a clean cache state before a specific kernel launch.

2.10 Clock control

This feature controls the PPU clock during profiling.

Many metric values are directly affected by the current PPU CU and memory clock frequencies. For example, consider the following scenarios:

• If the profiled kernel has other kernels executing before it in the application, the PPU may already be at a higher clock frequency, which affects the kernel's execution time and other metrics.

• If the kernel is the first one launched in the application, the PPU clock frequency is typically lower.

• Because acu performs kernel replay, metric values can be affected by the collection pass, as later passes can cause higher clock frequencies.

To mitigate this variability, acu attempts to lock the PPU clock frequency to its base value (by using --clock-control base). This reduces the impact of the kernel's position in the application and the collection pass on the metric values.

However, in some scenarios, you may not need to lock the clock frequency to the base value. For example, if you have already fixed the clock frequency by using an external tool such as ppu-msi, you can set the --clock-control none option to prevent acu from locking any clock frequencies.

3. Warp sampling

Warp Sampling periodically samples the state of the warp scheduler. At each sampling interval, a sampler selects an active warp and records its PC address and scheduling state.

The acu tool enables Warp Sampling by default. To view the sampling data, open the report in the GUI.

3.1 GUI display

The Details Page and the Source Page display Warp Sampling data.

3.1.1 Details page

On the Details Page, in the Source Counters section, the Warp Stall Sampling table displays the top 5 stall reasons by count, including:

• Source location

• Stall reason count

• Stall reason percentage

See the warp stall reason introduction for details on stall reasons.

3.1.2 Source page

The Source Page presents stall reason data in a bar chart. Both the Source View and PASM View display stall reason values and their totals, line by line, for each code line or instruction.

3.2 Sampling interval

If the collected sampling data is too sparse for analysis, set a smaller sampling interval with the --sampling-interval option.

• The --sampling-interval option accepts a value in the range [0..9]. The actual sampling period is 2^(7 + value) cycles.

• If set to auto (default), the tool automatically selects the highest possible sampling frequency based on the current environment to avoid sparse data or buffer overflow.

Example: Set the sampling period to 256 cycles for pc sampling:

acu --sampling-interval 1 -o cuda_test  ./cuda_test

Example: Automatically select the highest sampling frequency for pc sampling:

acu --sampling-interval auto -o cuda_test  ./cuda_test

Instruction statistics (Beta)

The instruction statistics feature, enabled by default, counts instruction executions for functions like kernel and device functions. To disable it, set the following environment variable before launching acu: export ASIGHT_FEATURE_INSTRUCTION_COUNT=0. This feature supports profiling kernels or ranges in all replay modes, except for graphs (i.e., in --graph-profiling graph scenarios).

acu performs a separate pass to collect instruction execution information and parse assembly instructions. During this pass, the feature gathers warp and thread execution counts for each instruction, which the GUI then displays in various forms.

Source page display

On the Source Page, reports with instruction statistics show the Instructions Executed and Thread Instructions Executed columns in both the Source View and the PASM View.

• Instructions Executed: Counts instruction executions per independent warp, regardless of the number of active threads in each warp.

• Thread Instructions Executed: Counts total instruction executions across all threads.

The instruction statistics feature also enables the GUI to display assembly instruction details and source code correlation for device functions called by a kernel function. For more information, see Source Page.

Details page display

On the Details Page, under the Instruction Statistics Section, a new bar chart displays the instruction execution distribution. For more information, see Executed Instruction Mix Bar Chart:

On the Details Page, the Source Counters Section now displays information for Most Instruction Executed and Most Thread Instruction Executed. For more information, see Source Counters Section: