Collect a trace with the asys CLI

The Asight Systems command-line tool, asys, lets you collect performance profiling data from a target application without a GUI. You can transfer the resulting report to another system for analysis in the GUI.

Key capabilities of asys include:

• Trace collection

  • Collects execution information for CUDA, cuDNN, cuBLAS, NVTX, and OSRT APIs.

  • Collects PPU kernel execution and memory operation data, and correlates information between the CPU and the PPU.

  • Collects execution information for the PCCL communication process.

  • Collects call stacks for CUDA and OSRT APIs.

  • Collects CPU scheduling information and call stacks during CPU execution. Supports generating a function execution time breakdown based on call stacks.

  • Collects memory usage information from both the PPU and CPU.

  • Collects metrics such as throughput for network interface cards.

• Collection process control

  • You can control the collection duration, delay the start of a trace, and manually interrupt the collection process.

  • You can specify the collection range by using NVTX ranges.

  • You can specify the collection range by using the CUDA profiler API (cudaProfilerStart/Stop).

  • Supports repeated trace collection triggers with a configurable number of iterations.

  • Supports automatic generation of report names and lets you assemble names by using macros.

  • Supports customization of the application runtime environment, including configuring environment variables and controlling application output.

  • Supports profiling applications that run as daemons and lets you specify how to determine when the application has ended.

  • Supports report file rotation for long-duration collection.

• Interactive trace collection

  • Provides separate controls for launching the application and collecting the trace (start/stop/launch/shutdown subcommands).

  • You can start and stop trace collection multiple times while an application is running.

  • Supports multiple coexisting collection processes and lets you view a list of these processes.

  • Supports attaching to a running application to collect trace information (attach subcommand).

• Statistical analysis and post-processing

  • Supports grouped statistical analysis of device memory usage.

  • Supports statistical analysis of PPU time utilization.

To configure the asys environment, see Configure the asys command-line tool.

To view the help, run the asys -h command. asys provides several subcommands to support diverse tracing methods:

root@0b0f55fa89fd:~# asys -h

usage: asys [--version] [--help] <command> [<args>] [application] [<application args>]

The most commonly used asys commands are:
profile       Run an application and capture its profile into a asysrep file.
attach        Attach to process and capture its profile into a asysrep file.
launch        Launch an application ready to be profiled.
start         Start a profiling session.
stop          Stop a profiling session and capture its profile into a asysrep file.
cancel        Cancel a profiling session and discard any collected data.
shutdown      Disconnect launched processes from the profiler and shutdown the profiler.
sessions      List active sessions.
status        Provide current status of CLI or the collection environment.
export        Export asysrep file into another format.
stats         Generate statistics from an existing asysrep or SQLite file.
analyze       Identify optimization opportunities in a asysrep file.

Use 'asys --help <command> ' for more information about a specific command.

To view help for a specific subcommand, run asys <sub_command> -h. For example, to see the help for the profile subcommand, run:

asys profile -h

To check the currently installed version of asys, run:

asys -v

1. Collect trace information

You can run the asys profile command to specify trace items, run your application, and generate a trace report.

The syntax for the profile subcommand is: asys profile [option] <application> [application args]

1.1 Specify trace items

You can use the --trace or -t option to specify the trace types (trace items) to enable. Separate multiple trace items with a comma (,). For example:

asys profile -t hggc,hgtx,acblas -o baseline python test_linear.py

• -t hggc,hgtx,acblas specifies the trace items to enable: hggc, hgtx, and acblas.

• -o baseline specifies the name of the output report (the file extension is not required).

• python test_linear.py runs the application.

After the application finishes or is interrupted by pressing Ctrl + C, asys generates the corresponding trace report (in this example, baseline.asysrep). You can then view the report in Asight Systems.

2. Control the collection process

2.1 Control collection time

You can use the --delay (or -y) option to specify a delay between application launch and the start of trace collection. You can use the --duration (or -d) option to specify the collection duration. After the specified duration, asys stops the application and generates a report. For example:

asys profile -t hggc --delay 2 --duration 3 python test_linear.py

• --delay 2: Starts collection 2 seconds after the application launches.

• --duration 3: Collects data for 3 seconds. After this period, the application stops, and a report is generated.

The following figure illustrates the collection process timeline:

Tip: To prevent the application from stopping after the trace collection times out, use the --kill none option.

2.2 Event-triggered trace collection

You can insert specific events into your application code to define a collection range. By specifying start and end trigger events in asys, you can precisely control the scope of the trace.

asys supports two trigger methods, which you specify with the --capture-range or -c option. The corresponding trigger events are:

For example, use hggcProfilerApi as an event trigger to capture a trace for a specified code range by inserting the cudaProfilerStart and cudaProfilerStop event APIs into your application code.

cudaProfilerStart(); // profile start
DoProcess();
cudaProfilerStop(); // profile stop
DoOtherProcess();

asys profile -t hggc -c hggcProfilerApi cuda_test

• -c hggcProfilerApi starts trace collection at cudaProfilerStart and stops collection at cudaProfilerStop, and then the application terminates. In this example, asys captures only the trace for the DoProcess execution.

Tip: If hggcProfilerStart and hggcProfilerStop appear multiple times during collection, such as two starts followed by a stop, asys uses the first occurrence of start and the first occurrence of stop to define the collection time range.

Example: To use hgtx as an event trigger, insert an NVTX range named "DoProcess" in your application code to define a trace range.

for (int index = 0; index < 5; ++index) {
    nvtxRangePushA("DoProcess"); // start
    DoProcess();
    nvtxRangePop(); // stop
}

asys profile -t hggc,hgtx -c hgtx -p DoProcess cuda_test

• -c hgtx specifies HGTX/NVTX as the event trigger source.

• -p DoProcess specifies the HGTX/NVTX range name. In this example, the range name is "DoProcess" and the domain is the default domain. Trace collection starts when the "DoProcess" range begins and stops when the range ends, after which the application terminates.

When using HGTX/NVTX as the event trigger source, the --hgtx-capture or -p option supports several templates for matching domains and ranges:

For both hggcProfilerApi and hgtx triggers, asys supports repeated event triggering (each trigger generates a new report) and lets you specify the behavior after a trigger event ends. Use the --capture-range-end option to control this behavior:

This example shows repeated trace collection, where asys collects a trace twice, generates two reports, and then stops the application:

asys profile -t hggc,hgtx -c hgtx -p DoProcess --capture-range-end repeat-shutdown:2 cuda_test

• -c hgtx specifies HGTX/NVTX as the event trigger source.

• -p DoProcess specifies the HGTX/NVTX range name, which is "DoProcess" in this example.

• --capture-range-end repeat-shutdown:2 triggers trace collection twice and then stops the application.

Tip: When --capture-range-end is set to repeat[:N] or repeat-shutdown:N, if the interval between a stop event and the next start event in the application is very short:

• asys might not respond to the start event. In this case, asys starts the next collection when it matches a subsequent start event.

• You might lose trace data from other processes (which may have different trigger events) at the beginning of the collection.

2.3 Report file rotation

For long-duration traces, asys supports report file rotation using the --trace-rotation option in conjunction with the --duration option:

• asys collects data for the specified --duration, generates a report, and then starts the next collection cycle.

• asys keeps only the most recent report files. The number of files to keep is specified with the --trace-rotation option.

• You can use the --output option to specify a report name template. Use the %t parameter in the name to record the trace start time and avoid duplicate filenames.

For example, to generate a report file every 30 seconds and keep the three most recent files:

asys profile -t hggc --duration 30 --trace-rotation 3 --output test_report_%t cuda_test

• --duration 30 specifies a collection duration of 30 seconds for each file.

• --trace-rotation 3 specifies that the three most recent report files should be kept.

• --output test_report_%t specifies the report name template. asys replaces %t with the trace start time, resulting in a filename such as test_report_08_09_33.asysrep.

Tips:

1. The --trace-rotation option cannot be used with the --capture-range option.

2. There is a short gap between the end of one collection and the start of the next, during which asys does not collect trace data.

3. asys continues to collect trace data until the application ends. You can stop the application and end collection by pressing Ctrl + C.

2.4 Specify the report name

You can use the --output or -o option to specify the report name (asys adds the .asysrep extension automatically). Use the --force-overrite true or -f true option to allow overwriting of a file with the same name. When specifying a report name with the -o option, asys can replace macro variables with their corresponding values. The following macros are supported:

For example:

asys profile -t hggc -o report_%q{HGGC_DRIVER_CANDIDATE}_%i python test_linear.py

• -o report_%q{HGGC_DRIVER_CANDIDATE}_%i: In this example, the final report file is named report_UMD_2.asysrep.

  • asys replaces %q{HGGC_DRIVER_CANDIDATE} with the value of the HGGC_DRIVER_CANDIDATE environment variable.

  • asys replaces %i with a unique index value that prevents file name conflicts for files with the same prefix in the current directory.

Tip: The default value for the -o option is report%i. Therefore, if you do not specify the -o option, asys will not overwrite the report file.

2.5 Customize the application runtime environment

You can specify the runtime environment configuration for the application:

Use the --env-var or -e option to add an environment variable for the application. Separate multiple environment variables with a ,.

Use the --inherit-environment false or -n false option to prevent the application from inheriting system environment variables at runtime.

Use the --show-output false or -w false option to suppress application console output.

For example:

asys profile -e ENABLE_DEBUG=1,LOG_LEVEL=DEBUG -n false -w false python test_linear.py

• -e ENABLE_DEBUG=1,LOG_LEVEL=DEBUG: Configures the ENABLE_DEBUG and LOG_LEVEL environment variables when running the application.

• -n false: Prevents the application from inheriting system environment variables at runtime.

• -w false: Suppresses application console output.

2.6 Wait for the application to exit

You can specify how asys determines when an application has finished. By default, asys considers the application finished when all its forked processes have exited. For example, with an application running as a daemon, asys waits by default for all background daemon processes to exit before stopping trace collection. You can specify how to wait for the application to finish by using the --wait option:

2.7 System-level tracing

By default, asys collects trace information for the application's process tree (the root process and its descendants). If other background applications on the system contend for CPU resources and affect the application's performance, it can be difficult to identify such issues using only process tree trace information. In these cases, you can use asys to collect trace data for all processes on the operating system (a system-level trace) to analyze the impact of background applications.

You can use the --sample or -s option to enable system-level trace collection. For example:

asys profile -s system-wide -t hggc python test_linear.py

• -s system-wide: Collects CPU scheduling trace data for all processes on the system.

You can perform a system-level trace without specifying an application. When you do not specify an application, you cannot collect application-level trace items. For example:

asys profile -s system-wide --ppu-metrics-device all --nic-metrics true

• -s system-wide: Collects CPU scheduling trace data for all processes on the system.

• --ppu-metrics-device all: Enables metrics sampling for all PPU devices.

• --nic-metrics true: Collects network interface card performance metrics.

asys supports various system-level trace collection items, such as --ppu-metrics-device, which the help information marks as "System scope". Correspondingly, asys supports various application-level trace collection items, such as --trace, which the help information marks as "Application scope". You can set the -s option to system-wide or process-tree to switch CPU scheduling collection between system-level and application-level. For details on trace item types, see the asys profile -h help information.

3. Interactive trace collection

In addition to collecting traces with asys profile, you can use separate asys commands to control application launch, collection start, collection stop, and application shutdown. This gives you flexible control over the tracing process.

3.1 Interactive commands

asys provides the following subcommands to control the trace collection process. The options for these subcommands are similar to those for asys profile. You can view the help information by running asys <subcommand> -h:

By combining these subcommands, you can flexibly control the timing of application execution and trace collection.

Example: Launch an application, collect two traces, and then stop the application.

asys launch -t hggc cuda_test
asys start -o test_report1
asys stop
asys start -o test_report2
asys stop
asys shutdown

• asys launch ... launches the application.

• asys start ... starts trace collection and specifies the report file.

• asys stop stops trace collection. The application continues to run, allowing you to run asys start ... again to start a new trace collection.

• asys shutdown stops the application.

Example: Start trace collection, specify event trigger conditions, and then launch the application.

asys start -o test_report -c hgtx -p DoProcess
asys launch -t hggc,hgtx cuda_test

• asys start ... starts trace collection and uses the -c and -p options to specify event trigger conditions. The trace will begin after the application starts.

• asys launch ... launches the application. Since you have already initiated trace collection with asys start, tracing can begin as soon as the application launches. asys starts recording data when the specified event trigger conditions are met.

3.2 Manage multiple collection processes

asys supports multiple coexisting collection processes, such as multiple asys profile or asys launch / start processes running simultaneously. Each collection process corresponds to a session, which you use to differentiate between them. You can create or associate with a session using specific options. Each session has a name and ID, and session names must be unique.

Example: Run asys launch to create a session, then run asys start to associate with that session and begin trace collection.

asys launch -t hggc --session-new test cuda_test
asys start --session test

• --session-new test creates a session named "test".

• --session test associates with the session named "test".

To view the list of current collection process sessions:

asys sessions list

The ID column shows the session ID for each session. You can use this ID with the --session option to specify the session.

If you do not specify a session option, asys uses the following default session names:

• For the asys profile subcommand, the session name is profile-<pid>-<application>.

• For the asys launch, start, stop, cancel, and shutdown subcommands, the session name is [default].

Tip: Multiple asys profile subcommands can coexist by default because their session names are different. For interactive subcommands, only one session with the default name can exist at a time.

4. Attaching to a process (Beta)

asys supports attaching to a running application to collect trace data. You do not need to launch the application with asys beforehand. The asys attach subcommand can attach to multiple application processes simultaneously, and you can use it repeatedly on the same application. After collection is complete, the application continues to run without interruption.

4.1 Collecting a trace with attach

The following example shows how to use asys attach to collect trace data from multiple application processes:

asys attach -t hggc -f true -o attach_report 94644,94923

• -t hggc collects HGGC-related traces.

• -f true allows overwriting of a report file with the same name.

• -o attach_report specifies the report name as attach_report.

• 94644,94923 attaches to processes with PIDs 94644 and 94923 simultaneously. Separate multiple PIDs with a ,.

After you run asys attach, trace data collection begins. You can stop collection and generate a report file by pressing Ctrl + C, or you can specify the collection duration with the --duration option. For example:

asys attach -t hggc --duration 10 94644,94923 

• --duration 10 collects data for 10 seconds, then stops collection and generates a report. The application's execution is not affected.

You can use asys attach repeatedly to collect trace data from an application. It does not stop the application after collection ends. asys attach has minimal performance impact on the application after collection stops, allowing the application to continue running without interference.

4.2 Supported trace collection scope

The trace collection features of asys attach are similar to those of asys profile. To view the specific features, run asys attach -h.

The following table describes the information collected by each trace item available with the -t option:

asys attach also supports the following trace collection features:

• Supports CPU sampling with --sample and call stack collection with --backtrace.

• Supports Python call stack sampling with --python-sampling.

• Supports host memory usage sampling with --host-memory-sampling.

• Supports device metrics collection with --ppu-metrics-device.

• Supports network interface card metrics collection with --nic-metrics.

• Supports HGGC call stack collection with --hggc-backtrace.

• Supports device memory usage and locked-page memory usage collection with --hggc-memory-usage.

4.3 Attach usage notes

When using asys attach for trace collection, keep the following in mind:

• asys attach can only run in x86_64/arm instruction set environments.

• Using asys attach to attach to an application may cause the application to deadlock.

• Do not mix different release versions of asys attach on the same application process.

asys attach has the following limitations:

• Does not currently support collecting trace data for HGTX, ACDNN, ACBLAS, OSRT, or Video.

• Does not support collecting trace information for legacy HGGC graphs that the application instantiated before you attached.

• Does not support attaching to a process that is already using the HGPTI library.