FAQ and known issues

FAQ

1. Reduce test environment variance

When profiling with acu, you can reduce variance across data collection runs by setting the device to exclusive mode. This setting prevents other applications from running while acu is sampling.

# Set the device to exclusive mode
ppu-smi -c EXCLUSIVE_PROCESS

# Restore the device to default mode
ppu-smi -c DEFAULT

You can also lock the device frequency to minimize the impact of frequency scaling on acu sampling:

# Lock the device frequency to 1.5 GHz
ppu-smi -lpc 1500

# Unlock the device frequency
ppu-smi -rpc

2. acu hangs after the target application exits

Some target applications fork multiple child processes. If a child process continues to run after the main process exits, acu waits for it to terminate. This may cause acu to appear unresponsive.

To identify this issue:

1. Install the pstree utility.

2. If acu appears unresponsive, open a new terminal and run the following command: pstree <acu_pid>. Check if any processes forked by the target application appear in the output. Their presence indicates that a child process did not exit correctly.

Workaround:

Use the --wait primary parameter in your acu command. This instructs acu to wait only for the main process of the target application to exit.

3. Unable to collect PPU performance data

acu may fail to collect PPU performance data if the required resources are in use by another application. Common causes include:

• Another acu instance is collecting trace data.

• Another asys application is collecting device operational metrics.

• DCGM is collecting profiling metrics. You can pause DCGM collection by running dcgmi profile --pause.

Known issues

• PPU SDK v1.2.0: acu may crash when profiling applications that subscribe to hgpti.

• Profiling individual kernel nodes within an HGGC graph is not supported. However, you can profile the entire graph by using the --graph-profiling graph option.

• Profiling applications that use hggcLaunchCooperativeKernelMultiDevice is not supported.

• If the file I/O speed of the location specified by the acu -o option is low, the generated acu report may be incomplete.

• The --profile-from-start option does not support cross-process control of the collection time range.

• Applications must be compiled with the -pthread option. Otherwise, acu may crash during runtime.

• If the acu collection process is interrupted, the collected trace information might be displayed incorrectly.

• In the Warp State Statistics section, the value for Avg. Active Threads Per Warp [inst] is lower than expected.

• The following memory capture and replay APIs are unstable in Range Replay mode and may affect data accuracy.

  # HGGC runtime API
  hggcMemcpy2D,
  hggcMemcpy2DArrayToArray,
  hggcMemcpy2DArrayToArray_ptds,
  hggcMemcpy2DAsync,
  hggcMemcpy2DAsync_ptsz,
  hggcMemcpy2DFromArray,
  hggcMemcpy2DFromArrayAsync,
  hggcMemcpy2DFromArrayAsync_ptsz,
  hggcMemcpy2DFromArray_ptds,
  hggcMemcpy2DToArray,
  hggcMemcpy2DToArrayAsync,
  hggcMemcpy2DToArrayAsync_ptsz,
  hggcMemcpy2DToArray_ptds,
  hggcMemcpy2D_ptds,
  hggcMemcpy3D,
  hggcMemcpy3DAsync,
  hggcMemcpy3DAsync_ptsz,
  hggcMemcpy3DPeer,
  hggcMemcpy3DPeerAsync,
  hggcMemcpy3DPeerAsync_ptsz,
  hggcMemcpy3DPeer_ptds,
  hggcMemcpy3D_ptds,
  hggcMemcpyArrayToArray,
  hggcMemcpyArrayToArray_ptds,
  hggcMemcpyAsync,
  hggcMemcpyAsync_ptsz,
  hggcMemcpyFromArray,
  hggcMemcpyFromArrayAsync,
  hggcMemcpyFromArrayAsync_ptsz,
  hggcMemcpyFromArray_ptds,
  hggcMemcpyFromSymbol,
  hggcMemcpyFromSymbolAsync,
  hggcMemcpyFromSymbolAsync_ptsz,
  hggcMemcpyFromSymbol_ptds,
  hggcMemcpyPeer,
  hggcMemcpyPeerAsync,
  hggcMemcpyToArray,
  hggcMemcpyToArrayAsync,
  hggcMemcpyToArrayAsync_ptsz,
  hggcMemcpyToArray_ptds,
  hggcMemcpyToSymbol,
  hggcMemcpyToSymbolAsync,
  hggcMemcpyToSymbolAsync_ptsz,
  hggcMemcpyToSymbol_ptds,
  hggcMemcpy_ptds,
  hggcMemset,
  hggcMemset2D,
  hggcMemset2DAsync,
  hggcMemset2DAsync_ptsz,
  hggcMemset2D_ptds,
  hggcMemset3D,
  hggcMemset3DAsync,
  hggcMemset3DAsync_ptsz,
  hggcMemset3D_ptds,
  hggcMemsetAsync,
  hggcMemsetAsync_ptsz,
  hggcMemset_ptds,
  hggcStreamAttachMemAsync,
  hggcStreamAttachMemAsync_ptsz,
  hggcHostAlloc,
  hggcMalloc,
  hggcMalloc3D,
  hggcMalloc3DArray,
  hggcMallocArray,
  hggcMallocHost,
  hggcMallocManaged,
  hggcMallocMipmappedArray,
  hggcMallocPitch,
  hggcFree,
  hggcFreeArray,
  hggcFreeHost,
  hggcFreeMipmappedArray,
  hggcLaunch,
  hggcLaunch_ptsz,
  hggcLaunchKernel_ptsz,
  hggcLaunchCooperativeKernel_ptsz,
  hggcLaunchCooperativeKernelMultiDevice,
  
  # Hggc Driver API
  hgMemcpyHtoD,
  hg64MemcpyHtoD,
  hgMemcpyDtoH,
  hg64MemcpyDtoH,
  hgMemcpyDtoD,
  hg64MemcpyDtoD,
  hgMemcpyDtoA,
  hg64MemcpyDtoA,
  hgMemcpyAtoD,
  hg64MemcpyAtoD,
  hgMemcpyHtoA,
  hgMemcpyAtoH,
  hgMemcpyAtoA,
  hgMemcpy2D,
  hgMemcpy2DUnaligned,
  hgMemcpy3D,
  hg64Memcpy3D,
  hgMemcpyHtoDAsync,
  hg64MemcpyHtoDAsync,
  hgMemcpyDtoHAsync,
  hg64MemcpyDtoHAsync,
  hgMemcpyDtoDAsync,
  hg64MemcpyDtoDAsync,
  hgMemcpyHtoAAsync,
  hgMemcpyAtoHAsync,
  hgMemcpy2DAsync,
  hgMemcpy3DAsync,
  hg64Memcpy3DAsync,
  hg64Memcpy2D,
  hg64Memcpy2DUnaligned,
  hg64Memcpy2DAsync,
  hgMemcpy_v2,
  hgMemcpyHtoD_v2,
  hgMemcpyHtoDAsync_v2,
  hgMemcpyDtoH_v2,
  hgMemcpyDtoHAsync_v2,
  hgMemcpyDtoD_v2,
  hgMemcpyDtoDAsync_v2,
  hgMemcpyAtoH_v2,
  hgMemcpyAtoHAsync_v2,
  hgMemcpyAtoD_v2,
  hgMemcpyDtoA_v2,
  hgMemcpyAtoA_v2,
  hgMemcpy2D_v2,
  hgMemcpy2DUnaligned_v2,
  hgMemcpy2DAsync_v2,
  hgMemcpy3D_v2,
  hgMemcpy3DAsync_v2,
  hgMemcpyHtoA_v2,
  hgMemcpyHtoAAsync_v2,
  hgMemcpy,
  hgMemcpyAsync,
  hgMemcpyPeer,
  hgMemcpyPeerAsync,
  hgMemcpy3DPeer,
  hgMemcpy3DPeerAsync,
  hgMemcpyHtoD_v2_ptds,
  hgMemcpyDtoH_v2_ptds,
  hgMemcpyDtoD_v2_ptds,
  hgMemcpyDtoA_v2_ptds,
  hgMemcpyAtoD_v2_ptds,
  hgMemcpyHtoA_v2_ptds,
  hgMemcpyAtoH_v2_ptds,
  hgMemcpyAtoA_v2_ptds,
  hgMemcpy2D_v2_ptds,
  hgMemcpy2DUnaligned_v2_ptds,
  hgMemcpy3D_v2_ptds,
  hgMemcpy_ptds,
  hgMemcpyPeer_ptds,
  hgMemcpy3DPeer_ptds,
  hgMemcpyAsync_ptsz,
  hgMemcpyHtoAAsync_v2_ptsz,
  hgMemcpyAtoHAsync_v2_ptsz,
  hgMemcpyHtoDAsync_v2_ptsz,
  hgMemcpyDtoHAsync_v2_ptsz,
  hgMemcpyDtoDAsync_v2_ptsz,
  hgMemcpy2DAsync_v2_ptsz,
  hgMemcpy3DAsync_v2_ptsz,
  hgMemcpyPeerAsync_ptsz,
  hgMemcpy3DPeerAsync_ptsz,
  hgMemsetD8,
  hg64MemsetD8,
  hgMemsetD16,
  hg64MemsetD16,
  hgMemsetD32,
  hg64MemsetD32,
  hgMemsetD2D8,
  hg64MemsetD2D8,
  hgMemsetD2D16,
  hg64MemsetD2D16,
  hgMemsetD2D32,
  hg64MemsetD2D32,
  hgMemsetD8Async,
  hg64MemsetD8Async,
  hgMemsetD16Async,
  hg64MemsetD16Async,
  hgMemsetD32Async,
  hg64MemsetD32Async,
  hgMemsetD2D8Async,
  hg64MemsetD2D8Async,
  hgMemsetD2D16Async,
  hg64MemsetD2D16Async,
  hgMemsetD2D32Async,
  hg64MemsetD2D32Async,
  hgMemsetD8_v2,
  hgMemsetD16_v2,
  hgMemsetD32_v2,
  hgMemsetD2D8_v2,
  hgMemsetD2D16_v2,
  hgMemsetD2D32_v2,
  hgMemsetD8_v2_ptds,
  hgMemsetD16_v2_ptds,
  hgMemsetD32_v2_ptds,
  hgMemsetD2D8_v2_ptds,
  hgMemsetD2D16_v2_ptds,
  hgMemsetD2D32_v2_ptds,
  hgMemsetD8Async_ptsz,
  hgMemsetD16Async_ptsz,
  hgMemsetD32Async_ptsz,
  hgMemsetD2D8Async_ptsz,
  hgMemsetD2D16Async_ptsz,
  hgMemsetD2D32Async_ptsz,
  hgMemSetAccess,
  hgStreamAttachMemAsync,
  hgStreamAttachMemAsync_ptsz,
  hgMemAlloc,
  hg64MemAlloc,
  hgMemAllocPitch,
  hg64MemAllocPitch,
  hgMemAllocHost,
  hgMemHostAlloc,
  hg64MemHostAlloc,
  hgMemAlloc_v2,
  hgMemAllocPitch_v2,
  hgMemHostAlloc_v2,
  hgMemAllocHost_v2,
  hgMemAllocAsync,
  hgMemAllocAsync_ptsz,
  hgMemAllocFromPoolAsync,
  hgMemAllocFromPoolAsync_ptsz,
  hgMemFree,
  hg64MemFree,
  hgMemFreeHost,
  hgMemFree_v2,
  hgMemAddressFree,
  hgMemFreeAsync,
  hgMemFreeAsync_ptsz,

• In App-Range Replay mode with instruction statistics enabled, profiling in a multi-GPU scenario may be slow because the process is single-threaded.

• The logical icnlink throughput shown in the ICNlink Section is based on MAC layer data. As a result, this value is higher than the actual data sent and received by users over the ICNlink.

• On macOS, force-quitting the GUI while it is opening a report may cause the application to crash or become unresponsive.