SDK release note

1.3.1 Firmware

• Supports installation from binary packages (RPM or DEB).

• Supports dynamic power management. By default, the PPU firmware dynamically adjusts the core frequency (from 200 MHz to the maximum operating frequency) and voltage based on real-time workload, temperature, and power consumption.

• Supports PPU frequency locking. For more information, see Device Management Tool PPU-SMI.

• Supports secure firmware signing and a dual-backup mechanism to ensure firmware integrity and reliability.

• Supports firmware upgrades for the power management chip.

• Supports out-of-band management through a Baseboard Management Controller (BMC), including device status monitoring and out-of-band firmware upgrades.

1.3.2 Kernel driver

• Supports installation from binary packages (RPM or DEB) and a runfile package.

• The kernel driver is decoupled from the PPU SDK, allowing versions released after v1.0 to be used with any PPU SDK version.

• Supports flexible machine configurations, including single-node single-card, single-node 8-card/16-card (ICN interconnect), multi-node multi-card (ICN interconnect), and multi-node multi-card (GDR). Using the GDR feature requires that thealixpu-peermem kernel module, which is released with the PPU kernel driver, is installed.

• Supports PPU fault reporting and handling.

• Includes anauto-reset feature that automatically resets the PPU device upon detecting akill overtime orcp invalid cmd error, enabling automatic recovery. This feature is enabled by default, and its status can be queried or disabled using ppu-smi.

• Supports Multi-Pipe Service (MPS), which you can enable or disable by runningppudbg --config_submit_mode 1/0.

• Supports mixed-workload scenarios with both small and large models in MPS mode.

• Supports the collection of NVML Generic Performance Metrics (GPM), including SM utilization, SM occupancy, tensor core utilization, graphics memory bandwidth utilization, PCIe read/write speeds, and ICN link read/write speeds.

• Supports multi-instance GPU (MIG), allowing a single PPU to be partitioned into up to 8 instances. Note that the ICN interconnect feature cannot be used when MIG is enabled.

• Supports full-card passthrough for virtualization. You can unbind the PPU driver on the host machine and pass the entire device through to a virtual machine. When doing so, you must ensure proper ICN isolation between different virtual machines.

• Supports SR-IOV virtualization, allowing a single PPU to be divided into up to 8 Virtual Functions (VFs). Note that when SR-IOV and ICN interconnect are enabled simultaneously, only one VF can access the ICN interconnect.

• Supports hot upgrade of the PPU driver on the host machine in SR-IOV virtualization mode, allowing driver updates without interrupting workloads in the virtual machines.

• Supports hot migration of virtual machines in SR-IOV virtualization mode. This feature is limited to single-card configurations and does not yet support migration over ICN interconnect.

• Adds official support for single-node 16-card Zhenwu 810E configurations and optimizes the default PPU device ordering (sorted from the rear of the chassis).

• Adds GPM collection for L1/L2 cache hit rates, which are queryable via the DCGM tool.

• Fixes high CPU usage when collecting PPU utilization metrics with driver versions v1.2 or v1.2.1.

• Fixes an issue where creating a new CUDA context during a switch in MPS mixed-workload mode could cause the application to hang.

• Fixes a system crash that could occur when a PPU device reset and GPM metric monitoring happened simultaneously.

• Fixes an issue where frequent binding and unbinding of a PPU device in RUND scenarios could lead to MCU timeouts or cause the card to become unresponsive.

• Adjusts the logical numbering scheme for 16-card Zhenwu 810E devices to improve interconnect performance.

• Optimizes driver loading time and memory footprint.

• Adds support for querying the status of the MPS mixed-workload mode.

• Adds support for querying XID error information.

• Adds support for querying, enabling, and disabling GPM status.

• Fixes a discrepancy between process utilization and full-card utilization data reported by the ppu-smi tool.

• Fixes a discrepancy between process graphics memory usage and full-card graphics memory usage reported by the ppu-smi tool.

• Fixes an issue that could randomly cause system hangs when using unified memory (cudaMallocManaged).

• Fixes an issue where the ECC page retirement pending flag was not cleared during DCGM stress tests.

• Disables XID error reporting for HBM parity errors.

1.3.3 User-mode driver and runtime

• Provides compatibility with most CUDA Runtime APIs (cudaXXX) and CUDA Driver APIs (cuXXX).

• Adds more detailed error logging for XID 896 errors to simplify debugging.

• Optimized the performance of cudaMemcpy and cudaMemset for 2D and 3D operations.

• Fixes an intermittenthgpti activity record cycle invalid error that could occur duringasys profiling.

• Supports a circular print buffer to enhance the printf functionality in CUDA device code.

• Supports the stream memory operation v2 API.

• Supports APIs for managing edge data in graphs.

• Supports APIs for batch memory op nodes in graph management.

• Fixed an issue where the qwen2-72b large model produced garbled inference output on the Zhenwu 810E platform when vllm cuda graph was enabled.

• Fixes a graphics memory leak that occurred during FCN model training.

• Fixes an issue that caused intermittent failures in DCGM stress tests on Zhenwu 810E platforms.

• Fixes a core dump issue caused by LLVM symbol conflicts that could occur after a CUDA launch error.

• Fixes an error that occurred when running model inference using the Python multiprocessing module.

1.3.4 Compiler

• Provides a Clang/LLVM-based compiler designed for the PPU architecture. It supports a mixed host/device programming style and is fully compatible with the CUDA C/C++ language specifications.

• Offers a rich set of compilation modules, allowing developers to flexibly create compilation workflows through API calls and easily integrate Just-In-Time (JIT) compilation capabilities.

• Includes a comprehensive suite of development and debugging tools, such as hgobjdump, memcheck, ppu-gdb, and a sanitizer library, to simplify algorithm debugging.

• Supports system-level reserved shared memory.

• Supports Triton versions 2.3.x and 3.0.x.

• Supports GCC host compiler versions from 5.5 to 12.3.

• ppu-gdb:

  • Optimizes the display length of kernel managed names in layout assembly mode.

  • Supports conditional breakpoints on internal variables, such as blockIdx, and registers.

  • Supports the gdb python extension for Python versions 3.6 to 3.10.

  • Supports multiple trigger behaviors for device kernel breakpoints: single-trigger and multi-trigger. The default is single-trigger.

1.3.5 Acceleration libraries

Compute acceleration library

• Proprietary compute libraries include acdnn, acblas, acfft, acsolver, and acrand.

  • acdnn supports the following operators:

    Conv	BatchNorm	Pooling	Softmax
    Activation	CTCLoss	Dropout	LRN
    LSTM	GRU	MultiHeadAttn	Tensor Ops
    SpatialTransform	Backend fusion

  • acblas supports the following operators:

    Level-1 operators	Gemv	Gemm	Matmul + epilogue
    MatrixTransform	trsm	getrfBatched	getrsBatched
    geqrfBatched	gelsBatched

  • acfft supports R2C/C2R/C2C/D2Z/Z2Z + FFT/iFFT transforms.

  • acsolver supports LU decomposition/solve, Cholesky decomposition/solve, QR decomposition, and SVD decomposition (Jacobi method).

  • acrand supports:

    • Pseudo-random number generators: XORWOW, MRG32K3A, and PHILOX4_32_10.

    • Data distributions: Default, Uniform, Normal, and LogNormal.

  • acext quantization library supports:

    • Various kernel variants for A16W8/A16W4 and PerChannel/GroupWise.

    • Various kernel variants for A8W8 and PerChannel/PerToken.

    • Accelerated kernels for WeightonlyBatchedGemv with small batch sizes.

• Key updates:

  • cutlass3: Adds support for the group GEMM persistent strategy.

  • Further improves FlashAttention performance on Zhenwu 810E by up to 85%.

  • acext: Adds support for a16w8 sub-channel quantization.

  • MoE: Adds support and optimizations for a16w4 quantization.

  • xformers: Fixes an infinity issue in attention mask scenarios.

  • RTC: Added support for group conv and gemv.

  • Add more pre-compiled instances to reduce the chance of RTC.

  • BLAS: Added support for rank-1 and rank-2 Level 2 APIs.

  • BLAS: Fixed incorrect gemm behavior when k = 0.

  • blas INT8 gemm: Now supports INT8 input and NN, TN, and TT configurations.

  • BLAS: Added support for cublasSgetriBatched.

  • conv: Improves performance by over 100x for wgrad scenarios with inputs larger than 4 GB.

  • conv: Fixes a 1D SpatialTF exception issue.

  • acFFT: Adds support for C2C, D2Z, and Z2Z, along with several new helper APIs.

  • solver: Adds support for Cholesky decomposition/solve, QR decomposition, and SVD decomposition (Jacobi method).

Interconnection acceleration library

• Provides compatibility with most NCCL APIs (ncclXXX) and environment variables.

• Supports standard communication operators such as AllReduce, AllGather, ReduceScatter, Broadcast, Reduce, Send, and Recv.

• Supports multi-card communication within a single node using ICN, PCIE, and ShareMemory.

• Supports multi-node communication using RDMA (GDR & non GDR) and Socket.

• Supports ICN scale-out communication between 610 cards on multiple nodes.

• The debugging feature for multi-node and multi-card systems is enhanced. It now supports automatic monitoring for ppu device kernel execution hangs, hierarchical communication status dumps, and triggering dumps through a configuration file.

• Optimizes the synchronization overhead between network interface card (NIC) communication and ppu devices in multi-node scenarios, and supports the NCCL_GDR_COPY_SYNC and PCCL_GDR_USE_DEV_MEM_FOR_RX_TAIL features.

• Adds support for newer NCCL versions, including new APIs, environment variables, and configurations:

  • Added API feature support: ncclCommInitRankConfig, ncclCommFinalize, ncclCommSplit, ncclCommRegister, ncclCommDeregister, ncclMemAlloc, ncclMemFree, and ncclCommGetAsyncError.

  • Added support for the following environment variables: NCCL_P2P_DIRECT_DISABLE, NCCL_NET_PLUGIN, NCCL_COMM_BLOCKING, NCCL_LOCAL_REGISTER, and NCCL_REPORT_CONNECT_PROGRESS.

  • Supports the new debug log subsystem fields: NCCL_PROFILE & NCCL_ALLOC & NCCL_DUMP;

  • Added support for the ncclRemoteError error field.

• Other key features and fixes:

  • Optimizes the graph search logic for multi-node, multi-card environments to fix inconsistencies in proxyRank between the search and connect phases.

  • Optimized device memory usage in single-node, multi-GPU environments.

  • Optimized runtime host memory allocation and release in group mode.

  • Fixed a crash that occurred when using the FC algorithm in a fully interconnected, non-ICN scenario on a single node with four cards.

  • Fixed a failure to select the optimal path type when path types were inconsistent between nodes in a multi-node scenario.

• PCCL tools:

  • pccl-perf:

    • Supports AllReduce, AllGather, ReduceScatter, All-to-All, Broadcast, and Reduce.

    • Optimizes event-based timing for device kernels.

  • p2pBandwidthAndLatencyPerf tool enhancements:

    • Supports ICN P2P interconnect bandwidth and latency performance evaluation for products like Zhenwu 810E in various multi-card interconnect topologies.

    • Supports batch output of bandwidth and latency data for all read and write modes, similar to nv p2p perf bench.

    • Supports displaying the p2p bandwidth performance ratio matrix.

    • Supports specifying a device subset for peer-to-peer (p2p) performance benchmarks.

    • Fixed a performance issue with data reads in bidirectional CE mode.

  • PCCL check tools:

    • Supports functional readiness checks for products like Zhenwu 810E in multi-card or multi-node, multi-card scenarios.

    • Adds a check to verify the 16 x Zhenwu 810E interconnect topology.

    • Added a check to verify the ICN link speed.

    • Added a validity check for the acsCtrl configuration.

1.3.6 Hardware-accelerated video and image codecs

Compatible with the NVIDIA Video Codec SDK (including cuvid decode, nvenc, and nvjpeg) and NVIDIA Performance Primitives (NPP) for 2D images, enabling out-of-the-box hardware acceleration for frameworks like FFmpeg, OpenCV, DALI, and PyTorch without requiring code modifications.

Video decode

• Supports NVIDIA cuvid decoder.

• Codecs:

  • HEVC (H.265) - ITU-T Rec. H.265 (04/2013), ISO/IEC 23008-2

    • Main Profile, Level 5.1, High Tier

    • Main10 Profile, Level 5.1, High Tier

    • Main Still Profile

  • VP9 - vp9-bitstream-specification-v0.6-20160331-draft

    • Profile 0, 8-bit

    • Profile 2, 10-bit

  • AVC (H.264) - ITU-T Rec. H.264 (03/2010) / ISO/IEC 14496-10

    • Main Profile, levels 1 - 5.2

    • High Profile, levels 1 - 5.2

    • High 10 Profile, levels 1 - 5.2

    • Baseline Profile, levels 1 - 5.2

  • AV1 Bitstream & Decoding Process Specification Version 1.0.0 with Errata 1

    • Main Profile, Level 5.1

  • AVS2

• Resolution up to 8192x8192.

• Up to 192 FHD streams.

Video encode

• Supports nvenc.

• Codecs:

  • AVC (H.264): Spec Version 12:ISO/IEC 14496-10 / ITU-T Rec. H.264 (03/2010)

    • Baseline Profile, levels 1 – 5.2

    • Main Profile, levels 1 - 5.2

    • High Profile, levels 1 – 5.2

    • High 10 Profile, levels 1 - 5.2

  • HEVC (H.265): ITU-T Rec. H.265 (04/2013), ISO/IEC 23008-2

    • Main Profile, Level 5.1, High Tier

    • Main10 profile, Level 5.1, High Tier

    • Main Still Profile

  • AV1 Bitstream Specification Version 1.0.0 with Errata 1

    • Main Profile, Level 5.1

• Resolution up to 4K.

• Supports RGB input format (converted to YUV420 via inlinePP).

• Supports crop, scale, and rotate with inlinePP.

• Up to 32 FHD streams.

JPEG

• Supports nvjpeg decoder and encoder.

• Resolution up to 32Kx32K.

• Supports RGB format input and output with inlinePP.

• Supports crop, scale, and rotate with inlinePP.

• Up to 960 FPS at UHD resolution.

Image processing

• Supports NVIDIA Performance Primitives (NPP) for 2D images.

1.3.7 Software tools

• The following PPU management and monitoring tools are provided for large-scale cloud cluster monitoring and can be integrated into existing operations and maintenance systems:

  • The PPU device management tool ppu-smi, which is similar to nvidia-smi. For details, see Device Management Tool PPU-SMI.

  • New features in ppu-smi v1.4 include:

    • Adds support for querying descriptions of XID error codes.

    • Added a description of how to query the status of performance counters.

    • Added a description for the gpm subcommand to query and set GPM.

    • Added a description for querying the burstable mode.

    • Added a description of how to query the product architecture and minor number.

    • Added a description for setting and querying MPS mode.

  • The data center management and monitoring tool PPU DCGM, which is similar to NVIDIA DCGM.

  • New features in DCGM v1.4 include:

    • Supports field IDs for ICN per-link send/receive rates.

    • Updated the field ID support list

  • The hgml library, which is similar to NVIDIA NVML.

• The PPU performance analysis tools Asight System and Asight Compute (similar to NVIDIA Nsight Systems and Nsight Compute) are available for performance analysis of single-node and multi-node training and inference scenarios.

  • Asight System is a low-overhead, system-level performance analysis tool that collects various system events, CPU and PPU activities, API execution times, call stacks, NVTX ranges, and correlations between CPU/PPU activities. It provides a unified visualization in the Timeline View, allowing developers to easily analyze CPU/PPU workloads and their relationships, identify performance bottlenecks, and ensure coordinated operation and maximum parallelism between the CPU and PPU. For details, see Performance Analysis Suite Asight Systems.

  • New features in Asight System v1.4 include:

    • Improves UI styles and layout, with more tab styles for switching views.

    • asys lets you set the call stack depth for CPU profiling and Python profiling.

    • asys supports collecting trace data from specific processes of an application.

    • asys collects basic information, such as PPU frequency and temperature.

    • asys supports a shorter sampling period for collecting CPU call stack information.

    • asys stats supports exporting statistics and traces for HGGC kernel grid blocks

    • asys collects HGGC Python backtrace call stacks.

    • asys supports collecting memory Python backtraces.

    • asys Python functions trace collects all threads in a Python process.

    • Timeline View supports custom colors for HGTX.

    • In Timeline View, the time percentage of PPU nodes updates after filtering.

    • Timeline View displays metrics for RDMA NICs.

    • Timeline View displays PPU Activity dependencies

    • The Timeline View displays HGGC Graph information separately.

    • Timeline View lets you mark the timeline.

    • Timeline View displays different types of Video timelines in different colors.

    • The HGGC Launch API now supports display by kernel name in Timeline View.

    • You can close a report tab while it is opening.

    • Added a tile view to the Events View and enhanced autofill for search history.

    • Function View now includes flame graphs and icicle graphs.

    • HGTX range aggregation supports specifying a list of processes and threads.

  • Asight Compute is a kernel performance analysis tool that collects PPU hardware performance counters and presents them as metrics. The GUI offers various perspectives on these metrics for in-depth kernel analysis and optimization. For details, see Kernel Analyzer Asight Compute.

  • New features in Asight Compute v1.4 include:

    • Adds topology graphs for 16-card and 8-card Zhenwu 810E configurations.

    • The Source page displays source markers. It also collects and displays execution information for kernel assembly instructions, including Instructions Executed and Thread Instructions Executed.

    • Metrics fixes and stability enhancements

• Provides the PPU debugging tool PPU-GDB, which allows simultaneous debugging of both GPU and CPU code within the same application.

• Provides PPU Memcheck, a tool suite for functional correctness checks. The suite includes a series of checkers, such as memcheck, initcheck, synccheck, and racecheck.

• Provides the PPU binary tool hgobjdump, for extracting device-related information from binaries.

2.3.1 Supported open-source frameworks and libraries

Description:

• PyTorch support:

  • PyTorch 2.1 and earlier versions are based on NGC.

  • PyTorch 2.1.2 and later versions are based on the official community releases of torch.

• TensorFlow support:

  • Google's official community release of TensorFlow 1.12 supports only CUDA 9, while the corresponding release of version 1.15 supports only CUDA 10. https://www.tensorflow.org/install/source?hl=en#gpu_support_3

    The provided TensorFlow 1.15 is based on the open-source NVIDIA NGC nv22.04 release. https://docs.nvidia.com/deeplearning/frameworks/support-matrix/index.html#unique_1997996242

  • TensorFlow 1.12 is based on PAI TF 1.12.

  • TensorFlow 2.4 is based on NVIDIA NGC nv21.02.

  • TensorFlow 2.6 and 2.7 are based on the official community releases from Google.

  • TensorFlow 2.8 is based on NVIDIA NGC nv22.04.

  • TensorFlow 2.10 and later are based on the official community releases from Google.

• Items marked with an asterisk (*), such as FlashAttention(*) and cutlass(*), fully match their respective community release versions. They provide only feature support and do not include performance optimization.

• Most other open source frameworks and libraries are based on their community release versions, with a few exceptions that are based on NGC. For details, refer to the user guide for each specific framework or library.

• For these open source frameworks and libraries, the PPU SDK supports compilation, basic runtime validation, and unit testing. However, due to resource constraints, the v1.4 release does not resolve all issues discovered during testing. For a list of outstanding issues, see the known issues section.

2.3.2 Open source framework compatibility