PPU driver release notes

None

Fixed an issue that occasionally caused device loss during multi-node training. This fix requires SDK 2.0.0. For a detailed description of the issue, refer to the notes for KMD v1.5.5.

None

None

Fixed an issue where applications would hang on CentOS 7.9.

On CentOS 7.9, a system hang may occur under specific conditions involving a read-only buffer and the cudaHostRegister function. A fix is scheduled for a future release.

Added support for the Hibernate and Resume power management features in Linux.

Fixed an issue where Zhenwu 810E devices with a 2-slice MIG configuration had an inconsistent instance type.

Fixed an issue where systems would hang in multi-process scenarios (with more than five processes) after disabling MPS mode.

None

None

Fixed an issue that occasionally caused device loss during multi-node training.

• In certain cross-node training tasks for autonomous driving models, the PPU driver incorrectly configured the request priority for PCIe access to PPU HBM. This caused the Last Level Cache (LLC) to prioritize cacheable requests over lower-priority PCIe read/write operations.

• Solution: Updated the register configuration to adjust the priority of PCIe operations that access PPU HBM. This fix is a joint update of the kernel driver (KMD) and the PCCL SDK, and it requires PPU SDK 1.6.1-hotfix2.

In rare cases, creating a CI/VF may fail on Zhenwu 810E cards when using the MIG/SR-IOV virtualization feature with a 2-slice configuration. A fix is scheduled for a future release.

None

Fixed an issue where PCCL all_reduce would hang for ACS customers using PPU driver v1.5.0.

In rare cases, creating a CI/VF may fail on Zhenwu 810E cards when using the MIG/SR-IOV virtualization feature with a 2-slice configuration. A fix is scheduled for a future release.

None

Fixed a device memory out-of-memory (OOM) error that occurred with PPU SDK 1.5.1 in CUDA 12.6 environments, even when ample device memory was available.

In rare cases, creating a CI/VF may fail on Zhenwu 810E cards when using the MIG/SR-IOV virtualization feature with a 2-slice configuration. A fix is scheduled for a future release.

None

[PAI] Fixed an issue that occasionally caused device loss when launching services with rund in Zhenwu 810E environments.

Fixed an issue where excessive Correctable ECC (CECC) errors caused dmesg to generate an overwhelming number of logs, causing the disk to fill.

In rare cases, creating a CI/VF may fail on Zhenwu 810E cards when using the MIG/SR-IOV virtualization feature with a 2-slice configuration. A fix is scheduled for a future release.

None

Fixed an issue that caused random crashes in multi-node DeepEP scenarios.

Fixed an issue where Uncorrectable ECC (UECC) errors were not repaired correctly after a system reboot.

In rare cases, creating a CI/VF may fail on Zhenwu 810E cards when using the MIG/SR-IOV virtualization feature with a 2-slice configuration. A fix is scheduled for a future release.

Fixed an issue where CUDA host memory usage was not reflected in process memory and cgroup memory statistics.

Fixed a DMA unmap CPU page error that occurred when iommu=nopt.

Improved performance when registering memory regions with GPUDirect RDMA (GDR).

Fixed an issue where ICN link 6 would occasionally become invalid.

Fixed an issue where ppu-smi reported less than 96 GB of total device memory.

Fixed an issue that caused accuracy anomalies during ONNX inference.

Fixed a system crash caused by creating an excessive number of streams during CUDA graph capture for vLLM 0.8.3 model inference.

In rare cases, creating a CI/VF may fail on Zhenwu 810E cards when using the MIG/SR-IOV virtualization feature with a 2-slice configuration. A fix is scheduled for a future release.

None

Fixed an issue that caused accuracy anomalies during ONNX inference.

None

None

Fixed an issue where CUDA host memory usage was not reflected in process memory and cgroup memory statistics.

In SR-IOV single-VF mode, device loss may occasionally occur when running multi-process video applications. A fix is scheduled for a future release.

Further optimized device memory usage in scenarios with a high volume of 1-bit ECC errors. The memory footprint is limited to a maximum of 256 MB to prevent OOM errors.

Freed up hidden repair resources to reduce the likelihood of remapping failures in scenarios with 2-bit ECC errors.

Fixed an issue where reinstalling the driver cleared the remap fail occur flag.

Improved the device utilization metric to more accurately reflect the actual workload.

In SR-IOV single-VF mode, device loss may occasionally occur when running multi-process video applications. A fix is scheduled for a future release.

Optimized driver load time, reducing it from approximately 45s to 22s on a 16-card Zhenwu 810E system.

Optimized driver load time in rund guests:

• Added the SkipFlr module parameter to optionally skip the Function-Level Reset (FLR) operation during driver loading. Note: This parameter can only be used the first time the driver is loaded within a guest.

• Changed the default behavior for large system memory allocations from zeroing on allocation to deferred zeroing.

Fixed an issue that caused application core dumps and occasional system crashes when switching MPS tidal mode on multi-card systems.

Fixed a CPU hard lockup crash caused by excessive logging when a large number of exceptions occurred.

Fixed an issue where 1-bit ECC overflow did not correctly report an Xid error.

Adjusted the TX buffer threshold to fix poor P2P performance when using Broadcom cables across chassis.