PAI-PPU best practices

3.1 Configure the PPU pip source

• Using an official Alibaba Cloud PAI-PPU image:

  Official Alibaba Cloud PAI-PPU images are pre-configured with the PPU pip source and necessary authentication. You can use pip install directly.

  pip install <package_name>

• Using a custom image:

  For custom images, you must configure the default pip source and provide the required authentication details:

  export PIP_INDEX_URL="https://<pip_user>:<pip_password>@art-pub.eng.t-head.cn/artifactory/api/pypi/ptgai-pypi_ppu_<os_name>_<cuda_version>_index/simple/"
  export PIP_EXTRA_INDEX_URL="http://mirrors.cloud.aliyuncs.com/pypi/simple/"
  echo -e "machine art-pub.eng.t-head.cn\nlogin <pip_user>\npassword <pip_password>" >> ~/.netrc

  Replace the following placeholder fields:

  • pip_user: The username for the PPU pip source. Contact your PAI PDSA to obtain it.

  • pip_password: The password for the PPU pip source. Contact your PAI PDSA to obtain it.

  • os_name: The operating system of your image, such as ubuntu or alios.

  • cuda_version: The CUDA compatibility version of the PPU SDK in your image, such as cu123 or cu126.

3.2 Identify PPU-specific packages

If installing a package involves CUDA compilation, you must use a PPU-specific version from the PPU-PIP Package List. The following methods help you determine if a package requires a PPU-specific version.

1. Quick check: The most common packages that require PPU-specific versions are:

   torch, torchvision, tensorflow, vllm, sglang, flash-attn, transformer-engine, triton, mmcv, spconv, and open3d.

2. Method 1: Check the open-source library's README file. The introduction usually describes the library's functionality, which can serve as a basic indicator.

3. Method 2: Look at the "Languages" section in the GitHub repository. If CUDA has a high percentage, it is likely a CUDA-related library. If the CUDA percentage is low, such as 2%, you need to use other methods to confirm.

   

4. Method 3: Unpack the wheel (.whl) file of the open-source library and search for files with a .cu extension.

   

   wget https://files.pythonhosted.org/packages/b8/92/fa66b0684c0eace3a480498e70ad40a0f3784890a25b480ae05fb8d7a458/lmdeploy-0.7.1-cp310-cp310-manylinux2014_x86_64.whl
   unzip lmdeploy-0.7.1-cp310-cp310-manylinux2014_x86_64.whl
   find ./ -name "*.cu" 

5. Method 4: Unpack the wheel file and inspect all shared object (.so) files for keywords like cuda or nv_fatbin.

   find ./ -name "*.so*" | xargs readelf -a | grep 'nv_fatbin'
   find ./ -name "*.so*" | xargs readelf -a | grep 'cuda'

   

3.3 Available PPU packages

The list of packages in the PPU pip source is updated with each PPU SDK release. You can view the latest package list here: PPU-PIP Package List.

Packages are categorized as either "optimized version" or "community version":

• Optimized version: Packages that have been performance-tuned for PPU.

• Community version: Open-source packages compiled with the PPU SDK without code modifications.

Unless you have specific requirements, always choose the optimized version to maximize PPU performance.

3.4 Handle missing packages

PAI-PPU is compatible with CUDA and Triton programming interfaces. For most packages not available in the PAI-PPU pip source, you can install them from source by following the package's official documentation. For example, to install SGLang from source, refer to its official guide: https://docs.sglang.ai/start/install.html#method-2-from-source

# Use the last release branch
git clone -b v0.4.4.post1 https://github.com/sgl-project/sglang.git
cd sglang

pip install --upgrade pip
pip install -e "python[all]" --find-links https://flashinfer.ai/whl/cu124/torch2.5/flashinfer-python

You can successfully compile and use it with the ppu-training:1.4.3-pytorch2.5.1-ppu-py310-cu126-ubuntu22.04 official image:

3.5 Troubleshoot missing dependencies

The PAI-PPU pip source has an upstream feature that automatically fetches missing dependencies from public pip sources. If a dependency still fails to install, follow these steps:

1. Run the pip config list command to verify that the pip source is configured correctly.

   

2. Check section 3.3 to confirm if the dependency is missing from the PPU pip source.

3. Follow the instructions in section 3.4 to install the dependency manually.

4. If these steps do not resolve the issue, contact your PAI PDSA.

3.6 Configure a PPU Conda environment

PAI-PPU supports only Python 3.10 or 3.8. When creating a Conda environment, ensure you include the python=3.10 or python=3.8 parameter in the conda create command.

For example, to set up an environment with Python 3.10 and PyTorch 2.4.0:

conda create -n torch240 python=3.10
conda activate torch240

pip install torch==2.4.0+ppu1.4.2.oe
pip install flash-attn==2.5.6+ppu1.4.2.oe
pip install xformers==0.0.29.post1+ppu1.4.2.oe
pip install -r requirement.txt # vllm env below
pip install vllm==0.6.6.post1+ppu1.4.2.oe

Step 1: Choose a base official image

From the official image list, select a base image that meets your requirements by following these principles:

1. Prioritize images that match your required operating system and CUDA compatibility version.

2. If the available CUDA versions do not meet your requirements, contact your PAI PDSA to request a dedicated image.

3. Choose a Python version. If you need Python 3.10, you can find a suitable base image by referring to the PAI-PPU-V1.4.X Official Image Release Notes.

4. If you need Python 3.8, we recommend installing the Python environment by using Conda or another Python package manager.

Step 2: Customize the PPU image

1. Launch a PAI DSW instance using the selected official image.

2. In the DSW instance, install the required Python environment by using Conda or another package manager.

3. Install the necessary packages in the DSW instance.

   # 1. Install PPU-specific versions of packages, such as torch.
   pip install xxx
   
   # 2. Remove packages that require special installation as described in Section 3, then install the remaining packages from your requirements file. Community versions are acceptable.
   pip install -r requirements.txt
   
   # 3. If the previous step accidentally overwrote any PPU-specific packages, reinstall them.

4. In PAI DSW, save the image for future use.

   

5.1 Check PPU runtime environment

You can check the PAI-PPU runtime environment by using the following commands:

• PPU SDK version: cat /usr/local/PPU_SDK/release.yaml

• CUDA compatibility version: nvcc --version

• Container operating system version: cat /etc/os-release

• Python version: python --version

• PyTorch version: python -c "import torch; print(torch.__version__)"

5.2 PPU-SMI tool

PPU-SMI (PPU System Management Interface) is a command-line tool based on HGML (HanGuang Management Library) that helps you manage and monitor PPU devices. The ppu-smi command-line tool allows you to:

• Modify device configurations and feature switches.

• Query runtime parameters and feature status for specific devices.

• Collect runtime data and specific events, and export them to a table for analysis.

• Analyze the device resource usage of different applications.

• Query the topology information of multiple PPU devices.

For detailed instructions on how to use the ppu-smi tool, see ppu-smi Basic Information Summary.

5.3 Asight Systems quick start

Asight Systems is a performance analysis suite for PPU programs. It tracks various CPU and GPU events and displays them on a timeline, allowing you to perform system-level analysis to identify performance bottlenecks. Asight Systems consists of two tools:

• The asys command-line tool: Runs on Linux to collect events from a target program and generate a report. asys supports collecting various event types, such as HGGC, HGTX, OSRT, and their call stack information, with flexible start and stop triggers.

• The Asight Systems GUI: Runs on Windows and macOS to display reports. The GUI provides multiple views, such as TimelineView, EventsView, and FunctionTable, and can smoothly display massive amounts of data in a user-friendly interface.

The workflow for using the Asight Systems tool suite is as follows:

1. Use the asys command-line tool to trace the target program and generate a report.

2. Open the report in the Asight Systems GUI for analysis.

For detailed instructions, see Asight Systems Performance Analysis Suite.

5.4 Asight Compute quick start

Asight Compute is a kernel analyzer for PPU applications. It collects and displays metrics from kernel execution to help you optimize kernel performance. Asight Compute consists of two tools:

• The acu command-line tool: Runs on Linux to collect kernel metrics and generate a report.

• The Asight Compute GUI: Runs on Windows and macOS to display reports. The GUI provides multiple views for metrics, such as Roofline Chart, Bar Chart, and Memory Table, and supports baseline comparison in a user-friendly interface.

The workflow for using the Asight Compute tool is as follows:

1. Use the acu command-line tool to trace the target program and generate a report.

2. Open the report in the Asight Compute GUI for analysis.

For detailed instructions, see Asight Compute kernel analyzer.

6.1 Proprietary RDMA network best practices

The PAI-PPU environment uses a proprietary RDMA network (not NVIDIA InfiniBand) and a unique multi-card interconnect method (ICN Link). Therefore, the configuration for RDMA and the collective communication library requires specific settings.

To configure the correct environment variables, refer to the following documentation: Best Practices for Using the ACCL-P Collective Communication Library.

6.2 PPU programming guide

The Zhenwu 810E accelerator is fully compatible with the CUDA software stack. You can write programs directly in the CUDA language, compile them on the PAI DSW platform, and then run them on Zhenwu 810E by using PAI DLC, PAI EAS, and other modules.

For information on the programming model, CUDA sample best practices, CuBLAS best practices, Cutlass best practices, and compatibility with NVCC and CUDA Samples on the Zhenwu 810E accelerator, refer to the following documentation:

PG1 Programming Guide.

6.3 Compatibility analysis

The PAI-PPU environment is highly compatible with the CUDA programming interface, Triton programming interface, nvcc compiler, and common PyTorch operators.

For detailed compatibility information, see the following documentation: Compatibility.