Accelerate PyTorch model inference with EAIS (C++)

Background information

This tutorial guides you through creating an EAIS instance of the eais.ei-a6.2xlarge instance type in the China (Hangzhou) region. Using the public resnet50 and bert-base models as examples, you will learn how to use an EAIS instance to run PyTorch model inference for image classification and NLP scenarios and accelerate performance. If you encounter performance or feature issues while using an EAIS instance, contact EAIS technical support for customized optimization solutions.

This tutorial focuses on running PyTorch model inference with a C++ program. To learn how to run PyTorch model inference by using a Python script, see Accelerate PyTorch model inference by using an EAIS instance (Python).

Compared to using a GPU instance (NVIDIA T4), using an EAIS instance significantly improves inference performance. The following table compares the inference performance of a C++ program (dependent on libtorch) running on an eais.ei-a6.2xlarge EAIS instance with that on a GPU instance (NVIDIA T4).

Step 1: Create and associate an ECS instance and an EAIS instance

1. Create an ECS instance.

   1. Log on to the ECS instance creation page.

   2. Create a virtual private cloud (VPC) ECS instance to associate with the EAIS instance.

      For more information, see Create an instance.

      Note

      Use Ubuntu 18.04 as the operating system for the ECS instance.

2. Create an EAIS instance.

   1. Log on to the EAIS console.

   2. In the left navigation pane, choose Elastic Accelerated Computing Instances EAIS > Elastic Accelerated Computing Instances List.

      Important

      If your Alibaba Cloud account has not been granted the AliyunServiceRoleForEais role, which uses the AliyunServiceRoleForEais access policy, you must first authorize EAIS to create a service-linked role before you can create an EAIS instance. For more information, see Service-linked Role.

   3. Click Create Elastic Accelerated Computing Instance.

   4. On the EAIS instance purchase page, configure the instance parameters and click Buy Now.

      The following table describes the configuration items:

      Configuration item	Example configuration
      Region	China (Hangzhou)
      Instance name	eais_test
      Instance type	eais.ei-a6.2xlarge
      Virtual Private Cloud	eais-vpc
      vSwitch	eais-vswitch
      Security group	eais-securitygroup
      Resource group	Default Resource Group

   5. On the Confirm Order page, agree to the Service Agreement by selecting the I have read and agree to the Elastic Accelerated Computing Instances Service Agreement checkbox, and then click Activate Now.

3. Associate the EAIS instance with the ECS instance.

   1. Click Management Console to navigate to the EAIS console.

   2. In the list of EAIS instances, find the eais_test instance and click Associate in the Operation column.

   3. In the dialog box, select the ECS instance that you want to associate.

   4. Click OK.

      The EAIS instance list auto-refreshes every 15 seconds. The EAIS instance is successfully associated when the instance status changes from Associating to Associated.

      Important

      An EAIS instance can be associated with only one ECS instance. An ECS instance can be associated with one or more EAIS instances. For more information, see Limits.

Step 2: Start the EAIS instance

1. In the EAIS instance list, find the eais_test instance and click Start in the Actions column.

2. In the Start Instance dialog box, click OK.

   The EAIS instance list auto-refreshes every 15 seconds. When the instance status changes from Associated to In Use, the EAIS instance starts.

Step 3: Log on to the ECS instance

1. Log on to the EAIS console.

2. In the EAIS instance list, click the ID of the ECS instance that is associated with the eais_test instance to go to the details page of the ECS instance.

3. Remotely log on to the ECS instance.

   For more information, see Select a method for remote connection to an ECS instance.

Step 5: Set up the runtime environment

In this tutorial, you set up the C++ runtime environment using an EAIS Debian package as an example. This package is compiled with CXX11 ABI and depends on libtorch 1.13.1.

1. Install CUDA.

   1. Run the following commands to download and install the CUDA package.

      wget https://developer.download.nvidia.com/compute/cuda/11.7.0/local_installers/cuda_11.7.0_515.43.04_linux.run
      sudo sh cuda_11.7.0_515.43.04_linux.run --silent --toolkit

   2. Run the following commands to configure environment variables for CUDA.

      export PATH=/usr/local/cuda/bin:$PATH
      export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

2. Install the eais-cuda package.

   1. Run the following commands to install the eais-cuda Debian package.

      export VERSION=4.2.5
      wget https://eais-rel-pub.oss-cn-beijing.aliyuncs.com/packages/eais-cuda_${VERSION}_amd64.deb
      sudo dpkg -i eais-cuda_${VERSION}_amd64.deb

   2. Run the following command to check whether the eais-cuda Debian package is installed.

      dpkg -l | grep eais-cuda

      If the command output contains the eais-cuda package information, the eais-cuda Debian package is installed.

3. Install the eais-torch package.

   1. Run the following commands to install the eais-torch package.

      export VERSION=4.2.5
      wget https://eais-rel-pub.oss-cn-beijing.aliyuncs.com/packages/eais-torch1.13.1-cxx11-abi_${VERSION}_amd64.deb
      sudo dpkg -i eais-torch1.13.1-cxx11-abi_${VERSION}_amd64.deb
      source /etc/profile

   2. Run the following command to check whether the eais-torch Debian package is installed.

      dpkg -l | grep eais-torch

      If the command output contains the eais-torch package information, the eais-torch Debian package is installed.

Step 6: Run and accelerate PyTorch model inference

1. Download the EAIS C++ sample.

   1. Run the following commands to download and decompress the eais-miniconda package that contains the EAIS C++ sample.

      wget https://eais-rel-pub.oss-cn-beijing.aliyuncs.com/packages/eais-miniconda.zip
      sudo apt-get update && sudo apt-get install -y unzip
      unzip -q eais-miniconda.zip

   2. Run the following command to go to the root directory of the eais-miniconda package.

      cd eais-miniconda/

      Note

      All subsequent commands in this tutorial should be run from this directory.

2. Use the C++ package to run and accelerate PyTorch model inference.

   1. Run the following command to go to the test directory of the C++ package.

      cd eais/cpp

   2. Compile the C++ sample script.

      1. Run the following command to install the packages that are required to compile and run the sample script.

         sudo apt-get install -y build-essential cmake libopencv-dev

      2. Run the following commands to compile the executable sample program.

         mkdir build
         cd build
         cmake ..
         make -j8
         cd ..

      3. Run the following command to view the generated sample program.

         ls build/src

         The following output indicates that the sample program is generated.

         root@...SZ:~/eais-miniconda/eais/cpp# ls build/src/
         CMakeFiles  cmake_install.cmake  libpre_post_proprocess.a  Makefile  pytorch_bert_perf  pytorch_resnet50  pytorch_resnet50_cuda  pytorch_resnet50_perf

3. Run the C++ sample program.

   1. Run the following command to configure the path of the dynamic library on which the sample program depends.

      export LD_LIBRARY_PATH=`pwd`/third_party/libtorch/lib:$LD_LIBRARY_PATH

   2. Run inference for the resnet50 model.

      1. Run the following command to run inference for the resnet50 model. The CUDA call information is not included.

         Note

         In this example, data preprocessing and post-processing are implemented on the CPU by using opencv and torch interfaces. The code is in the src directory.

         LD_PRELOAD=libtorcheais.so \
         build/src/pytorch_resnet50 \
           -m ../data/models/resnet50.pt \
           -l ../data/densenet_labels.txt \
           ../data/images

         The parameters are described as follows:

         • LD_PRELOAD: preloads the installed libtorcheais.so dynamic library.

         • build/src/pytorch_resnet50: the executable sample program pytorch_resnet50 that is generated by compilation.

         • ../data/models/resnet50.pt: the resnet50 model file that is provided in the eais-miniconda package.

         • ../data/densenet_labels.txt: the file that stores image category names and is provided in the eais-miniconda package.

         • ../data/images/: the path of the sample images that are provided in the eais-miniconda package.

         The execution result provides the image classification information. The following code shows a sample output.

         image_id: 0, image_name: ../data/images/car.jpg, max_prob: 0.700754, max_label_id: 817, class_name: SPORTS CAR
         image_id: 1, image_name: ../data/images/mug.jpg, max_prob: 0.849474, max_label_id: 504, class_name: COFFEE MUG
         image_id: 2, image_name: ../data/images/vulture.jpeg, max_prob: 0.933375, max_label_id: 23, class_name: VULTURE

      2. Run the following command to run inference for the resnet50 model. The CUDA call information is included.

         Note

         In this example, data preprocessing and post-processing are implemented by using a cuda kernel. The code is in the src directory.

         LD_PRELOAD=libtorcheais.so \
         build/src/pytorch_resnet50_cuda \
           -m ../data/models/resnet50.pt \
           -l ../data/densenet_labels.txt \
           ../data/images

         The parameters are described as follows:

         • LD_PRELOAD: preloads the installed libtorcheais.so dynamic library.

         • build/src/pytorch_resnet50_cuda: the executable sample program build/src/pytorch_resnet50_cuda that is generated by compilation.

         • ../data/models/resnet50.pt: the resnet50 model file that is provided in the eais-miniconda package.

         • ../data/densenet_labels.txt: the file that stores image category names and is provided in the eais-miniconda package.

         • ../data/images/: the path of the sample images that are provided in the eais-miniconda package.

   3. Test the inference performance of the resnet50 and bert-base models.

      1. Run the following command to test the inference performance of the resnet50 model on an EAIS instance.

         LD_PRELOAD=libtorcheais.so \
         build/src/pytorch_resnet50_perf \
           -m ../data/models/resnet50.pt \
           -d gpu

         The execution result provides the inference latency data for the resnet50 model. The following code shows a sample output.

         5Z:~/eais-miniconda/eais/cpp# LD_PRELOAD=libtorcheais.so \
         > build/src/pytorch_resnet50_perf \
         >   -m ../data/models/resnet50.pt \
         >   -d gpu
         test device: gpu
         average forward time: 2.20047 ms

      2. Run the following command to test the inference performance of the bert-base model on an EAIS instance.

         LD_PRELOAD=libtorcheais.so \
         build/src/pytorch_bert_perf \
           -m ../data/models/bert-base-uncased.pt \
           -d gpu

         The execution result provides the inference latency data for the bert-base model. A successful command returns the average forward time. The value 5.37935 ms represents the inference time after EAIS acceleration.

         average forward time: 5.37935 ms

(Optional) Step 7: Stop the EAIS instance

You can stop the EAIS instance if you do not temporarily need its GPU capabilities.

1. Log on to the EAIS console.

2. In the EAIS instance list, find the eais_test instance, and in its Operation column, click Stop.

3. In the Stop Instance dialog box, click OK.

   The EAIS instance list auto-refreshes every 15 seconds. A change in the instance status from In Use to Associated indicates that the EAIS instance has stopped.

(Optional) Step 8: Disassociate and release the EAIS instance

After inference is complete, you can disassociate the EAIS instance from the current ECS instance and associate it with another one. If you no longer need the EAIS instance, you must first disassociate it before you can release it. To disassociate and release an EAIS instance, perform the following steps:

1. Log on to the EAIS console.

2. Disassociate the EAIS instance from the ECS instance.

   1. In the EAIS instance list, find the eais_test instance and click Disassociate in the Operation column.

   2. In the Disassociate from ECS Instance dialog box, click OK.

      The EAIS instance list auto-refreshes every 15 seconds. When the instance status changes from Associated to Available, the EAIS instance is disassociated.

3. Release the disassociated EAIS instance.

   1. In the list of EAIS instances, find the eais_test instance and click Release Resource in the Operation column.

   2. In the Release Resource dialog box, click OK.

      The EAIS instance is deleted from the EAIS instance list.

References

• You can also use a Python script to run PyTorch model inference with an EAIS instance. For more information, see Accelerate PyTorch model inference by using an EAIS instance (Python).

• To get started quickly with EAIS, you can also run TensorFlow model inference on an EAIS instance. For more information, see Accelerate TensorFlow model inference by using an EAIS instance.