Accelerate PyTorch model inference with EAIS (Python)

Background

This tutorial guides you through creating an eais.ei-a6.2xlarge EAIS instance in the China (Hangzhou) region. Using the resnet50 and bert-base models as examples, it demonstrates the complete process of using EAIS to accelerate PyTorch models for image classification and NLP inference. If you encounter performance or functional issues during EAIS inference, contact EAIS Technical Support for customized optimization solutions.

Inference methods

EAIS supports the following two methods for PyTorch model inference:

• Use the EAIS miniconda environment

  The EAIS miniconda environment, developed by Alibaba Cloud, is a ready-to-use environment based on miniconda. It includes sample scripts and model files for testing and evaluation.

  This tutorial focuses on using Python scripts. For C++ instructions, see Use EAIS to accelerate PyTorch model inference (C++).

• Install the EAIS Python package in an existing PyTorch environment

  For more information, see Use EAIS to accelerate PyTorch model inference (Python) in the Operations Guide.

Inference performance

EAIS delivers significantly higher inference performance than a GPU instance (NVIDIA T4). The following table compares the inference performance of an eais.ei-a6.2xlarge EAIS instance with a GPU instance (NVIDIA T4) when using a Python script.

Step 1: Create and attach ECS and EAIS instances

1. Create an ECS instance.

   1. Go to the ECS instance creation page.

   2. Create an ECS instance in a VPC to which you can attach the EAIS instance.

      For detailed instructions, see Create an instance by using the wizard.

      Note

      We recommend using Ubuntu 18.04 as the operating system for the ECS instance.

2. Create an EAIS instance.

   1. Log on to the EAIS console.

   2. In the navigation pane on the left, choose Elastic Acceleration Instance EAIS > Instances.

      Important

      If your Alibaba Cloud account does not have the AliyunServiceRoleForEais role (with the AliyunServiceRoleForEais policy), you must first authorize EAIS to create a service-linked role before you can create an EAIS instance. For more information, see Service-linked roles for EAIS.

   3. Click Create Elastic Acceleration Instance.

   4. On the purchase page, configure the parameters for the EAIS instance, and then click Buy Now.

      The following table describes the parameters.

      Parameter	Example configuration
      Region	China (Hangzhou)
      Instance name	eais_test
      Instance type	eais.ei-a6.2xlarge
      VPC	eais-vpc
      vSwitch	eais-vswitch
      Security group	eais-securitygroup
      Resource group	Default Resource Group

   5. On the Confirm Order page, for the Service Agreement, select I have read and agree to the Elastic Acceleration Instance Service Agreement, and then click Activate Now.

3. Attach the EAIS instance to the ECS instance.

   1. Click Management Console to return to the EAIS console.

   2. In the EAIS instance list, find the instance named eais_test and click Attach in the Actions column.

   3. In the dialog box that appears, select the ECS instance to attach.

   4. Click OK.

      The instance list refreshes automatically every 15 seconds. When the instance status changes from Attaching to Attached, the attachment is complete.

      Important

      An EAIS instance can be attached to only one ECS instance at a time, but an ECS instance can have one or more EAIS instances attached. For more information, see Limits.

Step 2: Start the EAIS instance

1. In the EAIS instance list, find the instance named eais_test and click Start in the Actions column.

2. In the Start Instance dialog box, click OK.

   The instance list refreshes automatically every 15 seconds. When the instance status changes from Attached to In Use, the instance has started.

Step 3: Log on to the ECS instance

1. Log on to the EAIS console.

2. In the EAIS instance list, click the ECS instance ID associated with eais_test to open the instance details page.

3. Remotely connect to the ECS instance.

   For detailed instructions, see Connection methods. On the ECS instance details page, click Remote Connection.

Step 4: View EAIS information

1. Install the eais-tool package.

   1. Run the following commands to install the eais-tool Debian package.

      export VERSION=4.2.5
      wget https://eais-rel-pub.oss-cn-beijing.aliyuncs.com/packages/eais-tool_${VERSION}_amd64.deb
      sudo dpkg -i eais-tool_${VERSION}_amd64.deb
      source /etc/profile

   2. Run the following command to verify the installation.

      dpkg -l | grep eais-tool

      If the output contains information about the eais-tool package, the installation was successful.

2. View EAIS information.

   After you install the eais-tool package, run the eais_smi command to view information such as the EAIS instance type and GPU utilization. The following is a sample output.

   root@ixxx:~# eais_smi
   |-------------------------------+----------------------+----------------------|
   | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
   | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
   |                               |                      |               MIG M. |
   |===============================+======================+======================|
   |   0  eais.ei-a6.2xlarge  On   | 00000000:00:07.0 Off |                    0 |
   | N/A   28C    P8    9W /  70W  |      0MiB / 15109MiB |      0%      Default |
   |                               |                      |               N/A    |
   +-------------------------------+----------------------+----------------------+

Step 5: Set up the runtime environment

1. Install CUDA.

   1. Run the following commands to download and install the CUDA package.

      wget https://developer.download.nvidia.com/compute/cuda/11.7.0/local_installers/cuda_11.7.0_515.43.04_linux.run
      sudo sh cuda_11.7.0_515.43.04_linux.run --silent --toolkit

   2. Run the following commands to set the CUDA-related environment variables.

      export PATH=/usr/local/cuda/bin:$PATH
      export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

2. Install the eais-cuda package.

   Note

   This topic uses an ECS instance that runs Ubuntu as an example. For instructions on how to install cGPU on CentOS or Alibaba Cloud Linux, see eais-cuda.

   1. Run the following commands to install the eais-cuda Debian package.

      export VERSION=4.2.5
      wget https://eais-rel-pub.oss-cn-beijing.aliyuncs.com/packages/eais-cuda_${VERSION}_amd64.deb
      sudo dpkg -i eais-cuda_${VERSION}_amd64.deb

   2. Run the following command to verify the installation.

      dpkg -l | grep eais-cuda

      If the output contains information about the eais-cuda package, the installation was successful.

3. Download and activate EAIS miniconda.

   1. Run the following commands to download and unzip the eais-miniconda package.

      wget https://eais-rel-pub.oss-cn-beijing.aliyuncs.com/packages/eais-miniconda.zip
      sudo apt-get update && sudo apt-get install -y unzip
      unzip -q eais-miniconda.zip

   2. Run the following command to change to the root directory of the eais-miniconda package.

      cd eais-miniconda/

      Note

      All subsequent operations are performed from this root directory.

   3. Run the following command to activate the EAIS miniconda environment.

      source bin/activate

Step 6: Run and accelerate PyTorch inference

1. Run the following command to change to the Python script test directory.

   cd eais/python

2. Run the following command to use EAIS for inference with the resnet50 model.

   python pytorch_resnet50.py \
   -m ../data/models/resnet50.pt \
   -l ../data/densenet_labels.txt \
   ../data/images/

   The parameters are described as follows:

   • pytorch_resnet50.py: A sample inference script from the eais-miniconda package.

   • ../data/models/resnet50.pt: The resnet50 model file from the eais-miniconda package.

   • ../data/densenet_labels.txt: A file that contains image class names, from the eais-miniconda package.

   • ../data/images/: The path to sample images from the eais-miniconda package.

   The output shows the image classification results. The following is a sample output.

   image_id: 0, image_name: ../data/images/car.jpg, max_prob: 0.7719907164573669, max_label_id: 817, class_name: SPORTS CAR
   image_id: 1, image_name: ../data/images/vulture.jpeg, max_prob: 0.96502685546875, max_label_id: 23, class_name: VULTURE
   image_id: 2, image_name: ../data/images/mug.jpg, max_prob: 0.8115575909614563, max_label_id: 504, class_name: COFFEE MUG

3. Run the following command to test the inference performance of the resnet50 model with EAIS.

   python pytorch_resnet50_perf.py \
     -m ../data/models/resnet50.pt

   The output shows the inference latency data for the resnet50 model. The following is a sample output.

   (eais-miniconda) root@iZZzed7gollsbkwlqs1fh5Z:~/eais-miniconda/eais/python# python pytorch_resnet50_perf.py \
       -m ../data/models/resnet50.pt
   use 2.192821741104126 ms each inference

4. Run the following command to test the inference performance of the bert-base model with EAIS.

   python pytorch_bert_perf.py \
     -m ../data/models/bert-base-uncased.pt

   The output shows the inference latency data for the bert-base model. The following is a sample output.

   (eais-miniconda) root@xxx:~/eais-miniconda/eais/python# python pytorch_bert_perf.py \
       -m ../data/models/bert-base-uncased.pt
   use 5.374878644943237 ms each inference

(Optional) Step 7: Stop the EAIS instance

Stop the EAIS instance if you do not currently need its GPU acceleration.

1. Log on to the EAIS console.

2. In the EAIS instance list, find the instance named eais_test and click Stop in the Actions column.

3. In the Stop Instance dialog box, click OK.

   The instance list refreshes automatically every 15 seconds. When the instance status changes from In Use to Attached, the instance has stopped.

(Optional) Step 8: Detach and release EAIS instance

Detach an EAIS instance to attach it to a different ECS instance. If you no longer need the EAIS instance, you must first detach it and then release it.

1. Log on to the EAIS console.

2. Detach the EAIS instance from the ECS instance.

   1. In the EAIS instance list, find the instance named eais_test and click Detach in the Actions column.

   2. In the Detach ECS Instance dialog box, click OK.

      The instance list refreshes automatically every 15 seconds. When the instance status changes from Attached to Attachable, the detachment is complete.

3. Release the detached EAIS instance.

   1. In the EAIS instance list, find the instance named eais_test and click Release Resource in the Actions column.

   2. In the Release Resource dialog box, click OK.

      The EAIS instance is then removed from the instance list.

Related documents

• You can also use a C++ program for PyTorch model inference with EAIS. For instructions, see Use EAIS to accelerate PyTorch model inference (C++).

• To learn how to use EAIS for TensorFlow model inference, see Use EAIS to accelerate TensorFlow model inference.