DocsICP FilingConsole
官方文档
首页

Infer a PyTorch model using EAIS (C++)

更新时间：
复制 MD 格式
产品详情
我的收藏

After you attach an Elastic Accelerator Inference Service (EAIS) instance to an Elastic Compute Service (ECS) instance, you can remotely log on to the ECS instance and use the EAIS instance for AI inference. This topic describes how to use a C++ program to infer a PyTorch model using EAIS.

Prerequisites

Background information

EAIS supports PyTorch model inference in different programming languages based on your business scenario. The following two methods are available:

  • Use a Python script to infer a PyTorch model. For more information, see Infer a PyTorch model using EAIS (Python).

  • Use a C++ program to infer a PyTorch model. EAIS supports calls from C++ programs that include libtorch and CUDA, including the CUDA API and CUDA kernels.

This topic describes how to use this method for inference. If you encounter performance or feature issues when you use EAIS for inference, contact EAIS technical support for a custom optimization solution.

Compared with GPU-accelerated instances (NVIDIA T4), EAIS significantly improves inference performance. The following table compares the inference performance of an EAIS instance (eais.ei-a6.2xlarge) with a GPU-accelerated instance (NVIDIA T4) when using a C++ program that depends on libtorch.

Note

  • The data in this topic is for reference only. The actual data is subject to your inference results.

  • You can also refer to the C++ program code provided in the eais-miniconda package to test the inference performance of a GPU-accelerated instance (NVIDIA T4) and compare it with an EAIS instance.

Inference model

eais.ei-a6.2xlarge

GPU-accelerated instance (NVIDIA T4)

Performance improvement of EAIS over GPU-accelerated instance (NVIDIA T4)

resnet50

2.20 ms

6.24 ms

2.84 times

bert-base

5.38 ms

8.32 ms

1.55 times

Limits

EAIS supports the following software versions:

Important

If your ECS instance runs CentOS 7.9, the default glibc and libstdc++ software environments do not meet the requirements. You must manually upgrade the glibc and libstdc++ versions. For more information about how to view and upgrade the software versions, see Related operations.

Software

Version

glibc version

2.23 or greater

libstdc++ version

3.4.25 or later

Procedure

Deploy an inference program using EAIS

Develop and compile the code on your development machine. Then, copy the compiled program to the ECS instance to infer the PyTorch model using EAIS.

  1. Remotely log on to the ECS instance.

    1. Log on to the EAIS console.

    2. In the top-left corner of the page, select the region where the instance is located.

    3. In the instance list, click the ID of the ECS instance that is attached to the EAIS instance to go to the ECS console.

    4. Remotely log on to the ECS instance.

      For more information, see Connect to an ECS instance.

  2. Install the eais-tool package and view information about the EAIS instance.

    For more information, see eais-tool.

  3. Install the eais-cuda package.

    For more information, see eais-cuda.

  4. Install the eais-torch package.

    For more information, see eais-torch.

  5. Deploy the application.

    Copy the compiled C++ inference application, its dependent library files, and the PyTorch model files from your development machine to the ECS instance.

  6. Infer the PyTorch model using EAIS.

    For example, if the name of your C++ program for PyTorch model inference is my_program and you run the program using the following command:

    export LD_LIBRARY_PATH=/your/lib/path:$LD_LIBRARY_PATH
./my_program my_model.pt

    To infer your PyTorch model using EAIS, run your C++ program using one of the following commands:

    • Command 1:

      export LD_LIBRARY_PATH=/your/lib/path:$LD_LIBRARY_PATH
LD_PRELOAD=libtorcheais.so ./my_program my_model.pt

    • Command 2:

      export LD_LIBRARY_PATH=/your/lib/path:$LD_LIBRARY_PATH
export LD_PRELOAD=libtorcheais.so
./my_program my_model.pt

Compile and run an inference program on an ECS instance that is attached to an EAIS instance

You can write, compile, and run an inference program directly on the ECS instance that is attached to the EAIS instance.

  1. Remotely log on to the ECS instance.

    1. Log on to the EAIS console.

    2. In the top-left corner of the page, select the region where the instance is located.

    3. In the instance list, click the ID of the ECS instance that is attached to the EAIS instance to go to the ECS console.

    4. Remotely log on to the ECS instance.

      For more information, see Connect to an ECS instance.

  2. Install the CUDA 11.X.X package.

    1. Run the following command to download and install the CUDA package.

      Note

      This section uses CUDA 11.7.0 as an example. The actual command varies based on the CUDA version that you install.

      wget https://developer.download.nvidia.com/compute/cuda/11.7.0/local_installers/cuda_11.7.0_515.43.04_linux.run
sudo sh cuda_11.7.0_515.43.04_linux.run --silent --toolkit

    2. Run the following command to set CUDA-related environment variables.

      export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

  3. Install the eais-tool package and view information about the EAIS instance.

    For more information, see eais-tool.

  4. Install the eais-cuda package.

    For more information, see eais-cuda.

  5. Install the eais-torch package.

    For more information, see eais-torch.

  6. Develop and compile the model inference code.

    1. Run the following command to download and decompress the package that contains the C++ sample code.

      wget https://aiacc-inference-public.oss-cn-beijing.aliyuncs.com/eais/packages/eais-miniconda.zip
unzip eais-miniconda.zip

    2. Develop and compile the model inference code.

      • You can refer to the sample code in the package to develop the model inference code.

        The C++ sample code is saved in the eais-miniconda/eais/cpp/src folder.

      • If you want to run the inference program using EAIS, you do not need to modify the code. You can proceed to the next step.

      Note

      The decompressed sample code does not depend on EAIS. The model inference code is written based on libtorch and CUDA. You can use the code directly or run the sample code on your GPU-accelerated instance.

  7. Infer the PyTorch model using EAIS.

    For example, if the name of your C++ program for PyTorch model inference is my_program and you run the program using the following command:

    export LD_LIBRARY_PATH=/your/lib/path:$LD_LIBRARY_PATH
./my_program my_model.pt

    To infer your PyTorch model using EAIS, run your C++ program using one of the following commands:

    • Command 1:

      export LD_LIBRARY_PATH=/your/lib/path:$LD_LIBRARY_PATH
LD_PRELOAD=libtorcheais.so ./my_program my_model.pt

    • Command 2:

      export LD_LIBRARY_PATH=/your/lib/path:$LD_LIBRARY_PATH
export LD_PRELOAD=libtorcheais.so
./my_program my_model.pt

Related operations

View and upgrade the glibc version

  1. Remotely log on to the ECS instance.

    1. Log on to the EAIS console.

    2. In the top-left corner of the page, select the region where the instance is located.

    3. In the instance list, click the ID of the ECS instance that is attached to the EAIS instance to go to the ECS console.

    4. Remotely log on to the ECS instance.

      For more information, see Connect to an ECS instance.

  2. Run the following command to view the glibc version.

    strings /lib64/libc.so.6 | grep GLIBC_

    The following command output is returned. Check whether the glibc version is 2.23 or later.glibc版本

  3. If the glibc version is earlier than 2.23, run the following command to upgrade the glibc version.

    mkdir glibc
pushd glibc
wget https://ftp.gnu.org/gnu/glibc/glibc-2.23.tar.gz
tar -xf glibc-2.23.tar.gz
cd glibc-2.23/
mkdir build
pushd build
../configure --prefix=/usr --disable-profile --enable-add-ons --with-headers=/usr/include --with-binutils=/usr/bin
make -j`nproc`
sudo make DESTDIR=/tmp/stage install
sudo make install
popd
popd
sudo rm -rf glibc

View and upgrade the libstdc++ version

  1. Remotely log on to the ECS instance.

    1. Log on to the EAIS console.

    2. In the top-left corner of the page, select the region where the instance is located.

    3. In the instance list, click the ID of the ECS instance that is attached to the EAIS instance to go to the ECS console.

    4. Remotely log on to the ECS instance.

      For more information, see Connect to an ECS instance.

  2. Run the following command to view the libstdc++ version.

    strings /usr/lib64/libstdc++.so.6 | grep GLIBCXX_

    The following command output is returned. Check whether the libstdc++ version is 3.4.25 or later.libstdc++版本

  3. If the libstdc++ version is earlier than 3.4.25, run the following command to upgrade the libstdc++ version.

    mkdir libstd
pushd libstd
wget http://ftp.de.debian.org/debian/pool/main/g/gcc-8/libstdc++6_8.3.0-6_amd64.deb
ar -x libstdc++6_8.3.0-6_amd64.deb
tar xf data.tar.xz
rm /usr/lib64/libstdc++.so.6
cp usr/lib/x86_64-linux-gnu/* /usr/lib64/
popd
sudo rm -rf libstd