DocsICP FilingConsole
官方文档
首页

Infer a PyTorch model with EAIS (Python)

更新时间：
复制 MD 格式
我的收藏

After you attach an EAIS instance to an ECS instance, you can remotely log on to the ECS instance to perform AI inference. This topic describes how to use a Python script to perform inference on a PyTorch model with EAIS.

Prerequisites

Background information

EAIS lets you use different programming languages to perform inference on PyTorch models. The following two methods are supported:

  • Perform inference on a PyTorch model using a Python script. You can use PyTorch script mode or PyTorch eager mode for inference. This topic describes the procedure for this method.

  • Perform inference on a PyTorch model using a C++ program. For more information, see Infer a PyTorch model with EAIS (C++).

If you encounter performance or feature issues during EAIS inference, contact EAIS technical support for customized optimization solutions.

Runtime environment

This topic describes how to perform inference on a PyTorch model using a Python script. EAIS provides the following two methods to deploy the runtime environment:

  • Use the EAIS miniconda environment that is provided by EAIS.

    Note

    The EAIS miniconda environment is an EAIS environment that Alibaba Cloud developed based on miniconda. This environment includes the software required to perform inference on PyTorch models with EAIS. It also includes sample files and model files for testing. The following list describes some of the file folders:

    • eais/data/: Stores the data files and model files required to run the sample program.

    • eais/python/: Stores the Python scripts required for the inference samples.

  • Install the Python package that is provided by EAIS in your existing PyTorch model runtime environment.

Inference performance

Using EAIS for inference provides significantly improved performance compared with a GPU-accelerated instance (NVIDIA T4). The following table compares the inference performance of a Python script on an `eais.ei-a6.2xlarge` EAIS instance and on a GPU-accelerated instance (NVIDIA T4).

Note

  • The data in this topic is for reference only. Actual performance may vary based on your inference results.

  • You can also use the Python script provided in the eais-miniconda package to test the inference performance of a GPU-accelerated instance (NVIDIA T4) and compare the performance with that of an EAIS instance.

Reasoning model

eais.ei-a6.2xlarge

GPU-accelerated instance (NVIDIA T4)

Performance improvement of EAIS over the GPU-accelerated instance (NVIDIA T4)

resnet50

2.19 ms

6.24 ms

2.85 times

bert-base

5.37 ms

8.32 ms

1.55 times

Procedure

  1. Log on to the instance.

    1. Log on to the EAIS console.

    2. In the upper-left corner of the page, select the region where the instance is located.

    3. In the instance list, click the ECS Instance ID of the EAIS instance to go to the ECS console.

    4. Remotely log on to the ECS instance.

      For more information, see Select a method to connect to an ECS instance.

  2. Install the eais-tool package and view information about the EAIS instance.

    For more information, see eais-tool.

  3. Install a CUDA 11.x package.

    1. Run the following command to install the CUDA package.

      Note

      This topic uses CUDA 11.7.0 as an example. The command varies based on the version that you install.

      wget https://developer.download.nvidia.com/compute/cuda/11.7.0/local_installers/cuda_11.7.0_515.43.04_linux.run
sudo sh cuda_11.7.0_515.43.04_linux.run --silent --toolkit

    2. Run the following command to set the CUDA environment variables.

      export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

  4. Install the eais-cuda package.

    For more information, see eais-cuda.

  5. Set up the runtime environment.

    • Method 1: Use the EAIS miniconda environment that is provided by EAIS.

      Install the eais-miniconda package. For more information, see eais-miniconda.

    • Method 2: Install the Python package that is provided by EAIS in your existing PyTorch model runtime environment.

      Important

      Make sure that Python 3 and the pip3 package are installed in your runtime environment.

      1. Run the following command to install the official PyTorch package. This topic uses PyTorch 1.13.1 as an example.

        pip3 install torch==1.13.1

      2. Install the eais-torch package.

        For more information, see eais-torch.

  6. Develop a model inference script and use EAIS to accelerate inference.

    Compared with the standard inference flow, you only need to add the line import eais.torch_eais to your original script before inference. This imports the Python module that is provided by EAIS and lets you use EAIS to perform inference on the PyTorch model. You can use PyTorch script mode or PyTorch eager mode for inference. The following sections provide instructions and examples for script development:

    Infer using PyTorch script mode

    • Python script development instructions

      Assume that your original source code for PyTorch model inference is as follows:

      # Import the torch module
import torch
import torchvision

# Load the script model
model = torch.jit.load(model_file).cuda()
# Initialize the input
tensor input_tensor = torch.randn(...).cuda()
# Use the GPU for model inference
output_tensors = model(input_tensor)

      To use EAIS to perform inference on your PyTorch model, modify the source code as follows:

      # Import the torch module
import torch
import torchvision
# Import the Python module provided by EAIS
import eais.torch_eais

# Load the script model
model = torch.jit.load(model_file).cuda()
# Initialize the input
tensor input_tensor = torch.randn(...).cuda()
# Use EAIS for model inference
output_tensors = model(input_tensor)

    • Example

      1. Prepare your Python script for model inference.

        This example uses the pytorch_resnet50.py script to perform inference on the resnet50 model. The script is as follows:

        # Import the torch module
import torch
# Import the Python module provided by EAIS
import eais.torch_eais
# Import the parameter parsing module
import argparse

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('-m', '--model', type=str, required=True, help='model file path')
    FLAGS = parser.parse_args()
    
    # Load the script model
    model = torch.jit.load(FLAGS.model).cuda()
    # Initialize a random input tensor
    input_tensor = torch.rand((1, 3, 224, 224)).cuda()
    # Execute model inference
    output = model(input_tensor).cpu()
    print("output shape:", output.shape)

      2. Run the following command to run the EAIS model inference script.

        python3 pytorch_resnet50.py -m <resnet50_model_file>

    Infer using PyTorch eager mode

    • Python script development instructions

      Assume that your original source code for PyTorch model inference is as follows:

      # Import the torch module
import torch
import torchvision


class MyModule(torch.nn.Module):
    def __init__(self):
        super(MyModule, self).__init__()
        ......

    def forward(self, x):
        ......


# Initialize the torch model
model = MyModule.cuda()
# Initialize the input
tensor input_tensor = torch.randn(...).cuda()
# Use the GPU for model inference
output_tensors = model(input_tensor)

      To use EAIS to perform inference on your PyTorch model, modify the source code as follows:

      Important

      The forward function in the source code cannot contain python numpy operations or custom python functions.

      # Import the torch module
import torch
import torchvision
# Import the Python module provided by EAIS
import eais.torch_eais


class MyModule(torch.nn.Module):
    def __init__(self):
        super(MyModule, self).__init__()
        ......

    def forward(self, x):
        ......


# Initialize the torch model
model = MyModule().cuda()
# Initialize the input
tensor input_tensor = torch.randn(...).cuda()
# Use EAIS for model inference
output_tensors = model(input_tensor)

    • Example

      1. Prepare your Python script for model inference.

        This example uses the pytorch_resnet50.py script to perform inference on the resnet50 model. The script is as follows:

        # Import the torch module
import torch
import torchvision
# Import the Python module provided by EAIS
import eais.torch_eais

if __name__ == '__main__':
    # Initialize the resnet50 torch model
    model = torchvision.models.resnet50(pretrained=True).cuda()
    # Initialize a random input tensor
    input_tensor = torch.rand((1, 3, 224, 224)).cuda()
    # Execute model inference
    output = model(input_tensor).cpu()
    print("output shape:", output.shape)

      2. Run the following command to run the EAIS model inference script.

        python3 pytorch_resnet50.py