Accelerate TensorFlow model inference with EAIS

Background information

This tutorial shows you how to create an eais.ei-a6.2xlarge EAIS instance in the China (Hangzhou) region. It uses the public ResNet-50, Inception, and BERT-Base models as examples to demonstrate performing inference on TensorFlow models with EAIS.

Step 1: Create and bind ECS and EAIS instances

1. Create an ECS instance.

   1. Log on to the ECS instance creation page.

   2. Create a virtual private cloud (VPC) ECS instance to associate with the EAIS instance.

      For more information, see Create an instance.

      Note

      Use Ubuntu 18.04 as the operating system for the ECS instance.

2. Create an EAIS instance.

   1. Log on to the EAIS console.

   2. In the left navigation pane, choose Elastic Accelerated Computing Instances EAIS > Elastic Accelerated Computing Instances List.

      Important

      If your Alibaba Cloud account has not been granted the AliyunServiceRoleForEais role, which uses the AliyunServiceRoleForEais access policy, you must first authorize EAIS to create a service-linked role before you can create an EAIS instance. For more information, see Service-linked Role.

   3. Click Create Elastic Accelerated Computing Instance.

   4. On the EAIS instance purchase page, configure the instance parameters and click Buy Now.

      The following table describes the configuration items:

      Configuration item	Example configuration
      Region	China (Hangzhou)
      Instance name	eais_test
      Instance type	eais.ei-a6.2xlarge
      Virtual Private Cloud	eais-vpc
      vSwitch	eais-vswitch
      Security group	eais-securitygroup
      Resource group	Default Resource Group

   5. On the Confirm Order page, agree to the Service Agreement by selecting the I have read and agree to the Elastic Accelerated Computing Instances Service Agreement checkbox, and then click Activate Now.

3. Bind the EAIS instance to the ECS instance.

   1. Click Console to go to the EAIS console.

      Service activation takes 5 to 10 minutes.

   2. In the instance list, find the eais_test instance and click Bind in the Actions column.

   3. In the dialog box that appears, select the ECS instance to bind.

   4. Click OK.

      The instance list automatically refreshes every 15 seconds. When the status of the EAIS instance changes from Binding to Bound, the binding is successful.

      Important

      An EAIS instance can be bound to only one ECS instance, and an ECS instance can be bound to only one EAIS instance. For more information, see Usage Limits.

Step 2: Start the EAIS instance

1. In the EAIS instance list, find the eais_test instance and click Start in the Actions column.

2. In the Start Instance dialog box, click OK.

   The EAIS instance list auto-refreshes every 15 seconds. When the instance status changes from Associated to In Use, the EAIS instance starts.

Step 3: Log on to the ECS instance

1. Log on to the EAIS console.

2. In the EAIS instance list, click the ID of the ECS instance that is associated with the eais_test instance to go to the details page of the ECS instance.

3. Remotely log on to the ECS instance.

   For more information, see Select a method for remote connection to an ECS instance.

Step 4: Configure the environment

1. Run the following command to upgrade pip to the latest version:

   python3 -m pip install --upgrade pip

2. Run the following command to install TensorFlow.

   In this example, TensorFlow 1.15.5 is used.

   pip3 install tensorflow==1.15.5

3. Run the following command to install eais_tensorflow:

   pip3 install eais_tensorflow -f https://aiacc-inference-public.oss-cn-beijing.aliyuncs.com/eais/packages/index.html

4. Run the following command to download the model package:

   wget https://aiacc-inference-public.oss-cn-beijing.aliyuncs.com/eais/packages/eais2_example.tar

5. Run the following command to decompress the package:

   tar xvf eais2_example.tar

(Optional) Step 5: View EAIS instance information

After you configure the environment, you can run the eais_smi command to query information about the EAIS instance, such as the instance type and GPU utilization. The following code provides a sample output:

root@ixxx:~# eais_smi
|-------------------------------+----------------------+----------------------|
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  eais.ei-a6.2xlarge  On   | 00000000:00:07.0 Off |                    0 |
| N/A   28C    P8    9W /  70W  |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

Step 6: Run TensorFlow model inference

1. Run the following commands to use EAIS to perform inference on the ResNet-50 model:

pushd eais2_example/resnet50
python3 resnet50.py
popd

2. The following output is returned:

   average time is  10.636587142944336  ms
   type: EGYPTIAN CAT

3. Run the following commands to use EAIS to perform inference on the Inception model.

   In this example, the Inception V4 model is used.

   pushd eais2_example/inception
   python3 inception.py
   popd

   The commands return the following inference result:

   average time is  25.059733390808105  ms
   Results: n02124075 Egyptian cat

4. Run the following commands to use EAIS to perform inference on the BERT-Base model:

pushd eais2_example/bert
python3 bert_infer.py
popd

5. The following output is returned:

   100 -- elapse time is: 0.009909868240356445 sec
   input text: The scenery is beautiful.  -> result class is: news_travel

(Optional) Step 7: Stop the EAIS instance

You can stop the EAIS instance if you do not temporarily need its GPU capabilities.

1. Log on to the EAIS console.

2. In the EAIS instance list, find the eais_test instance, and in its Operation column, click Stop.

3. In the Stop Instance dialog box, click OK.

   The EAIS instance list auto-refreshes every 15 seconds. A change in the instance status from In Use to Associated indicates that the EAIS instance has stopped.

(Optional) Step 8: Disassociate and release the EAIS instance

After inference is complete, you can disassociate the EAIS instance from the current ECS instance and associate it with another one. If you no longer need the EAIS instance, you must first disassociate it before you can release it. To disassociate and release an EAIS instance, perform the following steps:

1. Log on to the EAIS console.

2. Disassociate the EAIS instance from the ECS instance.

   1. In the EAIS instance list, find the eais_test instance and click Disassociate in the Operation column.

   2. In the Disassociate from ECS Instance dialog box, click OK.

      The EAIS instance list auto-refreshes every 15 seconds. When the instance status changes from Associated to Available, the EAIS instance is disassociated.

3. Release the disassociated EAIS instance.

   1. In the list of EAIS instances, find the eais_test instance and click Release Resource in the Operation column.

   2. In the Release Resource dialog box, click OK.

      The EAIS instance is deleted from the EAIS instance list.

Related documents

You can also learn the complete process of using EAIS to perform inference on a PyTorch model. For more information, see Accelerate PyTorch model inference by using EAIS (Python) or Accelerate PyTorch model inference by using EAIS (C++). For more information about how to use EAIS instances, see Use an EAIS instance or Tutorials.