Use EAIS to infer TensorFlow models

Prerequisites

• You have attached an EAIS instance to an ECS instance. For more information, see Attach an instance.

• The TensorFlow model file for inference is located in a folder on the attached ECS instance.

Limits

• Python version: 3.6 to 3.7.

• TensorFlow version: 1.15.0 to 1.15.5.

Procedure

1. Log on to the ECS instance.

   1. Log on to the EAIS console.

   2. In the upper-left corner of the page, select the region where the instance is located.

   3. In the instance list, click the ECS Instance ID that corresponds to the EAIS instance to open the ECS instance console.

   4. Remotely log on to the ECS instance.

      For more information, see Select a method to connect to an ECS instance.

2. Set up the runtime environment.

   1. Run the following command to upgrade pip to the latest version.

      python3 -m pip install --upgrade pip

   2. Run the following command to install TensorFlow.

      This example uses TensorFlow version 1.15.5.

      pip3 install tensorflow==1.15.5

   3. Run the following command to install EAIS TensorFlow.

      pip3 install eais_tensorflow -f https://aiacc-inference-public.oss-cn-beijing.aliyuncs.com/eais/packages/index.html

   4. Run the following command to download the model package.

      wget https://aiacc-inference-public.oss-cn-beijing.aliyuncs.com/eais/packages/eais2_example.tar

   5. Run the following command to decompress the package.

      tar xvf eais2_example.tar

3. (Optional) Run the following command to view information about the EAIS instance.

   eais_smi

   The command output shows information such as the EAIS instance type and EAIS GPU utilization.

4. Develop a model inference script and use EAIS for accelerated inference.

   • Python script development instructions

   Compared with a standard inference flow, you only need to add the line import eais_tensorflow to your original inference script before the inference process starts. This imports the EAIS Python module and enables you to use EAIS to infer the TensorFlow model.

   Suppose your original source code for TensorFlow model inference is as follows:

   # Import the tensorflow module
   import tensorflow as tf
   
   model_file = "xxx.pb"
   
   with tf.gfile.FastGFile(model_file, 'rb') as f:
       graph_def = tf.GraphDef()
       graph_def.ParseFromString(f.read())
       tf.import_graph_def(graph_def, name='')
   
   with tf.Session() as sess:
       result = sess.run(...)

   To use EAIS to infer your TensorFlow model, modify the source code as follows:

   # Import the tensorflow module
   import tensorflow as tf
   # Import the eais_tensorflow module
   import eais_tensorflow
   
   model_file = "xxx.pb"
   
   with tf.gfile.FastGFile(model_file, 'rb') as f:
       graph_def = tf.GraphDef()
       graph_def.ParseFromString(f.read())
       tf.import_graph_def(graph_def, name='')
   
   with tf.Session() as sess:
       result = sess.run(...)

   • Example

     1. Prepare the Python script for model inference.

        This example uses the resnet50.py script for ResNet-50 model inference. The script is as follows:

        import tensorflow as tf
        import eais_tensorflow
        from tensorflow.core.protobuf import config_pb2
        from tensorflow.core.protobuf import rewriter_config_pb2
        import numpy as np
        from PIL import Image
        
        img = 'cat.jpg'
        
        def load_graph(model_path):
            with tf.gfile.FastGFile(model_path, "rb") as f:
                graph_def = tf.GraphDef()
                graph_def.ParseFromString(f.read())
                g_in = tf.import_graph_def(graph_def, name="")
            return g_in
        
        if __name__ == "__main__":
            shape = [1, 299, 299, 3]
            image = Image.open(img)
            image = image.resize((shape[2],shape[1]))
        
            image_data = np.array(image,dtype='float32')
            image_data /= 255.
            image_data = np.expand_dims(image_data, 0)
            image_input = image_data.repeat(shape[0],axis=0)
        
            model_path='resnet_v2_50.pb'
            input_name = 'input'
            output_name = 'classes'
            config = config_pb2.ConfigProto()
            config.graph_options.rewrite_options.remapping = (
              rewriter_config_pb2.RewriterConfig.OFF)
            session = tf.Session(graph=load_graph(model_path),config=config)
            logits_tensor = session.graph.get_tensor_by_name(output_name + ':0')
        
            logits = session.run(logits_tensor, feed_dict={input_name + ':0': image_input})
            print(logits[0])

     2. Run the following command to execute the prepared EAIS model inference script.

        python3 resnet50.py