Deploy EAS services with TensorFlow Serving-Platform For AI(PAI)-阿里云帮助中心

TensorFlow Serving is an inference engine for deep learning models. It allows you to deploy models in the TensorFlow SavedModel format as online services and supports features such as hot-swapping and version management. By using the image-based deployment feature of Elastic Algorithm Service (EAS), you can quickly launch TensorFlow Serving inference services.

Prerequisites

Model files

To deploy a service using a TensorFlow Serving image, you must store the model files in OSS, and the directory structure must meet the following requirements:

Model version directories: Each model must have at least one version directory. Version directories must be named with integers, which serve as version numbers. A larger number indicates a newer version.
Model files: Each model version directory must contain the exported model files in the SavedModel format. The service automatically loads the model from the directory with the highest version number.

Procedure:

Create a directory in an OSS bucket to store your models, for example, oss://examplebucket/models/tf_serving/. For more information, see Manage directories.

Upload your model files to the directory that you created in the previous step. You can download and use the sample TensorFlow Serving model files for this article. The directory structure must be in the following format:

tf_serving 
├── modelA
│   └── 1
│       ├── saved_model.pb
│       └── variables
│           ├── variables.data-00000-of-00001
│           └── variables.index
│
├── modelB
│   ├── 1
│   │   └── ...
│   └── 2
│       └── ...
│
└── modelC
    ├── 1
    │   └── ...
    ├── 2
    │   └── ...
    └── 3
        └── ...

Model configuration file

You can use a configuration file to run multiple models within a single service. If you only need to deploy a single-model service, you can skip this section.

Create a configuration file as described below and upload it to OSS. The sample files in the Model files section include a model configuration file named model_config.pbtxt that you can use directly or modify as needed. In this example, the model configuration file is uploaded to the oss://examplebucket/models/tf_serving/ directory.

The following code shows an example of the model_config.pbtxt file:

model_config_list {
  config {
    name: 'modelA'
    base_path: '/models/modelA/'
    model_platform: 'tensorflow'
    model_version_policy{
        all: {}
    }
  }
  config {
    name: 'modelB'
    base_path: '/models/modelB/'
    model_platform: 'tensorflow'
    model_version_policy{
        specific {
            versions: 1
            versions: 2
        }
    }
    version_labels {
      	key: 'stable'
      	value: 1
    }
    version_labels {
      	key: 'canary'
      	value: 2
    }
  }
  config {
    name: 'modelC'
    base_path: '/models/modelC/'
    model_platform: 'tensorflow'
    model_version_policy{
        latest {
            num_versions: 2
        }
    }
  }
}

The following table describes the key parameters.

Parameter	Required	Description
name	No	The name of the model. We recommend that you set this parameter. If a name is not provided, you cannot call the inference service.
base_path	Yes	The path to the model storage directory in the instance, which is used to read model files during service deployment. For example, if the mount directory is `/models` and the model directory to load is `/models/modelA`, this parameter is set to `/models/modelA`.
model_version_policy	No	The policy for loading model versions. If this parameter is not set, the service loads the latest version of the model by default. all: {}: Loads all versions of the model. In the example, all versions of modelA are loaded. latest{}: In the example, modelC is configured with `num_versions: 2`, which loads the two latest versions: 2 and 3. specific: {}: Loads the specified versions. In the example, Version 1 and Version 2 of modelB are loaded.
version_labels	No	Configure custom labels for a model version. If version_labels is not configured, you can distinguish model versions only by their version numbers. The request path is: `/v1/models/<model name>/versions/<version number>:predict`. If `version_labels` are set, you can request `/v1/models/<model name>/labels/<version label>:predict` to target a specific version number by using its version label. Note By default, labels can only be assigned to model versions that have been successfully loaded and started as services. To pre-assign labels to model versions that have not yet been loaded, you need to set the startup parameter `--allow_version_labels_for_unavailable_models=true` in the Command to Run. Scenario-based deployment does not support setting the Command to Run. Select custom deployment instead.

Deploy service

You can deploy a TensorFlow Serving image in the following ways:

Scenario-based deployment: Suitable for basic deployment scenarios. You can deploy a TensorFlow Serving inference service by configuring only a few parameters.
Custom deployment: Allows you to flexibly adjust more configuration options, such as modifying the port or configuring the model file polling interval.

Important

TensorFlow Serving inference services support ports 8501 and 8500.

8501: Supports HTTP requests. An HTTP or REST service starts on port 8501.
8500: Supports gRPC requests. A gRPC service starts on port 8500.

The scenario-based deployment method uses port 8501 by default, which cannot be changed. To use port 8500, you must use the custom deployment method.

Scenario-based

Procedure:

Log on to the PAI console. Select a region on the top of the page. Then, select the desired workspace and click Elastic Algorithm Service (EAS).
On the Elastic Algorithm Service (EAS) page, click Deploy Service. In the Scenario-based Model Deployment section, click Tensorflow Serving Deployment.

On the Tensorflow Serving Deployment page, configure the parameters. The following table describes the key parameters. For information about other parameters, see custom deployment.

Parameter

Description

Deployment Method

The following two deployment methods are supported:

Standard Model Deployment: Used to deploy a single-model service.
Configuration File Deployment: Used to deploy a multi-model service.

Model Settings

If you select Standard Model Deployment for Deployment Method, you must configure the OSS path for the model file.

If you select Configuration File Deployment for Deployment Method, you must configure the following parameters:

OSS: Select the OSS path where the model files are located.
Mount Path: The path in the service instance where the model files are mounted.
Configuration File: Select the OSS path where the model configuration file is located.

The following table shows configuration examples.

Parameter	Single-model example	Multi-model example
Service Name	modela_scene	multi_scene
Deployment Method	Select Standard Model Deployment.	Select Configuration File Deployment.
Model Settings	OSS: `oss://examplebucket/models/tf_serving/modelA/`.	OSS: `oss://examplebucket/models/tf_serving/`. Mount Path: /models Configuration File: `oss://examplebucket/models/tf_serving/model_config.pbtxt`

After you configure the parameters, click Deploy.

Custom

Procedure:

Log on to the PAI console. Select a region on the top of the page. Then, select the desired workspace and click Elastic Algorithm Service (EAS).
Click Deploy Service. In the Custom Model Deployment section, click Custom Deployment.

On the Custom Deployment page, configure the parameters. The following table describes the key parameters. For information about other parameters, see custom deployment.

Parameter	Description
Image Configuration	In the Alibaba Cloud Image list, select tensorflow-serving and the corresponding image version. We recommend selecting the latest version. Note If the service requires a GPU, the image version must end with -gpu, for example, x.xx.x-gpu.
Model Settings	You can configure model files in various ways. This example uses the OSS type. Uri: Select the OSS path for the model files. Mount Path: The path in the service instance where the model files are mounted.
Command to Run	When you select the tensorflow-serving image, the command `/usr/bin/tf_serving_entrypoint.sh` is automatically loaded. The following describes the other startup parameters for tensorflow-serving that you need to set. Startup parameters for single-model deployment: `--model_name`: The name of the model used in the URL for service requests. If not specified, the default name is 'model'. `--model_base_path`: Specifies the path of the model storage directory in the instance. If you do not configure this parameter, the default path is `/models/model`. Startup parameters for multi-model deployment: `--model_config_file`: Required. Specifies the model configuration file. `--model_config_file_poll_wait_seconds`: Optional. If you want to modify the content of the model configuration file after the service starts, you can configure a polling interval in seconds. The service then reads the content of the model configuration file at this interval. For example, `--model_config_file_poll_wait_seconds=30` means the service reads the model configuration file content every 30 seconds. Note When the inference service reads a new model configuration file, it only processes the content of the new file. For example, if the old configuration file contains model A and the new configuration file removes model A and adds the configuration for model B, the service unloads model A and loads model B. `--allow_version_labels_for_unavailable_models`: Optional. The default value is false. If you want to assign labels in advance to model versions that are not yet available, set this parameter to true. For example, `--allow_version_labels_for_unavailable_models=true`.

The following table shows configuration examples.

Parameter	Single-model example	Multi-model example
Deployment Method	Select Image-based Deployment.
Image Configuration	Select Alibaba Cloud Image: tensorflow-serving > tensorflow-serving:2.14.1.
Model Settings	Set the model type to OSS. Uri: `oss://examplebucket/models/tf_serving/`. Mount Path: Set this to `/models`.
Command to Run	`/usr/bin/tf_serving_entrypoint.sh --model_name=modelA --model_base_path=/models/modelA`	`/usr/bin/tf_serving_entrypoint.sh --model_config_file=/models/model_config.pbtxt --model_config_file_poll_wait_seconds=30 --allow_version_labels_for_unavailable_models=true`

By default, the service listens on port 8501 for HTTP/REST requests. To enable gRPC requests, perform the following steps:

In the Environment Information section, change the Port Number to 8500.
In the Environment Information section, turn on the Enable gRPC switch.
In the Service Configurations section, add the following configuration:
```
"networking": {
    "path": "/"
}
```

Click Deploy.

Send service requests

Depending on the port number configured during deployment, the service supports both HTTP and gRPC request protocols. The following examples use modelA.

Prepare test data
modelA is an image classification model trained on the Fashion-MNIST dataset, which consists of 28x28 grayscale images. The model outputs the probability that a sample belongs to one of 10 classes. For testing purposes, [[[[1.0]] * 28] * 28] is used as the test data for service requests to modelA.

Request examples

HTTP requests

When the port number is configured to 8501, the service supports HTTP requests. The following table summarizes the HTTP request paths for single-model and multi-model deployments.

Single model

Multi-model

Path format: <service_url>/v1/models/<model_name>:predict

Where:

For scenario-based deployment, <model_name> cannot be customized and defaults to model.
For custom deployment, <model_name> is the model name configured in the Command to Run field. If not configured, it defaults to model.

Requests can be made with or without specifying a model version. The path formats are as follows:

Without specifying a version (loads the latest version by default):
<service_url>/v1/models/<model_name>:predict
Specifying a model version:
/v1/models//versions/:predict
If version_labels are set:
<service_url>/v1/models/<model_name>/labels/<version label>:predict

Here, <model_name> is the model name configured in the model configuration file.

In the path, <service_url> is the service endpoint. You can find the service endpoint by clicking Invocation Information in the Service Type column for the desired service on the Elastic Algorithm Service (EAS) page. When you use the online debugging feature in the console, this part of the path is automatically populated.

For a scenario-based deployment of a single model, such as modelA, the HTTP request path is: <service_url>/v1/models/model:predict.

The following sections provide details on how to send service requests using the online debugging feature in the console and via Python code.

Online debugging

After the service is deployed, click Online Debugging in the Actions column of the service. In the Request Parameter Online Tuning section, the <service_url> is already populated. Add the path /v1/models/model:predict to the URL, and configure the service request data in the Body:

{"signature_name": "serving_default", "instances": [[[[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0],-1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0],-1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0],-1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.e-01]]]}

After you configure the parameters, click Send Request. The output is similar to the following.

Status Code: 200
Connection: Keep-Alive
Content-Length: 158
Content-Type: application/json
Date: Fri, 08 Nov 2024 02:28:08 GMT
Body: {
    "predictions": [[-9.40927601, -5.41267443, -15.9157038, -15.7119455, -16.194952, -42.5621414, -7.23207045, -43.1042442, 4.25585461, -31.3991375]
    ]
}

Send an HTTP request using Python code

The following is a Python code example:

from urllib import request
import json
# Replace <service_url> and <test-token> with your service endpoint and token.
# To obtain them, click Invocation Information for your inference service and then go to the Public Endpoint Invocation tab.
service_url = '<service_url>'
token = '<test-token>'
# For a scenario-specific single-model deployment, set this to "model". For other cases, refer to the path description table above.
model_name = "model"
url = "{}/v1/models/{}:predict".format(service_url, model_name)
# Create an HTTP request.
req = request.Request(url, method="POST")
req.add_header('authorization', token)
data = {
    'signature_name': 'serving_default',
    'instances': [[[[1.0]] * 28] * 28]
}
# Send the request.
response = request.urlopen(req, data=json.dumps(data).encode('utf-8')).read()
# Check the response.
response = json.loads(response)
print(response)

gRPC requests

After you set the port number to 8500 and add gRPC configurations, the service supports sending gRPC requests. The following is a Python code example:

import grpc
import tensorflow as tf
from tensorflow_serving.apis import predict_pb2
from tensorflow_serving.apis import prediction_service_pb2_grpc
from tensorflow.core.framework import tensor_shape_pb2
# The service endpoint. For the format, see the description of the host parameter below.
host = "tf-serving-multi-grpc-test.166233998075****.cn-hangzhou.pai-eas.aliyuncs.com:80"
# Replace <test-token> with the service token. You can view the token on the Invocation Information page.
token = "<test-token>"
# The model name. See the description of the name parameter below.
name = "<model_name>"
signature_name = "serving_default"
# The model version number. You can send requests to only one model version at a time.
version = "<version_num>"
# Create a gRPC request.
request = predict_pb2.PredictRequest()
request.model_spec.name = name
request.model_spec.signature_name = signature_name
request.model_spec.version.value = version
request.inputs["keras_tensor"].CopyFrom(tf.make_tensor_proto([[[[1.0]] * 28] * 28]))
# Request the service.
channel = grpc.insecure_channel(host)
stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)
metadata = (("authorization", token),)
response, _ = stub.Predict.with_call(request, metadata=metadata)
print(response)

The key parameters are described as follows:

Parameter	Description
host	Set this parameter to the service endpoint. Omit `http://` from the endpoint and append `:80` to the end. On the Elastic Algorithm Service (EAS) page, click Invocation Information in the Service Type column of the target service to view the service endpoint.
name	For gRPC requests to a single model: Scenario-based deployment: Set this to `model`. Custom deployment: Set this to the model name that is configured in the Command to Run. If no model name is configured, the default value is `model`. For gRPC requests to multiple models: Set this to the model name that is configured in the model configuration file.
version	Set this to the model version number. You can send requests to only one model version at a time.
metadata	Set this to the service token. You can view the token on the Invocation Information page.

Deploy services with TensorFlow Serving images

Prerequisites

Model files

Model configuration file

Deploy service

Scenario-based

Custom

Send service requests

HTTP requests

Online debugging

Send an HTTP request using Python code

gRPC requests

Related documents