Deploy inference services

Deploy from a pre-built image

Package your inference code and dependencies into a Docker image, push it to an ACR repository, then build an InferenceSpec from the image URI:

from pai.model import InferenceSpec, Model, container_serving_spec

container_infer_spec = container_serving_spec(
    # URI of your custom Docker image.
    image_uri="<your-custom-image-uri>",
    # Port the service listens on inside the container. PAI routes prediction requests to this port.
    port=8000,
    environment_variables=environment_variables,
    # Command to start the inference service.
    command=command,
    # Additional Python packages to install at startup.
    requirements=[
        "scikit-learn",
        "fastapi==0.87.0",
    ],
)

m = Model(
    model_data="oss://<your-oss-bucket>/path-to-tensorflow-saved-model",
    inference_spec=container_infer_spec,
)
p = m.deploy(instance_type="ecs.c6.xlarge")

Deploy from local code without building an image

The SDK can package and upload a local code directory automatically — no manual image build required. Specify source_dir in container_serving_spec(). The SDK uploads that directory to an OSS bucket and mounts it to /ml/usercode/ in the running container. The start command runs with /ml/usercode as the working directory.

from pai.model import InferenceSpec, container_serving_spec

inference_spec = container_serving_spec(
    # Local directory containing your inference code.
    # The SDK uploads it to OSS and mounts it at /ml/usercode/ in the container.
    source_dir="./src",
    # Runs with /ml/usercode as the working directory.
    command="python run.py",
    image_uri="<serving-image-uri>",
    requirements=[
        "fastapi",
        "uvicorn",
    ],
)

Mount additional data to the container

To load tokenizers, lookup tables, or other artifacts into the container at runtime, call InferenceSpec.mount():

# Upload a local directory to OSS and mount it at /ml/tokenizers/.
inference_spec.mount("./bert_tokenizers/", "/ml/tokenizers/")

# Mount data already stored in OSS to /ml/data/.
inference_spec.mount("oss://<your-oss-bucket>/path/to/data/", "/ml/data/")

Retrieve PAI-provided runtime images

PAI publishes inference runtime images for TensorFlow, PyTorch, and XGBoost. Pass image_scope=ImageScope.INFERENCE to list_images() or retrieve() to get them:

from pai.image import retrieve, ImageScope, list_images

# List all PAI PyTorch inference images.
for image_info in list_images(framework_name="PyTorch", image_scope=ImageScope.INFERENCE):
    print(image_info)

# Get a specific version (CPU).
retrieve(framework_name="PyTorch", framework_version="1.12", image_scope=ImageScope.INFERENCE)

# Get a GPU-accelerated image.
retrieve(framework_name="PyTorch", framework_version="1.12",
    accelerator_type="GPU", image_scope=ImageScope.INFERENCE)

# Get the latest GPU image.
retrieve(framework_name="PyTorch", framework_version="latest",
    accelerator_type="GPU", image_scope=ImageScope.INFERENCE)

JsonSerializer

JsonSerializer serializes a numpy.ndarray or list to a JSON string and deserializes the response back to a Python object. XGBoost and PMML processors use JsonSerializer by default.

from pai.serializers import JsonSerializer

# Deploy with an explicit serializer (optional for XGBoost — it's the default).
p = Model(
    inference_spec=InferenceSpec(processor="xgboost"),
    model_data="oss://<your-oss-bucket>/path-to-xgboost-model"
).deploy(
    instance_type="ecs.c6.xlarge",
    serializer=JsonSerializer(),
)

# Or attach a serializer to an existing Predictor.
p = Predictor(
    service_name="example_xgb_service",
    serializer=JsonSerializer(),
)

result = p.predict([[2, 3, 4], [4, 5, 6]])

TensorFlowSerializer

Use the PAI TensorFlow processor to deploy a TensorFlow SavedModel. The service exchanges data using Protocol Buffers. TensorFlowSerializer converts numpy.ndarray inputs to Protocol Buffers messages and deserializes responses back to numpy.ndarray. See tf_predict.proto for the message format.

The TensorFlow processor uses TensorFlowSerializer by default.

tf_predictor = Model(
    inference_spec=InferenceSpec(processor="tensorflow_cpu_2.7"),
    model_data="oss://<your-oss-bucket>/path-to-tensorflow-saved-model"
).deploy(
    instance_type="ecs.c6.xlarge",
)

# Inspect the model's input/output signature.
print(tf_predictor.inspect_signature_def())

# Input: a dict where each key is an input signature name and each value is a numpy.ndarray.
tf_result = tf_predictor.predict(data={
    "flatten_input": numpy.zeros(28 * 28 * 2).reshape((-1, 28, 28))
})

assert tf_result["dense_1"].shape == (2, 10)

PyTorchSerializer

Use the PAI PyTorch processor to deploy a model in TorchScript format. The service uses Protocol Buffers for data exchange. PyTorchSerializer converts numpy.ndarray inputs to Protocol Buffers messages and deserializes responses back to numpy.ndarray. See pytorch_predict_proto for the message format.

The PyTorch processor uses PyTorchSerializer by default.

torch_predictor = Model(
    inference_spec=InferenceSpec(processor="pytorch_cpu_1.10"),
    model_data="oss://<your-oss-bucket>/path-to-torchscript-model"
).deploy(
    instance_type="ecs.c6.xlarge",
)

# For multiple inputs, pass a list or tuple of numpy.ndarray objects.
torch_result = torch_predictor.predict(
    data=numpy.zeros(28 * 28 * 2).reshape((-1, 28, 28))
)
assert torch_result.shape == (2, 10)

Custom serializer

For services with non-standard data formats, implement a custom Serializer by inheriting from pai.serializers.SerializerBase and implementing serialize and deserialize.

The following example shows a NumpySerializer that exchanges data in NumPy's .npy binary format:

1. predict() receives a numpy.ndarray or pandas.DataFrame. NumpySerializer.serialize() converts it to .npy format and sends it in the request body.

2. The server deserializes the .npy data, runs inference, and returns the result in .npy format.

3. NumpySerializer.deserialize() converts the response back to a numpy.ndarray.