Deploy an MLLM application in EAS-Platform For AI(PAI)-阿里云帮助中心

Elastic Algorithm Service (EAS) lets you deploy an MLLM inference service with a single click in five minutes. This topic describes how to deploy and call the MLLM inference service.

Background

In recent years, large language models (LLMs) have achieved unprecedented success in language tasks. They excel not only at generating natural language text but also demonstrate powerful capabilities in multitask scenarios such as sentiment analysis, machine translation, and text summarization. However, these models are limited to text and cannot process other data modalities, such as images, audio, or video.

This has led to a surge in research on multimodal large language models (MLLMs). With the widespread industry adoption of models like GPT-4o, MLLMs have become a popular application.

EAS provides a one-click solution to automate MLLM deployment. You can deploy a popular MLLM inference service in just five minutes.

Prerequisites

You have activated PAI and created a default workspace. For more information, see Activate PAI and create a default workspace.
If you use a RAM user to deploy the model, you must grant the RAM user the required permissions to manage EAS. For more information, see Cloud product dependencies and authorization: EAS.

Deploy the EAS service

Log on to the PAI console. Select a region on the top of the page. Then, select the desired workspace and click Elastic Algorithm Service (EAS).
Click Deploy Service. In the Custom Model Deployment section, click Custom Deployment.

On the Custom Deployment page, configure the following key parameters. For information about other parameters, see Custom Deployment.

Parameter		Description
Environment Information	Deployment Method	Select Image-based Deployment and select the Enable Web App check box.
	Image Configuration	From the list of official images, select chat-mllm-webui > chat-mllm-webui:1.0. Note Because versions are updated frequently, we recommend selecting the latest image version.
	Command to Run	After you select an image, the command is automatically configured. You can modify the `model_type` to deploy different models. For a list of supported models, see the Models table below.
Resource Information	Deployment	Select a GPU-accelerated instance type. We recommend using ml.gu7i.c16m60.1-gu30 for the best cost-effectiveness.

Models

model_type	Model link
qwen_vl_chat	qwen/Qwen-VL-Chat
qwen_vl_chat_int4	qwen/Qwen-VL-Chat-Int4
qwen_vl	qwen/Qwen-VL
glm4v_9b_chat	ZhipuAI/glm-4v-9b
llava1_5-7b-instruct	swift/llava-1___5-7b-hf
llava1_5-13b-instruct	swift/llava-1___5-13b-hf
internvl_chat_v1_5_int8	AI-ModelScope/InternVL-Chat-V1-5-int8
internvl-chat-v1_5	AI-ModelScope/InternVL-Chat-V1-5
mini-internvl-chat-2b-v1_5	OpenGVLab/Mini-InternVL-Chat-2B-V1-5
mini-internvl-chat-4b-v1_5	OpenGVLab/Mini-InternVL-Chat-4B-V1-5
internvl2-2b	OpenGVLab/InternVL2-2B
internvl2-4b	OpenGVLab/InternVL2-4B
internvl2-8b	OpenGVLab/InternVL2-8B
internvl2-26b	OpenGVLab/InternVL2-26B
internvl2-40b	OpenGVLab/InternVL2-40B

After you configure the parameters, click Deploy.

Call the service

Use the WebUI for model inference

On the Elastic Algorithm Service (EAS) page, click the name of the target service. In the upper-right corner of the page, click View Web App to open the WebUI.
On the WebUI page, test the model inference. You can optionally upload an image in the image upload area on the left. Enter your question in the text box at the bottom, and then click Send to generate a response.

Use the API for model inference

Get the service endpoint and token.
1. On the Elastic Algorithm Service (EAS) page, click the name of the target service. In the Basic Information section, click View Endpoint Information.
2. In the Invocation Method panel, get the service token and endpoint.

Use the API for model inference.

PAI provides the following three API operations:

Infer forward

Retrieves the inference result.

Note

The WebUI and API are mutually exclusive. If you have used the WebUI, you must call the clear chat history operation before calling infer forward.

Replace the following key parameters in the sample code:

Parameter	Description
hosts	The service endpoint obtained in Step 1.
authorization	The service token obtained in Step 1.
prompt	The content of your prompt. We recommend using English.
image_path	The local path to the image.

Request and response parameters

The following table lists the request parameters.

Parameter	Type	Description	Default
prompt	String	The content of the prompt.	None. Required.
image	Base64-encoded	The input image.	None
chat_history	List[List]	The chat history.	[]
temperature	Float	Controls the randomness of the model's output. A higher value increases randomness, while a value of 0 produces deterministic output. Valid values range from 0 to 1.	0.2
top_p	Float	Controls diversity by selecting from the most likely tokens whose cumulative probability exceeds the top_p value.	0.7
max_output_tokens	Int	The maximum length of the output, in tokens.	512
use_stream	Bool	Specifies whether to use streaming output: True False	True

The output is the response to the prompt, returned as a string.

The following sample code shows how to call the API in Python:

import requests
import json
import base64
def post_get_history(url='http://127.0.0.1:7860', headers=None):
    r = requests.post(f'{url}/get_history', headers=headers, timeout=1500)
    data = r.content.decode('utf-8')
    return data
def post_infer(prompt, image=None, chat_history=[], temperature=0.2, top_p=0.7, max_output_tokens=512, use_stream = True, url='http://127.0.0.1:7860', headers={}):
    datas = {
        "prompt": prompt,
        "image": image,
        "chat_history": chat_history,
        "temperature": temperature,
        "top_p": top_p,
        "max_output_tokens": max_output_tokens,
        "use_stream": use_stream,
    }
    if use_stream:
        headers.update({'Accept': 'text/event-stream'})
        response = requests.post(f'{url}/infer_forward', json=datas, headers=headers, stream=True, timeout=1500)
        if response.status_code != 200:
            print(f"Request failed with status code {response.status_code}")
            return
        process_stream(response)
    else:
        r = requests.post(f'{url}/infer_forward', json=datas, headers=headers, timeout=1500)
        data = r.content.decode('utf-8')
        print(data)
def image_to_base64(image_path):
    """
    Convert an image file to a Base64 encoded string.
    :param image_path: The local path to the image.
    :return: A Base64 encoded string representation of the image.
    """
    with open(image_path, "rb") as image_file:
        # Read the binary data of the image
        image_data = image_file.read()
        # Encode the binary data to Base64
        base64_encoded_data = base64.b64encode(image_data)
        # Convert bytes to string and remove any trailing newline characters
        base64_string = base64_encoded_data.decode('utf-8').replace('\n', '')
    return base64_string
def process_stream(response, previous_text=""):
    MARK_RESPONSE_END = '##END'  # DO NOT CHANGE
    buffer = previous_text
    current_response = ""
    for chunk in response.iter_content(chunk_size=100):
        if chunk:
            text = chunk.decode('utf-8')
            current_response += text
            parts = current_response.split(MARK_RESPONSE_END)
            for part in parts[:-1]:
                new_part = part[len(previous_text):]
                if new_part:
                    print(new_part, end='', flush=True)
                previous_text = part
            current_response = parts[-1]
    remaining_new_text = current_response[len(previous_text):]
    if remaining_new_text:
        print(remaining_new_text, end='', flush=True)
if __name__ == '__main__':
    # Replace <service_url> with the service endpoint.
    hosts = '<service_url>'
    # Replace <token> with the service token.
    head = {
        'Authorization': '<token>'
    }
    # get chat history
    chat_history = json.loads(post_get_history(url=hosts, headers=head))['chat_history']
    # The content of your prompt. We recommend using English.
    prompt = 'Please describe the image'
    # Replace path_to_your_image with the local path of your image.
    image_path = 'path_to_your_image'
    image_base_64 = image_to_base64(image_path)
    post_infer(prompt = prompt, image = image_base_64, chat_history = chat_history, use_stream=False, url=hosts, headers=head)

Get chat history

Retrieves the chat history.

Replace the following key parameters in the sample code:

Parameter	Description
hosts	The service endpoint obtained in Step 1.
authorization	The service token obtained in Step 1.

No input parameters are required.

The following table lists the output parameters.

Parameter	Type	Description
chat_history	List[List]	The chat history.

The following sample code shows how to call the API in Python:

import requests
import json
def post_get_history(url='http://127.0.0.1:7860', headers=None):
    r = requests.post(f'{url}/get_history', headers=headers, timeout=1500)
    data = r.content.decode('utf-8')
    return data
if __name__ == '__main__':
    # Replace <service_url> with the service endpoint.
    hosts = '<service_url>'
    # Replace <token> with the service token.
    head = {
        'Authorization': '<token>'
    }
    chat_history = json.loads(post_get_history(url=hosts, headers=head))['chat_history']
    print(chat_history)

Clear chat history

Clears the chat history.

Replace the following key parameters in the sample code:

Parameter	Description
hosts	The service endpoint obtained in Step 1.
authorization	The service token obtained in Step 1.

No input parameters are required.
Returns the string success upon completion.

The following sample code shows how to call the API in Python:

import requests
import json
def post_clear_history(url='http://127.0.0.1:7860', headers=None):
    r = requests.post(f'{url}/clear_history', headers=headers, timeout=1500)
    data = r.content.decode('utf-8')
    return data
if __name__ == '__main__':
    # Replace <service_url> with the service endpoint.
    hosts = '<service_url>'
    # Replace <token> with the service token.
    head = {
        'Authorization': '<token>'
    }
    clear_info = post_clear_history(url=hosts, headers=head)
    print(clear_info)