Deploy large language models on PAI-EAS

cURL

curl -X POST <EAS_ENDPOINT>/v1/chat/completions \
    -H "Content-Type: application/json" \
    -H "Authorization: <EAS_TOKEN>" \
    -d '{
        "model": "<model_name>",
        "messages": [
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": "hello"
        }
        ],
        "max_tokens":1024,
        "temperature": 0.7,
        "top_p": 0.8,
        "stream":true
    }'

In the preceding code, replace the following parameters:

• Replace <EAS_ENDPOINT> and <EAS_TOKEN> with the endpoint and token of your deployed service.

• Replace <model_name> with the name of the model. For vLLM or SGLang, you can get the model name from the model list API at <EAS_ENDPOINT>/v1/models.

  curl -X GET <EAS_ENDPOINT>/v1/models -H "Authorization: <EAS_TOKEN>"

OpenAI SDK

We recommend using the official Python SDK to interact with the service. Make sure that you have installed the OpenAI SDK: pip install openai.

from openai import OpenAI

# 1. Configure the client.
# Replace <EAS_TOKEN> with the token of your deployed service.
openai_api_key = "<EAS_TOKEN>"
# Replace <EAS_ENDPOINT> with the endpoint of your deployed service.
openai_api_base = "<EAS_ENDPOINT>/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

# 2. Get the model name.
# For BladeLLM, set model = "". BladeLLM does not require a model parameter
# and does not support getting the model name by using client.models.list().
# Set it to an empty string to meet the mandatory parameter requirement of the OpenAI SDK.
models = client.models.list()
model = models.data[0].id
print(model)

# 3. Send a chat request.
# Both streaming (stream=True) and non-streaming (stream=False) outputs are supported.
stream = True
chat_completion = client.chat.completions.create(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "hello"},          
    ],
    model=model,
    top_p=0.8,
    temperature=0.7,
    max_tokens=1024,
    stream=stream,
)

if stream:
    for chunk in chat_completion:
        print(chunk.choices[0].delta.content, end="")
else:
    result = chat_completion.choices[0].message.content
    print(result)

Python requests

If you do not want to add a dependency on the OpenAI SDK, you can use the requests library.

import json
import requests

# Replace <EAS_ENDPOINT> with the endpoint of your deployed service.
EAS_ENDPOINT = "<EAS_ENDPOINT>"
# Replace <EAS_TOKEN> with the token of your deployed service.
EAS_TOKEN = "<EAS_TOKEN>"
# Replace <model_name> with the model name. You can get it from the <EAS_ENDPOINT>/v1/models API.
# For BladeLLM, this API is not supported. You can omit the "model" field or set it to "".
model = "<model_name>"

url = f"{EAS_ENDPOINT}/v1/chat/completions"
headers = {
    "Content-Type": "application/json",
    "Authorization": EAS_TOKEN,
}

stream = True
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "hello"},
]

req = {
    "messages": messages,
    "stream": stream,
    "temperature": 0.7,
    "top_p": 0.8,
    "max_tokens": 1024,
    "model": model,
}
response = requests.post(
    url,
    json=req,
    headers=headers,
    stream=stream,
)

if stream:
    for chunk in response.iter_lines(chunk_size=8192, decode_unicode=False):
        msg = chunk.decode("utf-8")
        # The following code processes Server-Sent Events (SSE) formatted streaming responses.
        if msg.startswith("data:"):
            info = msg[6:]
            if info == "[DONE]":
                break
            else:
                resp = json.loads(info)
                if resp["choices"][0]["delta"].get("content") is not None:
                    print(resp["choices"][0]["delta"]["content"], end="", flush=True)
else:
    resp = json.loads(response.text)
    print(resp["choices"][0]["message"]["content"])

Dify

1. Install the OpenAI-API-compatible model provider

   Click your avatar in the upper-right corner and select Settings. In the left pane, select Model Providers. If OpenAI-API-compatible is not in the Model List, find it in the list below and click Install.

2. Add a model

   Click Add Model in the lower-right corner of the OpenAI-API-compatible card and configure the following parameters:

   • Model type: Select LLM.

   • Model name: For a vLLM deployment, send a GET request to the /v1/models endpoint to get the name. For this example, enter Qwen3-8B.

   • API key: Enter your PAI-EAS service token.

   • API endpoint URL: Enter the public endpoint of your PAI-EAS service and append /v1 to the end.

3. Test the integration

   1. On the Dify main page, click Create Blank App, select the Chatflow type, enter an application name and other information, and then click Create.

   2. Click the LLM node, select the model that you added, and set the context and prompts. Select Qwen3-8B CHAT from the OpenAI-API-compatible category. In the SYSTEM prompt, enter a role description and reference the context variable sys.query. In the USER area, add the sys.query and sys.files variables. You can adjust model parameters such as Max Tokens (default is 512), Presence Penalty, and Thinking Mode as needed.

   3. Click Preview in the upper-right corner and enter a question.

      The bot processes the question through the workflow and returns a reply. The top of the conversation area displays a Deep thinking status and the elapsed time.

Chatbox

1. Go to Chatbox to download and install the client for your device, or click Launch Web App for the web version. This example uses macOS M3.

2. Add a model provider. Click Settings, add a model provider, enter a name such as pai, and select OpenAI API Compatible as the API mode.

3. Select the pai provider and configure the following parameters:

   • API key: Enter your PAI-EAS service token.

   • API host: Enter the public endpoint of your PAI-EAS service and append /v1 to the end.

   • API path: Leave this field empty.

   • Model: Click Get to add the model. The BladeLLM inference engine does not support API-based model retrieval. If you are using BladeLLM, add the model manually by clicking New.

4. Test the chat. Click New Chat and select the model service in the lower-right corner of the text input box.

   The top of the chat interface displays the system prompt, such as You are a helpful assistant.. Enter a question such as "Who are you?" in the text box and send it. The assistant's reply area shows a collapsed Deep thinking section with the thinking time, followed by the model's response.

Cherry Studio

1. Install the client

   Visit Cherry Studio to download and install the client.

   You can also download it from https://github.com/CherryHQ/cherry-studio/releases.

2. Configure the model service

   1. Click the settings icon in the lower-left corner. Under the Model Service section, click Add. In Provider name, enter a custom name, such as PAI, and set the provider type to OpenAI. Click OK.

   2. In the API key field, enter your PAI-EAS service token. In the API address field, enter the public endpoint of your PAI-EAS service.

   3. Click Add and enter the model name in the Model ID field. For vLLM deployments, you can obtain the name by sending a GET request to the /v1/models API. For example, enter Qwen3-8B. The name is case-sensitive. For the API Address, enter the URL but omit the v1 version number at the end. If the URL ends with a # character, the system forces the use of the /v1/chat/completions path. After the configuration is complete, the Model section automatically identifies and lists available models, such as Qwen3-8B.

   4. Click Test next to the API key input field to verify connectivity.

3. Quickly test the model

   Return to the chat interface, select the model at the top, and start a conversation.

   For example, if you enter Who are you? and the Qwen3-8B model responds with a self-introduction, it indicates that the model is deployed and working correctly.