Deploy large language models on PAI-EAS

cURL

curl -X POST <EAS_ENDPOINT>/v1/chat/completions \
    -H "Content-Type: application/json" \
    -H "Authorization: <EAS_TOKEN>" \
    -d '{
        "model": "<model_name>",
        "messages": [
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": "hello"
        }
        ],
        "max_tokens":1024,
        "temperature": 0.7,
        "top_p": 0.8,
        "stream":true
    }'

Where:

• Replace <EAS_ENDPOINT> and <EAS_TOKEN> with your service's endpoint and token.

• Replace <model_name> with the model name. For vLLM/SGLang, get the model name by calling the <EAS_ENDPOINT>/v1/models API endpoint.

  curl -X GET <EAS_ENDPOINT>/v1/models -H "Authorization: <EAS_TOKEN>"

OpenAI SDK

We recommend using the official Python SDK to interact with the service. Ensure you have the OpenAI SDK installed: pip install openai.

from openai import OpenAI

# 1. Configure the client
# Replace <EAS_TOKEN> with the token of your deployed service.
openai_api_key = "<EAS_TOKEN>"
# Replace <EAS_ENDPOINT> with the endpoint of your deployed service.
openai_api_base = "<EAS_ENDPOINT>/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

# 2. Get the model name.
# For BladeLLM, set model = "". BladeLLM does not use the model parameter or support retrieval with client.models.list().
# This parameter is set to an empty string to meet the OpenAI SDK's requirement.
models = client.models.list()
model = models.data[0].id
print(model)

# 3. Send a chat request.
# Streaming (stream=True) and non-streaming (stream=False) outputs are supported.
stream = True
chat_completion = client.chat.completions.create(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "hello"},          
    ],
    model=model,
    top_p=0.8,
    temperature=0.7,
    max_tokens=1024,
    stream=stream,
)

if stream:
    for chunk in chat_completion:
        print(chunk.choices[0].delta.content, end="")
else:
    result = chat_completion.choices[0].message.content
    print(result)

Python requests library

If you prefer not to use the OpenAI SDK, you can use the requests library.

import json
import requests

# Replace <EAS_ENDPOINT> with the endpoint of your deployed service.
EAS_ENDPOINT = "<EAS_ENDPOINT>"
# Replace <EAS_TOKEN> with the token of your deployed service.
EAS_TOKEN = "<EAS_TOKEN>"
# Replace <model_name> with the model name. You can get the model name by calling the <EAS_ENDPOINT>/v1/models API endpoint.
# (For BladeLLM, this API is not supported. You can omit the "model" field or set it to "".)
model = "<model_name>"

url = f"{EAS_ENDPOINT}/v1/chat/completions"
headers = {
    "Content-Type": "application/json",
    "Authorization": EAS_TOKEN,
}

stream = True
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "hello"},
]

req = {
    "messages": messages,
    "stream": stream,
    "temperature": 0.7,
    "top_p": 0.8,
    "max_tokens": 1024,
    "model": model,
}
response = requests.post(
    url,
    json=req,
    headers=headers,
    stream=stream,
)

if stream:
    for chunk in response.iter_lines(chunk_size=8192, decode_unicode=False):
        msg = chunk.decode("utf-8")
        # The following code handles streaming responses in Server-Sent Events (SSE) format.
        if msg.startswith("data:"):
            info = msg[6:]
            if info == "[DONE]":
                break
            else:
                resp = json.loads(info)
                if resp["choices"][0]["delta"].get("content") is not None:
                    print(resp["choices"][0]["delta"]["content"], end="", flush=True)
else:
    resp = json.loads(response.text)
    print(resp["choices"][0]["message"]["content"])

Dify

1. Install the OpenAI-API-compatible model provider

   In the upper-right corner of the page, click your profile picture and select Settings. In the left-side navigation pane, click Model Providers. If OpenAI-API-compatible is not in the Model List, find it in the list below and click Install.

2. Add a model

   On the OpenAI-API-compatible card, click Add Model in the lower-right corner and configure the following parameters:

   • Model type: Select LLM.

   • Model name: For vLLM deployments, send a GET request to the /v1/models API endpoint to get the model name. For this example, enter Qwen3-8B.

   • API key: Enter the token of the PAI-EAS service.

   • API endpoint URL: Enter the public endpoint of the PAI-EAS service. Note: The URL must end with /v1.

3. Test the service

   1. On the Dify main page, click Create Blank App, select the Chatflow type, enter the application name and other information, and then click Create.

   2. Click the LLM node and select the model you added. To configure the node, select Qwen3-8B CHAT from the OpenAI-API-compatible category. In the SYSTEM prompt, enter the role description and reference the context variable sys.query. In the USER area, add the sys.query and sys.files variables. You can adjust model parameters such as Max Tokens (default is 512), Presence Penalty, and Thought Mode as needed.

   3. Click Preview in the upper-right corner and enter a question.

      The bot processes the question through the workflow and returns a reply. The top of the conversation area displays Deeply thought and the processing time.

Chatbox

1. Go to Chatbox to download and install the client for your device, or click Launch Web App to use the web version. This example uses a Mac with an M3 chip.

2. Add a model provider. Click Settings, add a model provider, enter a name such as pai, and select OpenAI API Compatible as the API Mode.

3. Select the pai model provider and configure the following parameters.

   • API key: Enter the token of the PAI-EAS service.

   • API host: Enter the public endpoint of the PAI-EAS service. Note: The URL must end with /v1.

   • API path: Leave this empty.

   • Model: Click Get to add the model. If the inference engine is BladeLLM, which does not support API retrieval, click New to enter the model name manually.

4. Test the conversation. Click New Chat and select the model service in the lower-right corner of the text input box.

   The top of the conversation interface displays the system prompt, such as You are a helpful assistant.. Enter a question such as "Who are you?" in the text input box and send it. A collapsible Deeply thought area and the processing time are displayed at the top of the assistant's reply area, followed by the model's reply.

Cherry Studio

1. Install the client

   Visit Cherry Studio to download and install the client.

   You can also download it from https://github.com/CherryHQ/cherry-studio/releases.

2. Configure the model service.

   1. Click the settings icon in the lower-left corner. Under the Model Service section, click Add. For Provider Name, enter a custom name such as PAI, and set the provider type to OpenAI. Click OK.

   2. In the API key field, enter the token of the PAI-EAS service. In the API address field, enter the public endpoint of the PAI-EAS service.

   3. Click Add. In the Model ID field, enter the model name. For vLLM deployments, send a GET request to the /v1/models API endpoint to get the model name. For this example, enter Qwen3-8B. Note that the name is case-sensitive. When you enter the API address, do not include the /v1 suffix. If your URL ends with #, the client automatically uses the /v1/chat/completions path. After the configuration is complete, the Model area automatically identifies and lists available models, such as Qwen3-8B.

   4. Click Test next to the API key input box to verify connectivity.

3. Quickly test the model

   Return to the dialog box, select the model at the top, and start a conversation.

   For example, enter Who are you? and send the message. The Qwen3-8B model replies with a self-introduction, confirming that the model is ready.