messagesarray(Required)The context to pass to the Large Language Model (LLM), arranged in conversational order.
Message types
System Messageobject(Optional)
A system message that sets the role, tone, task objectives, or constraints for the LLM. It is usually placed first in the messages array.
Properties
contentstring(Required)
System instructions that define the model's role, behavior, response style, and task constraints.
rolestring(Required)
The role of the system message. The value is fixed to system.
User Messageobject(Required)
A user message.
Properties
contentarray(Required)
The message content.
Properties
typestring(Required)
Valid values:
text
Set to text for text input.
image_url
Set to image_url for image input.
textstring(Optional)
The input text. This parameter is required when type is set to text.
image_urlobject
The input image information. This parameter is required when type is set to image_url.
Properties
urlstring(Required)
The URL or Base64 Data URL of the image. To pass a local file, see Pass a local file.
min_pixelsinteger(Optional)
Sets the minimum pixel threshold for the input image. The unit is pixels.
If the input image has fewer pixels than min_pixels, the image is enlarged until its total pixel count exceeds min_pixels.
The default and minimum value is 3136.
max_pixelsinteger(Optional)
Sets the maximum pixel threshold for the input image. The unit is pixels.
If the pixel count of the input image is within the [min_pixels, max_pixels] range, the model processes the original image. If the pixel count exceeds max_pixels, the image is downscaled until its total pixel count is below max_pixels.
Specifies whether to increase the maximum pixel count of the input image to the value corresponding to 16384 tokens.
If vl_high_resolution_images is true, a fixed resolution policy is applied. The maximum pixel count is fixed at 12845056, and the max_pixels setting is ignored. If the image resolution exceeds this limit, the image's total pixel count is downscaled to fit within this limit.
If vl_high_resolution_images is false, the maximum pixel count is determined by max_pixels. If the pixel count of the input image exceeds max_pixels, the image is downscaled to fit within the max_pixels limit. The model's default maximum pixel count is the default value of max_pixels.
enable_thinkingboolean (Optional)
When using a hybrid thinking model (which can think before replying), specifies whether to enable the thinking mode. Among the interface interaction series models, only gui-plus-2026-02-26 is a hybrid thinking model. For more information, see Visual reasoning.
Valid values:
true: Enable.
If enabled, the thinking process content is returned in the reasoning_content field.
false: Disable.
This is not a standard OpenAI parameter. When calling with the Python SDK, place it in the extra_body object. The configuration is: extra_body={"enable_thinking": xxx}.
seedinteger(Optional)
A random number seed. This ensures that results are reproducible with the same input and parameters. If you make a call with the same seed and other parameters unchanged, the model returns the same result as much as possible.
Value range: [0,2<sup>31</sup>−1].
temperaturefloat(Optional)Default: 0.01
The sampling temperature, which controls the diversity of the text generated by the model.
A higher temperature results in more diverse text. A lower temperature results in more deterministic text.
Value range: [0, 2)
Both temperature and top_p control the diversity of the generated text. We recommend setting only one of them.
top_pfloat(Optional) Default: 0.01
The probability threshold for nucleus sampling, which controls the diversity of the text generated by the model.
A higher top_p results in more diverse text. A lower top_p results in more deterministic text.
Value range: (0,1.0]
Both temperature and top_p control the diversity of the generated text. We recommend setting only one of them.
top_kinteger(Optional) Default: 1
The size of the candidate set for sampling during generation. For example, a value of 50 means that only the 50 tokens with the highest scores in a single generation are used to form the random sampling candidate set. A larger value increases randomness. A smaller value increases determinism. A value of None or a value greater than 100 indicates that the top_k policy is not enabled, and only the top_p policy takes effect.
The value must be greater than or equal to 0.
This is not a standard OpenAI parameter. When calling with the Python SDK, place it in the extra_body object. The configuration is: extra_body={"top_k": xxx}. When calling with the Node.js SDK or by HTTP, pass it as a top-level parameter.
repetition_penaltyfloat(Optional) Default: 1.0
The degree of repetition in consecutive sequences during model generation. Increasing repetition_penalty reduces repetition. A value of 1.0 means no penalty. This parameter has a significant impact on the model's performance. We recommend keeping the default value.
presence_penaltyfloat(Optional)
Controls content repetition when the model generates text. The default value is 1.5.
In scenarios that require diversity, interest, or creativity, such as creative writing or brainstorming, increase this value. In scenarios that emphasize consistency and terminological accuracy, such as technical documents or formal texts, decrease this value.
How it works
If the parameter value is positive, the model applies a penalty to tokens that already exist in the current text (the penalty is independent of the number of occurrences). This reduces the chance of these tokens reappearing, thereby reducing content repetition and increasing word diversity.
Example
Prompt: Translate this sentence into Chinese: “This movie is good. The plot is good, the acting is good, the music is good, and overall, the whole movie is just good. It is really good, in fact. The plot is so good, and the acting is so good, and the music is so good.”
Parameter value is 2.0: This movie is great. The plot is fantastic, the acting is superb, and the music is beautiful. Overall, the entire film is simply outstanding. It is truly excellent, in fact. The story is very compelling, the performances are brilliant, and the score is so moving.
Parameter value is 0.0: This movie is good. The plot is good, the acting is good, and the music is good. Overall, the whole movie is good. In fact, it is really great. The plot is very good, the acting is very good, and the music is also excellent.
Parameter value is -2.0: This movie is good. The plot is good, the acting is good, and the music is good. Overall, the whole movie is good. In fact, it is really good. The plot is very good, the acting is very good, and the music is very good.
stopstring or array(Optional)
Specifies stop words. When a string or token_id specified in stop appears in the generated text, generation stops immediately.
You can pass sensitive words to control the model's output.
When stop is an array, you cannot mix token_ids and strings as elements. For example, ["Hello",104307] is not a valid specification.
Non-streaming output
Python
import os
from openai import OpenAI
system_prompt = """# Tools
You may call one or more functions to assist with the user query.
You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "computer_use", "description": "Use a mouse and keyboard to interact with a computer, and take screenshots.\\n* This is an interface to a desktop GUI. You do not have access to a terminal or applications menu. You must click on desktop icons to start applications.\\n* Some applications may take time to start or process actions, so you may need to wait and take successive screenshots to see the results of your actions. E.g. if you click on Firefox and a window doesn't open, try wait and taking another screenshot.\\n* The screen's resolution is 1000x1000.\\n* Make sure to click any buttons, links, icons, etc with the cursor tip in the center of the element. Don't click boxes on their edges unless asked.", "parameters": {"properties": {"action": {"description": "The action to perform. The available actions are:\\n* `key`: Performs key down presses on the arguments passed in order, then performs key releases in reverse order.\\n* `type`: Type a string of text on the keyboard.\\n* `mouse_move`: Move the cursor to a specified (x, y) pixel coordinate on the screen.\\n* `left_click`: Click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `left_click_drag`: Click and drag the cursor to a specified (x, y) pixel coordinate on the screen.\\n* `right_click`: Click the right mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `middle_click`: Click the middle mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `double_click`: Double-click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `triple_click`: Triple-click the left mouse button at a specified (x, y) pixel coordinate on the screen (simulated as double-click since it's the closest action).\\n* `scroll`: Performs a scroll of the mouse scroll wheel.\\n* `hscroll`: Performs a horizontal scroll (mapped to regular scroll).\\n* `wait`: Wait specified seconds for the change to happen.\\n* `terminate`: Terminate the current task and report its completion status.\\n* `answer`: Answer a question.\\n* `interact`: Resolve the blocking window by interacting with the user.", "enum": ["key", "type", "mouse_move", "left_click", "left_click_drag", "right_click", "middle_click", "double_click", "triple_click", "scroll", "hscroll", "wait", "terminate", "answer", "interact"], "type": "string"}, "keys": {"description": "Required only by `action=key`.", "type": "array"}, "text": {"description": "Required only by `action=type`, `action=answer` and `action=interact`.", "type": "string"}, "coordinate": {"description": "(x, y): The x (pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to. Required only by `action=mouse_move` and `action=left_click_drag`.", "type": "array"}, "pixels": {"description": "The amount of scrolling to perform. Positive values scroll up, negative values scroll down. Required only by `action=scroll` and `action=hscroll`.", "type": "number"}, "time": {"description": "The seconds to wait. Required only by `action=wait`.", "type": "number"}, "status": {"description": "The status of the task. Required only by `action=terminate`.", "type": "string", "enum": ["success", "failure"]}}, "required": ["action"], "type": "object"}}}
</tools>
For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call>
# Response format
Response format for every step:
1) Action: a short imperative describing what to do in the UI.
2) A single <tool_call>...</tool_call> block containing only the JSON: {"name": <function-name>, "arguments": <args-json-object>}.
Rules:
- Output exactly in the order: Action, <tool_call>.
- Be brief: one for Action.
- Do not output anything else outside those two parts.
- If finishing, use action=terminate in the tool call."""
messages = [
{
"role": "system",
"content": system_prompt
},
{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "https://img.alicdn.com/imgextra/i2/O1CN016iJ8ob1C3xP1s2M6z_!!6000000000026-2-tps-3008-1758.png"}},
{"type": "text", "text": "Help me open the browser"}
]
}
]
client = OpenAI(
# If you have not configured the environment variable, replace the following line with your Model Studio API key: api_key="sk-xxx",
api_key=os.getenv("DASHSCOPE_API_KEY"),
base_url="https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/compatible-mode/v1",
)
completion = client.chat.completions.create(
model="gui-plus-2026-02-26",
messages=messages,
extra_body={"vl_high_resolution_images": True}
)
print(completion.choices[0].message.content)
Node.js
import OpenAI from "openai";
const systemPrompt = `# Tools
You may call one or more functions to assist with the user query.
You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "computer_use", "description": "Use a mouse and keyboard to interact with a computer, and take screenshots.\\n* This is an interface to a desktop GUI. You do not have access to a terminal or applications menu. You must click on desktop icons to start applications.\\n* Some applications may take time to start or process actions, so you may need to wait and take successive screenshots to see the results of your actions. E.g. if you click on Firefox and a window doesn't open, try wait and taking another screenshot.\\n* The screen's resolution is 1000x1000.\\n* Make sure to click any buttons, links, icons, etc with the cursor tip in the center of the element. Don't click boxes on their edges unless asked.", "parameters": {"properties": {"action": {"description": "The action to perform. The available actions are:\\n* \`key\`: Performs key down presses on the arguments passed in order, then performs key releases in reverse order.\\n* \`type\`: Type a string of text on the keyboard.\\n* \`mouse_move\`: Move the cursor to a specified (x, y) pixel coordinate on the screen.\\n* \`left_click\`: Click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* \`left_click_drag\`: Click and drag the cursor to a specified (x, y) pixel coordinate on the screen.\\n* \`right_click\`: Click the right mouse button at a specified (x, y) pixel coordinate on the screen.\\n* \`middle_click\`: Click the middle mouse button at a specified (x, y) pixel coordinate on the screen.\\n* \`double_click\`: Double-click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* \`triple_click\`: Triple-click the left mouse button at a specified (x, y) pixel coordinate on the screen (simulated as double-click since it's the closest action).\\n* \`scroll\`: Performs a scroll of the mouse scroll wheel.\\n* \`hscroll\`: Performs a horizontal scroll (mapped to regular scroll).\\n* \`wait\`: Wait specified seconds for the change to happen.\\n* \`terminate\`: Terminate the current task and report its completion status.\\n* \`answer\`: Answer a question.\\n* \`interact\`: Resolve the blocking window by interacting with the user.", "enum": ["key", "type", "mouse_move", "left_click", "left_click_drag", "right_click", "middle_click", "double_click", "triple_click", "scroll", "hscroll", "wait", "terminate", "answer", "interact"], "type": "string"}, "keys": {"description": "Required only by \`action=key\`.", "type": "array"}, "text": {"description": "Required only by \`action=type\`, \`action=answer\` and \`action=interact\`.", "type": "string"}, "coordinate": {"description": "(x, y): The x (pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to. Required only by \`action=mouse_move\` and \`action=left_click_drag\`.", "type": "array"}, "pixels": {"description": "The amount of scrolling to perform. Positive values scroll up, negative values scroll down. Required only by \`action=scroll\` and \`action=hscroll\`.", "type": "number"}, "time": {"description": "The seconds to wait. Required only by \`action=wait\`.", "type": "number"}, "status": {"description": "The status of the task. Required only by \`action=terminate\`.", "type": "string", "enum": ["success", "failure"]}}, "required": ["action"], "type": "object"}}}
</tools>
For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call>
# Response format
Response format for every step:
1) Action: a short imperative describing what to do in the UI.
2) A single <tool_call>...</tool_call> block containing only the JSON: {"name": <function-name>, "arguments": <args-json-object>}.
Rules:
- Output exactly in the order: Action, <tool_call>.
- Be brief: one for Action.
- Do not output anything else outside those two parts.
- If finishing, use action=terminate in the tool call.`;
const client = new OpenAI({
apiKey: process.env.DASHSCOPE_API_KEY,
baseURL: "https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/compatible-mode/v1",
});
const messages = [
{
role: "system",
content: systemPrompt,
},
{
role: "user",
content: [
{
type: "image_url",
image_url: {
url: "https://img.alicdn.com/imgextra/i2/O1CN016iJ8ob1C3xP1s2M6z_!!6000000000026-2-tps-3008-1758.png",
},
},
{ type: "text", text: "Help me open the browser" },
],
},
];
const completion = await client.chat.completions.create({
model: "gui-plus-2026-02-26",
messages: messages,
extra_body: { vl_high_resolution_images: true },
});
console.log(completion.choices[0].message.content);
curl
# The following URL is for the China (Beijing) region. Replace {WorkspaceId} with your Workspace ID. URLs vary by region.
curl -X POST https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/compatible-mode/v1/chat/completions \
-H "Authorization: Bearer $DASHSCOPE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gui-plus-2026-02-26",
"messages": [
{
"role": "system",
"content": "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"type\": \"function\", \"function\": {\"name\": \"computer_use\", \"description\": \"Use a mouse and keyboard to interact with a computer, and take screenshots.\\n* This is an interface to a desktop GUI. You do not have access to a terminal or applications menu. You must click on desktop icons to start applications.\\n* Some applications may take time to start or process actions, so you may need to wait and take successive screenshots to see the results of your actions. E.g. if you click on Firefox and a window doesn'\''t open, try wait and taking another screenshot.\\n* The screen'\''s resolution is 1000x1000.\\n* Make sure to click any buttons, links, icons, etc with the cursor tip in the center of the element. Don'\''t click boxes on their edges unless asked.\", \"parameters\": {\"properties\": {\"action\": {\"description\": \"The action to perform. The available actions are:\\n* `key`: Performs key down presses on the arguments passed in order, then performs key releases in reverse order.\\n* `type`: Type a string of text on the keyboard.\\n* `mouse_move`: Move the cursor to a specified (x, y) pixel coordinate on the screen.\\n* `left_click`: Click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `left_click_drag`: Click and drag the cursor to a specified (x, y) pixel coordinate on the screen.\\n* `right_click`: Click the right mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `middle_click`: Click the middle mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `double_click`: Double-click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `triple_click`: Triple-click the left mouse button at a specified (x, y) pixel coordinate on the screen (simulated as double-click since it'\''s the closest action).\\n* `scroll`: Performs a scroll of the mouse scroll wheel.\\n* `hscroll`: Performs a horizontal scroll (mapped to regular scroll).\\n* `wait`: Wait specified seconds for the change to happen.\\n* `terminate`: Terminate the current task and report its completion status.\\n* `answer`: Answer a question.\\n* `interact`: Resolve the blocking window by interacting with the user.\", \"enum\": [\"key\", \"type\", \"mouse_move\", \"left_click\", \"left_click_drag\", \"right_click\", \"middle_click\", \"double_click\", \"triple_click\", \"scroll\", \"hscroll\", \"wait\", \"terminate\", \"answer\", \"interact\"], \"type\": \"string\"}, \"keys\": {\"description\": \"Required only by `action=key`.\", \"type\": \"array\"}, \"text\": {\"description\": \"Required only by `action=type`, `action=answer` and `action=interact`.\", \"type\": \"string\"}, \"coordinate\": {\"description\": \"(x, y): The x (pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to. Required only by `action=mouse_move` and `action=left_click_drag`.\", \"type\": \"array\"}, \"pixels\": {\"description\": \"The amount of scrolling to perform. Positive values scroll up, negative values scroll down. Required only by `action=scroll` and `action=hscroll`.\", \"type\": \"number\"}, \"time\": {\"description\": \"The seconds to wait. Required only by `action=wait`.\", \"type\": \"number\"}, \"status\": {\"description\": \"The status of the task. Required only by `action=terminate`.\", \"type\": \"string\", \"enum\": [\"success\", \"failure\"]}}, \"required\": [\"action\"], \"type\": \"object\"}}}\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call>\n\n# Response format\n\nResponse format for every step:\n1) Action: a short imperative describing what to do in the UI.\n2) A single <tool_call>...</tool_call> block containing only the JSON: {\"name\": <function-name>, \"arguments\": <args-json-object>}.\n\nRules:\n- Output exactly in the order: Action, <tool_call>.\n- Be brief: one for Action.\n- Do not output anything else outside those two parts.\n- If finishing, use action=terminate in the tool call."
},
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://img.alicdn.com/imgextra/i2/O1CN016iJ8ob1C3xP1s2M6z_!!6000000000026-2-tps-3008-1758.png"
}
},
{
"type": "text",
"text": "Help me open the browser"
}
]
}
],
"vl_high_resolution_images": true
}'
import os
from openai import OpenAI
system_prompt = """# Tools
You may call one or more functions to assist with the user query.
You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "computer_use", "description": "Use a mouse and keyboard to interact with a computer, and take screenshots.\\n* This is an interface to a desktop GUI. You do not have access to a terminal or applications menu. You must click on desktop icons to start applications.\\n* Some applications may take time to start or process actions, so you may need to wait and take successive screenshots to see the results of your actions. E.g. if you click on Firefox and a window doesn't open, try wait and taking another screenshot.\\n* The screen's resolution is 1000x1000.\\n* Make sure to click any buttons, links, icons, etc with the cursor tip in the center of the element. Don't click boxes on their edges unless asked.", "parameters": {"properties": {"action": {"description": "The action to perform. The available actions are:\\n* `key`: Performs key down presses on the arguments passed in order, then performs key releases in reverse order.\\n* `type`: Type a string of text on the keyboard.\\n* `mouse_move`: Move the cursor to a specified (x, y) pixel coordinate on the screen.\\n* `left_click`: Click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `left_click_drag`: Click and drag the cursor to a specified (x, y) pixel coordinate on the screen.\\n* `right_click`: Click the right mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `middle_click`: Click the middle mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `double_click`: Double-click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `triple_click`: Triple-click the left mouse button at a specified (x, y) pixel coordinate on the screen (simulated as double-click since it's the closest action).\\n* `scroll`: Performs a scroll of the mouse scroll wheel.\\n* `hscroll`: Performs a horizontal scroll (mapped to regular scroll).\\n* `wait`: Wait specified seconds for the change to happen.\\n* `terminate`: Terminate the current task and report its completion status.\\n* `answer`: Answer a question.\\n* `interact`: Resolve the blocking window by interacting with the user.", "enum": ["key", "type", "mouse_move", "left_click", "left_click_drag", "right_click", "middle_click", "double_click", "triple_click", "scroll", "hscroll", "wait", "terminate", "answer", "interact"], "type": "string"}, "keys": {"description": "Required only by `action=key`.", "type": "array"}, "text": {"description": "Required only by `action=type`, `action=answer` and `action=interact`.", "type": "string"}, "coordinate": {"description": "(x, y): The x (pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to. Required only by `action=mouse_move` and `action=left_click_drag`.", "type": "array"}, "pixels": {"description": "The amount of scrolling to perform. Positive values scroll up, negative values scroll down. Required only by `action=scroll` and `action=hscroll`.", "type": "number"}, "time": {"description": "The seconds to wait. Required only by `action=wait`.", "type": "number"}, "status": {"description": "The status of the task. Required only by `action=terminate`.", "type": "string", "enum": ["success", "failure"]}}, "required": ["action"], "type": "object"}}}
</tools>
For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call>
# Response format
Response format for every step:
1) Action: a short imperative describing what to do in the UI.
2) A single <tool_call>...</tool_call> block containing only the JSON: {"name": <function-name>, "arguments": <args-json-object>}.
Rules:
- Output exactly in the order: Action, <tool_call>.
- Be brief: one for Action.
- Do not output anything else outside those two parts.
- If finishing, use action=terminate in the tool call."""
messages = [
{
"role": "system",
"content": system_prompt
},
{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "https://img.alicdn.com/imgextra/i2/O1CN016iJ8ob1C3xP1s2M6z_!!6000000000026-2-tps-3008-1758.png"}},
{"type": "text", "text": "Help me open the browser"}
]
}
]
client = OpenAI(
# If you have not configured the environment variable, replace the following line with your Model Studio API key: api_key="sk-xxx",
api_key=os.getenv("DASHSCOPE_API_KEY"),
base_url="https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/compatible-mode/v1",
)
completion = client.chat.completions.create(
model="gui-plus-2026-02-26",
messages=messages,
stream=True,
stream_options={"include_usage":True}
)
for chunk in completion:
print(chunk.model_dump_json())
import OpenAI from "openai";
const systemPrompt = `# Tools
You may call one or more functions to assist with the user query.
You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "computer_use", "description": "Use a mouse and keyboard to interact with a computer, and take screenshots.\\n* This is an interface to a desktop GUI. You do not have access to a terminal or applications menu. You must click on desktop icons to start applications.\\n* Some applications may take time to start or process actions, so you may need to wait and take successive screenshots to see the results of your actions. E.g. if you click on Firefox and a window doesn't open, try wait and taking another screenshot.\\n* The screen's resolution is 1000x1000.\\n* Make sure to click any buttons, links, icons, etc with the cursor tip in the center of the element. Don't click boxes on their edges unless asked.", "parameters": {"properties": {"action": {"description": "The action to perform. The available actions are:\\n* \`key\`: Performs key down presses on the arguments passed in order, then performs key releases in reverse order.\\n* \`type\`: Type a string of text on the keyboard.\\n* \`mouse_move\`: Move the cursor to a specified (x, y) pixel coordinate on the screen.\\n* \`left_click\`: Click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* \`left_click_drag\`: Click and drag the cursor to a specified (x, y) pixel coordinate on the screen.\\n* \`right_click\`: Click the right mouse button at a specified (x, y) pixel coordinate on the screen.\\n* \`middle_click\`: Click the middle mouse button at a specified (x, y) pixel coordinate on the screen.\\n* \`double_click\`: Double-click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* \`triple_click\`: Triple-click the left mouse button at a specified (x, y) pixel coordinate on the screen (simulated as double-click since it's the closest action).\\n* \`scroll\`: Performs a scroll of the mouse scroll wheel.\\n* \`hscroll\`: Performs a horizontal scroll (mapped to regular scroll).\\n* \`wait\`: Wait specified seconds for the change to happen.\\n* \`terminate\`: Terminate the current task and report its completion status.\\n* \`answer\`: Answer a question.\\n* \`interact\`: Resolve the blocking window by interacting with the user.", "enum": ["key", "type", "mouse_move", "left_click", "left_click_drag", "right_click", "middle_click", "double_click", "triple_click", "scroll", "hscroll", "wait", "terminate", "answer", "interact"], "type": "string"}, "keys": {"description": "Required only by \`action=key\`.", "type": "array"}, "text": {"description": "Required only by \`action=type\`, \`action=answer\` and \`action=interact\`.", "type": "string"}, "coordinate": {"description": "(x, y): The x (pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to. Required only by \`action=mouse_move\` and \`action=left_click_drag\`.", "type": "array"}, "pixels": {"description": "The amount of scrolling to perform. Positive values scroll up, negative values scroll down. Required only by \`action=scroll\` and \`action=hscroll\`.", "type": "number"}, "time": {"description": "The seconds to wait. Required only by \`action=wait\`.", "type": "number"}, "status": {"description": "The status of the task. Required only by \`action=terminate\`.", "type": "string", "enum": ["success", "failure"]}}, "required": ["action"], "type": "object"}}}
</tools>
For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call>
# Response format
Response format for every step:
1) Action: a short imperative describing what to do in the UI.
2) A single <tool_call>...</tool_call> block containing only the JSON: {"name": <function-name>, "arguments": <args-json-object>}.
Rules:
- Output exactly in the order: Action, <tool_call>.
- Be brief: one for Action.
- Do not output anything else outside those two parts.
- If finishing, use action=terminate in the tool call.`;
const client = new OpenAI({
apiKey: process.env.DASHSCOPE_API_KEY,
baseURL: "https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/compatible-mode/v1",
});
const messages = [
{
role: "system",
content: systemPrompt,
},
{
role: "user",
content: [
{
type: "image_url",
image_url: {
url: "https://img.alicdn.com/imgextra/i2/O1CN016iJ8ob1C3xP1s2M6z_!!6000000000026-2-tps-3008-1758.png",
},
},
{ type: "text", text: "Help me open the browser" },
],
},
];
const openai = new OpenAI({
// If you have not configured the environment variable, replace the following line with your Model Studio API key: apiKey: "sk-xxx"
apiKey: process.env.DASHSCOPE_API_KEY,
baseURL: "https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/compatible-mode/v1"
});
async function main() {
const response = await openai.chat.completions.create({
model: "gui-plus",
messages: messages,
stream:true,
stream_options:{"include_usage":true}
});
console.log("The streaming output is:")
for await (const chunk of response) {
// If stream_options.include_usage is true, the choices field of the last chunk is an empty array and should be skipped. You can get the token usage from chunk.usage.
if (chunk.choices[0] && chunk.choices[0].delta.content != null) {
console.log(chunk.choices[0].delta.content);
}
}
}
main()
# The following URL is for the China (Beijing) region. Replace {WorkspaceId} with your Workspace ID. URLs vary by region.
curl -X POST https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/compatible-mode/v1/chat/completions \
-H "Authorization: Bearer $DASHSCOPE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gui-plus-2026-02-26",
"messages": [
{
"role": "system",
"content": "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"type\": \"function\", \"function\": {\"name\": \"computer_use\", \"description\": \"Use a mouse and keyboard to interact with a computer, and take screenshots.\\n* This is an interface to a desktop GUI. You do not have access to a terminal or applications menu. You must click on desktop icons to start applications.\\n* Some applications may take time to start or process actions, so you may need to wait and take successive screenshots to see the results of your actions. E.g. if you click on Firefox and a window doesn'\''t open, try wait and taking another screenshot.\\n* The screen'\''s resolution is 1000x1000.\\n* Make sure to click any buttons, links, icons, etc with the cursor tip in the center of the element. Don'\''t click boxes on their edges unless asked.\", \"parameters\": {\"properties\": {\"action\": {\"description\": \"The action to perform. The available actions are:\\n* `key`: Performs key down presses on the arguments passed in order, then performs key releases in reverse order.\\n* `type`: Type a string of text on the keyboard.\\n* `mouse_move`: Move the cursor to a specified (x, y) pixel coordinate on the screen.\\n* `left_click`: Click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `left_click_drag`: Click and drag the cursor to a specified (x, y) pixel coordinate on the screen.\\n* `right_click`: Click the right mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `middle_click`: Click the middle mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `double_click`: Double-click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `triple_click`: Triple-click the left mouse button at a specified (x, y) pixel coordinate on the screen (simulated as double-click since it'\''s the closest action).\\n* `scroll`: Performs a scroll of the mouse scroll wheel.\\n* `hscroll`: Performs a horizontal scroll (mapped to regular scroll).\\n* `wait`: Wait specified seconds for the change to happen.\\n* `terminate`: Terminate the current task and report its completion status.\\n* `answer`: Answer a question.\\n* `interact`: Resolve the blocking window by interacting with the user.\", \"enum\": [\"key\", \"type\", \"mouse_move\", \"left_click\", \"left_click_drag\", \"right_click\", \"middle_click\", \"double_click\", \"triple_click\", \"scroll\", \"hscroll\", \"wait\", \"terminate\", \"answer\", \"interact\"], \"type\": \"string\"}, \"keys\": {\"description\": \"Required only by `action=key`.\", \"type\": \"array\"}, \"text\": {\"description\": \"Required only by `action=type`, `action=answer` and `action=interact`.\", \"type\": \"string\"}, \"coordinate\": {\"description\": \"(x, y): The x (pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to. Required only by `action=mouse_move` and `action=left_click_drag`.\", \"type\": \"array\"}, \"pixels\": {\"description\": \"The amount of scrolling to perform. Positive values scroll up, negative values scroll down. Required only by `action=scroll` and `action=hscroll`.\", \"type\": \"number\"}, \"time\": {\"description\": \"The seconds to wait. Required only by `action=wait`.\", \"type\": \"number\"}, \"status\": {\"description\": \"The status of the task. Required only by `action=terminate`.\", \"type\": \"string\", \"enum\": [\"success\", \"failure\"]}}, \"required\": [\"action\"], \"type\": \"object\"}}}\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call>\n\n# Response format\n\nResponse format for every step:\n1) Action: a short imperative describing what to do in the UI.\n2) A single <tool_call>...</tool_call> block containing only the JSON: {\"name\": <function-name>, \"arguments\": <args-json-object>}.\n\nRules:\n- Output exactly in the order: Action, <tool_call>.\n- Be brief: one for Action.\n- Do not output anything else outside those two parts.\n- If finishing, use action=terminate in the tool call."
},
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://img.alicdn.com/imgextra/i2/O1CN016iJ8ob1C3xP1s2M6z_!!6000000000026-2-tps-3008-1758.png"
}
},
{
"type": "text",
"text": "Help me open the browser"
}
]
}
],
"vl_high_resolution_images": true,
"stream":true,
"stream_options":{"include_usage":true}
}'
Chat response object (non-streaming output)
idstring
The unique ID of this request.
choicesarray
An array of content generated by the model.
Properties
finish_reasonstring
The reason the model stopped generating.
There are two cases:
stop if the output stopped naturally.
length if the generation ended because it was too long.
indexinteger
The index of the current object in the choices array.
messageobject
The message output by the model.
Properties
contentstring
The result of the GUI task.
refusalstring
This parameter is currently fixed to null.
rolestring
The role of the message. The value is fixed to assistant.
audioobject
This parameter is currently fixed to null.
function_callobject
This parameter is currently fixed to null.
tool_callsarray
This parameter is currently fixed to null.
createdinteger
The UNIX timestamp when this request was created.
modelstring
The model used for this request.
objectstring
Always chat.completion.
service_tierstring
This parameter is currently fixed to null.
system_fingerprintstring
This parameter is currently fixed to null.
usageobject
Token usage information for this request.
Properties
completion_tokensinteger
The number of tokens in the model's output.
prompt_tokensinteger
The number of tokens in the input.
total_tokensinteger
The total number of tokens used, which is the sum of prompt_tokens and completion_tokens.
completion_tokens_detailsobject
A fine-grained categorization of the model's output tokens.
Properties
accepted_prediction_tokensinteger
This parameter is currently fixed to null.
audio_tokensinteger
This parameter is currently fixed to null.
reasoning_tokensinteger
This parameter is currently fixed to null.
text_tokensinteger
The number of tokens corresponding to the model's output text.
rejected_prediction_tokensinteger
This parameter is currently fixed to null.
prompt_tokens_detailsobject
A fine-grained categorization of the input tokens.
Properties
audio_tokensinteger
This parameter is currently fixed to null.
cached_tokensinteger
This parameter is currently fixed to null.
text_tokensinteger
The number of tokens corresponding to the model's input text.
image_tokensinteger
The number of tokens corresponding to the model's input image.
{
"id": "chatcmpl-ef17511a-aceb-4c47-8757-13a87af2152d",
"choices": [
{
"finish_reason": "stop",
"index": 0,
"logprobs": null,
"message": {
"content": "```json\n{\"thought\": \"The user wants to open the browser. I see a Google Chrome icon in the screenshot. It is the last one in the top-right row. The next step should be to click this Chrome browser icon to launch it.\", \"action\": \"click\", \"parameters\": {\"x\": 1086, \"y\": 129}}\n```",
"refusal": null,
"role": "assistant",
"annotations": null,
"audio": null,
"function_call": null,
"tool_calls": null
}
}
],
"created": 1763451557,
"model": "gui-plus",
"object": "chat.completion",
"service_tier": null,
"system_fingerprint": null,
"usage": {
"completion_tokens": 78,
"prompt_tokens": 2020,
"total_tokens": 2098,
"completion_tokens_details": {
"accepted_prediction_tokens": null,
"audio_tokens": null,
"reasoning_tokens": null,
"rejected_prediction_tokens": null,
"text_tokens": 78
},
"prompt_tokens_details": {
"audio_tokens": null,
"cached_tokens": null,
"image_tokens": 1244,
"text_tokens": 776
}
}
}
Chat response chunk object (streaming output)
idstring
The unique ID of this call. Each chunk object has the same ID.
choicesarray
An array of content generated by the model. If the include_usage parameter is set to true, this is empty in the last chunk.
Properties
deltaobject
The output content returned in the stream.
Properties
contentstring
qwen-mt-flash provides incremental updates for translation results, while qwen-mt-plus and qwen-mt-turbo provide non-incremental updates.
function_callobject
This parameter is currently fixed to null.
refusalobject
This parameter is currently fixed to null.
rolestring
The role of the message object. This has a value only in the first chunk.
finish_reasonstring
The reason the model stopped generating. There are three cases:
stop if the output stopped naturally.
null if generation is not finished.
length if the generation ended because it was too long.
indexinteger
The index of the current response in the choices array.
createdinteger
The UNIX timestamp when this request was created. Each chunk has the same timestamp.
modelstring
The model used for this request.
objectstring
Always chat.completion.chunk.
service_tierstring
This parameter is currently fixed to null.
system_fingerprintstring
This parameter is currently fixed to null.
usageobject
The tokens used by this request. This is returned only in the last chunk when include_usage is true.
Properties
completion_tokensinteger
The number of tokens in the model's output.
prompt_tokensinteger
The number of tokens in the input.
total_tokensinteger
The total number of tokens used, which is the sum of prompt_tokens and completion_tokens.
completion_tokens_detailsobject
A fine-grained categorization of the model's output tokens.
Properties
accepted_prediction_tokensinteger
This parameter is currently fixed to null.
audio_tokensinteger
This parameter is currently fixed to null.
reasoning_tokensinteger
This parameter is currently fixed to null.
text_tokensinteger
The number of tokens corresponding to the model's output text.
rejected_prediction_tokensinteger
This parameter is currently fixed to null.
prompt_tokens_detailsobject
A fine-grained categorization of the input tokens.
Properties
audio_tokensinteger
This parameter is currently fixed to null.
cached_tokensinteger
This parameter is currently fixed to null.
text_tokensinteger
The number of tokens corresponding to the model's input text.
image_tokensinteger
The number of tokens corresponding to the model's input image.
Sets the minimum pixel threshold for the input image. The unit is pixels.
If the input image has fewer pixels than min_pixels, the image is enlarged until its total pixel count exceeds min_pixels.
The default and minimum value is 3136.
Use with the image parameter. Example: {"image":"https://xxxx.jpeg","min_pixels":3136}
max_pixelsinteger(Optional)
Sets the maximum pixel threshold for the input image. The unit is pixels.
If the pixel count of the input image is within the [min_pixels, max_pixels] range, the model processes the original image. If the pixel count exceeds max_pixels, the image is downscaled until its total pixel count is below max_pixels.
Specifies whether to increase the maximum pixel count of the input image to the value corresponding to 16384 tokens.
If vl_high_resolution_images is true, a fixed resolution policy is applied. The maximum pixel count is fixed at 12845056, and the max_pixels setting is ignored. If the image resolution exceeds this limit, the image's total pixel count is downscaled to fit within this limit.
If vl_high_resolution_images is false, the maximum pixel count is determined by max_pixels. If the pixel count of the input image exceeds max_pixels, the image is downscaled to fit within the max_pixels limit. The model's default maximum pixel count is the default value of max_pixels.
enable_thinkingboolean (Optional)
When using a hybrid thinking model, specifies whether to enable the thinking mode. Among the interface interaction series models, only gui-plus-2026-02-26 is a hybrid thinking model. For more information, see Visual reasoning.
Valid values:
true: Enable.
If enabled, the thinking process content is returned in the reasoning_content field.
The Java SDK uses enableThinking. When making an HTTP call, place enable_thinking in the parameters object.
max_tokensinteger(Optional)
Maximum number of tokens to generate. If the output exceeds this value, the response is truncated.
The default and maximum values are the maximum output length of the model. For more information, see Model selection.
In the Java SDK, the parameter is maxTokens. When you call via HTTP, place max_tokens in the parameters object.
seedinteger(Optional)
Random number seed for reproducible results. Using the same seed and parameters produces consistent output.
Value range: [0, 2 31 -1].
When making an HTTP call, place seed in the parameters object.
temperaturefloat(Optional)Default: 0.01
The sampling temperature, which controls the diversity of the text generated by the model.
A higher temperature results in more diverse text. A lower temperature results in more deterministic text.
Value range: [0, 2)
Both temperature and top_p control the diversity of the generated text. We recommend setting only one of them.
When making an HTTP call, place temperature in the parameters object.
top_pfloat(Optional) Default: 0.01
The probability threshold for nucleus sampling, which controls the diversity of the text generated by the model.
A higher top_p results in more diverse text. A lower top_p results in more deterministic text.
Value range: (0,1.0]
Both temperature and top_p control the diversity of the generated text. We recommend setting only one of them.
In the Java SDK, the parameter is named topP. When you make HTTP calls, include top_p in the parameters object.
repetition_penaltyfloat(Optional) Default: 1.0
The degree of repetition in consecutive sequences during model generation. Increasing repetition_penalty reduces repetition. A value of 1.0 means no penalty. This parameter has a significant impact on the model's performance. We recommend keeping the default value.
In the Java SDK, this parameter is named repetitionPenalty. When you make an HTTP call, place repetition_penalty in the parameters object.
presence_penaltyfloat(Optional)
Controls content repetition when the model generates text. The default value is 1.5.
In scenarios that require diversity, interest, or creativity, such as creative writing or brainstorming, increase this value. In scenarios that emphasize consistency and terminological accuracy, such as technical documents or formal texts, decrease this value.
How it works
If the parameter value is positive, the model applies a penalty to tokens that already exist in the current text (the penalty is independent of the number of occurrences). This reduces the chance of these tokens reappearing, thereby reducing content repetition and increasing word diversity.
Example
Prompt: Translate this sentence into Chinese: “This movie is good. The plot is good, the acting is good, the music is good, and overall, the whole movie is just good. It is really good, in fact. The plot is so good, and the acting is so good, and the music is so good.”
Parameter value is 2.0: This movie is great. The plot is fantastic, the acting is superb, and the music is beautiful. Overall, the entire film is simply outstanding. It is truly excellent, in fact. The story is very compelling, the performances are brilliant, and the score is so moving.
Parameter value is 0.0: This movie is good. The plot is good, the acting is good, and the music is good. Overall, the whole movie is good. In fact, it is really great. The plot is very good, the acting is very good, and the music is also excellent.
Parameter value is -2.0: This movie is good. The plot is good, the acting is good, and the music is good. Overall, the whole movie is good. In fact, it is really good. The plot is very good, the acting is very good, and the music is very good.
top_kinteger(Optional) Default: 1
The size of the candidate set for sampling during generation. For example, a value of 50 means that only the 50 tokens with the highest scores in a single generation are used to form the random sampling candidate set. A larger value increases randomness. A smaller value increases determinism. A value of None or a value greater than 100 indicates that the top_k policy is not enabled, and only the top_p policy takes effect.
The value must be greater than or equal to 0.
This is not a standard OpenAI parameter. When calling with the Python SDK, place it in the extra_body object. The configuration is: extra_body={"top_k": xxx}. When calling with the Node.js SDK or by HTTP, pass it as a top-level parameter.
The Java SDK uses topKtop_k in the parameters object.
repetition_penaltyfloat(Optional) Default: 1.0
The degree of repetition in consecutive sequences during model generation. Increasing repetition_penalty reduces repetition. A value of 1.0 means no penalty. This parameter has a significant impact on the model's performance. We recommend keeping the default value.
streamboolean(Optional)
Specifies whether to return the response in a streaming manner.
Valid values:
false: Waits for the model to generate the complete response and returns it at once.
true: The model generates and returns data blocks as they are created. The client must read the blocks sequentially to reconstruct the full response.
This parameter is supported only by the Python SDK. To implement streaming output with the Java SDK, use the streamCall interface. To implement streaming output by HTTP, specify X-DashScope-SSE as enable in the header.
Specifies whether to enable incremental output in streaming mode. We recommend that you set this to true.
Parameter values:
false: Each output is the entire sequence generated so far. The final output is the complete result.
I
I like
I like apple
I like apple.
true (Recommended): Incremental output. Subsequent outputs do not include previously sent content. You must read these fragments in real time to get the complete result.
I
like
apple
.
In the Java SDK, this parameter is called incrementalOutput. When making HTTP calls, include incremental_output in the parameters object.
stopstring or array(Optional)
Specifies stop words. When a string or token_id specified in stop appears in the generated text, generation stops immediately.
You can pass sensitive words to control the model's output.
When stop is an array, you cannot mix token_ids and strings as elements. For example, ["Hello",104307] is not a valid specification.
Non-streaming output
import os
import dashscope
# China (Beijing) region. Replace {WorkspaceId} with your Workspace ID. URLs vary by region.
dashscope.base_http_api_url = "https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api/v1"
system_prompt = """# Tools
You may call one or more functions to assist with the user query.
You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "computer_use", "description": "Use a mouse and keyboard to interact with a computer, and take screenshots.\\n* This is an interface to a desktop GUI. You do not have access to a terminal or applications menu. You must click on desktop icons to start applications.\\n* Some applications may take time to start or process actions, so you may need to wait and take successive screenshots to see the results of your actions. E.g. if you click on Firefox and a window doesn't open, try wait and taking another screenshot.\\n* The screen's resolution is 1000x1000.\\n* Make sure to click any buttons, links, icons, etc with the cursor tip in the center of the element. Don't click boxes on their edges unless asked.", "parameters": {"properties": {"action": {"description": "The action to perform. The available actions are:\\n* `key`: Performs key down presses on the arguments passed in order, then performs key releases in reverse order.\\n* `type`: Type a string of text on the keyboard.\\n* `mouse_move`: Move the cursor to a specified (x, y) pixel coordinate on the screen.\\n* `left_click`: Click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `left_click_drag`: Click and drag the cursor to a specified (x, y) pixel coordinate on the screen.\\n* `right_click`: Click the right mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `middle_click`: Click the middle mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `double_click`: Double-click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `triple_click`: Triple-click the left mouse button at a specified (x, y) pixel coordinate on the screen (simulated as double-click since it's the closest action).\\n* `scroll`: Performs a scroll of the mouse scroll wheel.\\n* `hscroll`: Performs a horizontal scroll (mapped to regular scroll).\\n* `wait`: Wait specified seconds for the change to happen.\\n* `terminate`: Terminate the current task and report its completion status.\\n* `answer`: Answer a question.\\n* `interact`: Resolve the blocking window by interacting with the user.", "enum": ["key", "type", "mouse_move", "left_click", "left_click_drag", "right_click", "middle_click", "double_click", "triple_click", "scroll", "hscroll", "wait", "terminate", "answer", "interact"], "type": "string"}, "keys": {"description": "Required only by `action=key`.", "type": "array"}, "text": {"description": "Required only by `action=type`, `action=answer` and `action=interact`.", "type": "string"}, "coordinate": {"description": "(x, y): The x (pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to. Required only by `action=mouse_move` and `action=left_click_drag`.", "type": "array"}, "pixels": {"description": "The amount of scrolling to perform. Positive values scroll up, negative values scroll down. Required only by `action=scroll` and `action=hscroll`.", "type": "number"}, "time": {"description": "The seconds to wait. Required only by `action=wait`.", "type": "number"}, "status": {"description": "The status of the task. Required only by `action=terminate`.", "type": "string", "enum": ["success", "failure"]}}, "required": ["action"], "type": "object"}}}
</tools>
For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call>
# Response format
Response format for every step:
1) Action: a short imperative describing what to do in the UI.
2) A single <tool_call>...</tool_call> block containing only the JSON: {"name": <function-name>, "arguments": <args-json-object>}.
Rules:
- Output exactly in the order: Action, <tool_call>.
- Be brief: one for Action.
- Do not output anything else outside those two parts.
- If finishing, use action=terminate in the tool call."""
messages = [
{
"role": "system",
"content": system_prompt
},
{
"role": "user",
"content": [
{"image": "https://img.alicdn.com/imgextra/i2/O1CN016iJ8ob1C3xP1s2M6z_!!6000000000026-2-tps-3008-1758.png"},
{"text": "Help me open the browser."}]
}]
response = dashscope.MultiModalConversation.call(
# If you have not configured the environment variable, replace the following line with your Model Studio API key: api_key = "sk-xxx"
api_key=os.getenv('DASHSCOPE_API_KEY'),
model='gui-plus-2026-02-26',
messages=messages,
vl_high_resolution_images=True
)
print(response.output.choices[0].message.content[0]["text"])
import java.util.Arrays;
import java.util.Collections;
import com.alibaba.dashscope.aigc.multimodalconversation.MultiModalConversation;
import com.alibaba.dashscope.aigc.multimodalconversation.MultiModalConversationParam;
import com.alibaba.dashscope.aigc.multimodalconversation.MultiModalConversationResult;
import com.alibaba.dashscope.common.MultiModalMessage;
import com.alibaba.dashscope.common.Role;
import com.alibaba.dashscope.exception.ApiException;
import com.alibaba.dashscope.exception.NoApiKeyException;
import com.alibaba.dashscope.exception.UploadFileException;
import com.alibaba.dashscope.utils.Constants;
public class Main {
// China (Beijing) region. Replace {WorkspaceId} with your Workspace ID. URLs vary by region.
static {Constants.baseHttpApiUrl="https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api/v1";}
public static void simpleMultiModalConversationCall()
throws ApiException, NoApiKeyException, UploadFileException {
String systemPrompt = "# Tools\n\n" +
"You may call one or more functions to assist with the user query.\n\n" +
"You are provided with function signatures within <tools></tools> XML tags:\n" +
"<tools>\n" +
"{\"type\": \"function\", \"function\": {\"name\": \"computer_use\", \"description\": \"Use a mouse and keyboard to interact with a computer, and take screenshots.\\n* This is an interface to a desktop GUI. You do not have access to a terminal or applications menu. You must click on desktop icons to start applications.\\n* Some applications may take time to start or process actions, so you may need to wait and take successive screenshots to see the results of your actions. E.g. if you click on Firefox and a window doesn't open, try wait and taking another screenshot.\\n* The screen's resolution is 1000x1000.\\n* Make sure to click any buttons, links, icons, etc with the cursor tip in the center of the element. Don't click boxes on their edges unless asked.\", \"parameters\": {\"properties\": {\"action\": {\"description\": \"The action to perform. The available actions are:\\n* `key`: Performs key down presses on the arguments passed in order, then performs key releases in reverse order.\\n* `type`: Type a string of text on the keyboard.\\n* `mouse_move`: Move the cursor to a specified (x, y) pixel coordinate on the screen.\\n* `left_click`: Click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `left_click_drag\`: Click and drag the cursor to a specified (x, y) pixel coordinate on the screen.\\n* `right_click`: Click the right mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `middle_click`: Click the middle mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `double_click`: Double-click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `triple_click`: Triple-click the left mouse button at a specified (x, y) pixel coordinate on the screen (simulated as double-click since it's the closest action).\\n* `scroll`: Performs a scroll of the mouse scroll wheel.\\n* `hscroll`: Performs a horizontal scroll (mapped to regular scroll).\\n* `wait`: Wait specified seconds for the change to happen.\\n* `terminate`: Terminate the current task and report its completion status.\\n* `answer`: Answer a question.\\n* `interact`: Resolve the blocking window by interacting with the user.\", \"enum\": [\"key\", \"type\", \"mouse_move\", \"left_click\", \"left_click_drag\", \"right_click\", \"middle_click\", \"double_click\", \"triple_click\", \"scroll\", \"hscroll\", \"wait\", \"terminate\", \"answer\", \"interact\"], \"type\": \"string\"}, \"keys\": {\"description\": \"Required only by `action=key`.\", \"type\": \"array\"}, \"text\": {\"description\": \"Required only by `action=type`, `action=answer` and `action=interact`.\", \"type\": \"string\"}, \"coordinate\": {\"description\": \"(x, y): The x (pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to. Required only by `action=mouse_move` and `action=left_click_drag`.\", \"type\": \"array\"}, \"pixels\": {\"description\": \"The amount of scrolling to perform. Positive values scroll up, negative values scroll down. Required only by `action=scroll` and `action=hscroll`.\", \"type\": \"number\"}, \"time\": {\"description\": \"The seconds to wait. Required only by `action=wait`.\", \"type\": \"number\"}, \"status\": {\"description\": \"The status of the task. Required only by `action=terminate`.\", \"type\": \"string\", \"enum\": [\"success\", \"failure\"]}}, \"required\": [\"action\"], \"type\": \"object\"}}}\n" +
"</tools>\n\n" +
"For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n" +
"<tool_call>\n" +
"{\"name\": <function-name>, \"arguments\": <args-json-object>}\n" +
"</tool_call>\n\n" +
"# Response format\n\n" +
"Response format for every step:\n" +
"1) Action: a short imperative describing what to do in the UI.\n" +
"2) A single <tool_call>...</tool_call> block containing only the JSON: {\"name\": <function-name>, \"arguments\": <args-json-object>}.\n\n" +
"Rules:\n" +
"- Output exactly in the order: Action, <tool_call>.\n" +
"- Be brief: one for Action.\n" +
"- Do not output anything else outside those two parts.\n" +
"- If finishing, use action=terminate in the tool call.";
MultiModalConversation conv = new MultiModalConversation();
MultiModalMessage systemMsg = MultiModalMessage.builder().role(Role.SYSTEM.getValue())
.content(Arrays.asList(
Collections.singletonMap("text",systemPrompt))).build();
MultiModalMessage userMessage = MultiModalMessage.builder().role(Role.USER.getValue())
.content(Arrays.asList(
Collections.singletonMap("image", "https://img.alicdn.com/imgextra/i2/O1CN016iJ8ob1C3xP1s2M6z_!!6000000000026-2-tps-3008-1758.png"),
Collections.singletonMap("text", "Help me open the browser."))).build();
MultiModalConversationParam param = MultiModalConversationParam.builder()
// If you have not configured the environment variable, replace the following line with your Model Studio API key: .apiKey("sk-xxx")
.apiKey(System.getenv("DASHSCOPE_API_KEY"))
.model("gui-plus-2026-02-26")
.messages(Arrays.asList(systemMsg,userMessage))
.vlHighResolutionImages(true)
.build();
MultiModalConversationResult result = conv.call(param);
System.out.println(result.getOutput().getChoices().get(0).getMessage().getContent().get(0).get("text"));
}
public static void main(String[] args) {
try {
simpleMultiModalConversationCall();
} catch (ApiException | NoApiKeyException | UploadFileException e) {
System.out.println(e.getMessage());
}
System.exit(0);
}
}
# The following URL is for the China (Beijing) region. Replace {WorkspaceId} with your Workspace ID. URLs vary by region.
curl -X POST https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation \
-H "Authorization: Bearer $DASHSCOPE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gui-plus-2026-02-26",
"input": {
"messages": [
{
"role": "system",
"content": [
{
"text": "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"type\": \"function\", \"function\": {\"name\": \"computer_use\", \"description\": \"Use a mouse and keyboard to interact with a computer, and take screenshots.\\n* This is an interface to a desktop GUI. You do not have access to a terminal or applications menu. You must click on desktop icons to start applications.\\n* Some applications may take time to start or process actions, so you may need to wait and take successive screenshots to see the results of your actions. E.g. if you click on Firefox and a window doesn'\''t open, try wait and taking another screenshot.\\n* The screen'\''s resolution is 1000x1000.\\n* Make sure to click any buttons, links, icons, etc with the cursor tip in the center of the element. Don'\''t click boxes on their edges unless asked.\", \"parameters\": {\"properties\": {\"action\": {\"description\": \"The action to perform. The available actions are:\\n* `key`: Performs key down presses on the arguments passed in order, then performs key releases in reverse order.\\n* `type`: Type a string of text on the keyboard.\\n* `mouse_move`: Move the cursor to a specified (x, y) pixel coordinate on the screen.\\n* `left_click`: Click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `left_click_drag`: Click and drag the cursor to a specified (x, y) pixel coordinate on the screen.\\n* `right_click`: Click the right mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `middle_click`: Click the middle mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `double_click`: Double-click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `triple_click`: Triple-click the left mouse button at a specified (x, y) pixel coordinate on the screen (simulated as double-click since it'\''s the closest action).\\n* `scroll`: Performs a scroll of the mouse scroll wheel.\\n* `hscroll`: Performs a horizontal scroll (mapped to regular scroll).\\n* `wait`: Wait specified seconds for the change to happen.\\n* `terminate`: Terminate the current task and report its completion status.\\n* `answer`: Answer a question.\\n* `interact`: Resolve the blocking window by interacting with the user.\", \"enum\": [\"key\", \"type\", \"mouse_move\", \"left_click\", \"left_click_drag\", \"right_click\", \"middle_click\", \"double_click\", \"triple_click\", \"scroll\", \"hscroll\", \"wait\", \"terminate\", \"answer\", \"interact\"], \"type\": \"string\"}, \"keys\": {\"description\": \"Required only by `action=key`.\", \"type\": \"array\"}, \"text\": {\"description\": \"Required only by `action=type`, `action=answer` and `action=interact`.\", \"type\": \"string\"}, \"coordinate\": {\"description\": \"(x, y): The x (pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to. Required only by `action=mouse_move` and `action=left_click_drag`.\", \"type\": \"array\"}, \"pixels\": {\"description\": \"The amount of scrolling to perform. Positive values scroll up, negative values scroll down. Required only by `action=scroll` and `action=hscroll`.\", \"type\": \"number\"}, \"time\": {\"description\": \"The seconds to wait. Required only by `action=wait`.\", \"type\": \"number\"}, \"status\": {\"description\": \"The status of the task. Required only by `action=terminate`.\", \"type\": \"string\", \"enum\": [\"success\", \"failure\"]}}, \"required\": [\"action\"], \"type\": \"object\"}}}\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call>\n\n# Response format\n\nResponse format for every step:\n1) Action: a short imperative describing what to do in the UI.\n2) A single <tool_call>...</tool_call> block containing only the JSON: {\"name\": <function-name>, \"arguments\": <args-json-object>}.\n\nRules:\n- Output exactly in the order: Action, <tool_call>.\n- Be brief: one for Action.\n- Do not output anything else outside those two parts.\n- If finishing, use action=terminate in the tool call."
}
]
},
{
"role": "user",
"content": [
{
"image": "https://img.alicdn.com/imgextra/i2/O1CN016iJ8ob1C3xP1s2M6z_!!6000000000026-2-tps-3008-1758.png"
},
{
"text": "Help me open the browser"
}
]
}
]
},
"parameters": {
"vl_high_resolution_images": true
}
}'
Streaming output
import os
import dashscope
# China (Beijing) region. Replace {WorkspaceId} with your Workspace ID. URLs vary by region.
dashscope.base_http_api_url = "https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api/v1"
system_prompt = """# Tools
You may call one or more functions to assist with the user query.
You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "computer_use", "description": "Use a mouse and keyboard to interact with a computer, and take screenshots.\\n* This is an interface to a desktop GUI. You do not have access to a terminal or applications menu. You must click on desktop icons to start applications.\\n* Some applications may take time to start or process actions, so you may need to wait and take successive screenshots to see the results of your actions. E.g. if you click on Firefox and a window doesn't open, try wait and taking another screenshot.\\n* The screen's resolution is 1000x1000.\\n* Make sure to click any buttons, links, icons, etc with the cursor tip in the center of the element. Don't click boxes on their edges unless asked.", "parameters": {"properties": {"action": {"description": "The action to perform. The available actions are:\\n* `key`: Performs key down presses on the arguments passed in order, then performs key releases in reverse order.\\n* `type`: Type a string of text on the keyboard.\\n* `mouse_move`: Move the cursor to a specified (x, y) pixel coordinate on the screen.\\n* `left_click`: Click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `left_click_drag`: Click and drag the cursor to a specified (x, y) pixel coordinate on the screen.\\n* `right_click`: Click the right mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `middle_click`: Click the middle mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `double_click`: Double-click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `triple_click`: Triple-click the left mouse button at a specified (x, y) pixel coordinate on the screen (simulated as double-click since it's the closest action).\\n* `scroll`: Performs a scroll of the mouse scroll wheel.\\n* `hscroll`: Performs a horizontal scroll (mapped to regular scroll).\\n* `wait`: Wait specified seconds for the change to happen.\\n* `terminate`: Terminate the current task and report its completion status.\\n* `answer`: Answer a question.\\n* `interact`: Resolve the blocking window by interacting with the user.", "enum": ["key", "type", "mouse_move", "left_click", "left_click_drag", "right_click", "middle_click", "double_click", "triple_click", "scroll", "hscroll", "wait", "terminate", "answer", "interact"], "type": "string"}, "keys": {"description": "Required only by `action=key`.", "type": "array"}, "text": {"description": "Required only by `action=type`, `action=answer` and `action=interact`.", "type": "string"}, "coordinate": {"description": "(x, y): The x (pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to. Required only by `action=mouse_move` and `action=left_click_drag`.", "type": "array"}, "pixels": {"description": "The amount of scrolling to perform. Positive values scroll up, negative values scroll down. Required only by `action=scroll` and `action=hscroll`.", "type": "number"}, "time": {"description": "The seconds to wait. Required only by `action=wait`.", "type": "number"}, "status": {"description": "The status of the task. Required only by `action=terminate`.", "type": "string", "enum": ["success", "failure"]}}, "required": ["action"], "type": "object"}}}
</tools>
For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call>
# Response format
Response format for every step:
1) Action: a short imperative describing what to do in the UI.
2) A single <tool_call>...</tool_call> block containing only the JSON: {"name": <function-name>, "arguments": <args-json-object>}.
Rules:
- Output exactly in the order: Action, <tool_call>.
- Be brief: one for Action.
- Do not output anything else outside those two parts.
- If finishing, use action=terminate in the tool call."""
messages = [
{
"role": "system",
"content": system_prompt
},
{
"role": "user",
"content": [
{"image": "https://img.alicdn.com/imgextra/i2/O1CN016iJ8ob1C3xP1s2M6z_!!6000000000026-2-tps-3008-1758.png"},
{"text": "Help me open the browser."}]
}]
response = dashscope.MultiModalConversation.call(
# If you have not configured the environment variable, replace the following line with your Model Studio API key: api_key = "sk-xxx"
api_key = os.getenv('DASHSCOPE_API_KEY'),
model = 'gui-plus',
messages = messages,
stream=True
)
for chunk in response:
print(chunk)
import java.util.Arrays;
import java.util.Collections;
import com.alibaba.dashscope.aigc.multimodalconversation.MultiModalConversation;
import com.alibaba.dashscope.aigc.multimodalconversation.MultiModalConversationParam;
import com.alibaba.dashscope.aigc.multimodalconversation.MultiModalConversationResult;
import com.alibaba.dashscope.common.MultiModalMessage;
import com.alibaba.dashscope.common.Role;
import com.alibaba.dashscope.exception.ApiException;
import com.alibaba.dashscope.exception.NoApiKeyException;
import com.alibaba.dashscope.exception.UploadFileException;
import com.alibaba.dashscope.utils.Constants;
import io.reactivex.Flowable;
public class Main {
// China (Beijing) region. Replace {WorkspaceId} with your Workspace ID. URLs vary by region.
static {Constants.baseHttpApiUrl="https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api/v1";}
public static void streamCall()
throws ApiException, NoApiKeyException, UploadFileException {
String systemPrompt = "# Tools\n\n" +
"You may call one or more functions to assist with the user query.\n\n" +
"You are provided with function signatures within <tools></tools> XML tags:\n" +
"<tools>\n" +
"{\"type\": \"function\", \"function\": {\"name\": \"computer_use\", \"description\": \"Use a mouse and keyboard to interact with a computer, and take screenshots.\\n* This is an interface to a desktop GUI. You do not have access to a terminal or applications menu. You must click on desktop icons to start applications.\\n* Some applications may take time to start or process actions, so you may need to wait and take successive screenshots to see the results of your actions. E.g. if you click on Firefox and a window doesn't open, try wait and taking another screenshot.\\n* The screen's resolution is 1000x1000.\\n* Make sure to click any buttons, links, icons, etc with the cursor tip in the center of the element. Don't click boxes on their edges unless asked.\", \"parameters\": {\"properties\": {\"action\": {\"description\": \"The action to perform. The available actions are:\\n* `key`: Performs key down presses on the arguments passed in order, then performs key releases in reverse order.\\n* `type`: Type a string of text on the keyboard.\\n* `mouse_move`: Move the cursor to a specified (x, y) pixel coordinate on the screen.\\n* `left_click`: Click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `left_click_drag\`: Click and drag the cursor to a specified (x, y) pixel coordinate on the screen.\\n* `right_click`: Click the right mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `middle_click`: Click the middle mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `double_click`: Double-click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `triple_click`: Triple-click the left mouse button at a specified (x, y) pixel coordinate on the screen (simulated as double-click since it's the closest action).\\n* `scroll`: Performs a scroll of the mouse scroll wheel.\\n* `hscroll`: Performs a horizontal scroll (mapped to regular scroll).\\n* `wait`: Wait specified seconds for the change to happen.\\n* `terminate`: Terminate the current task and report its completion status.\\n* `answer`: Answer a question.\\n* `interact`: Resolve the blocking window by interacting with the user.\", \"enum\": [\"key\", \"type\", \"mouse_move\", \"left_click\", \"left_click_drag\", \"right_click\", \"middle_click\", \"double_click\", \"triple_click\", \"scroll\", \"hscroll\", \"wait\", \"terminate\", \"answer\", \"interact\"], \"type\": \"string\"}, \"keys\": {\"description\": \"Required only by `action=key`.\", \"type\": \"array\"}, \"text\": {\"description\": \"Required only by `action=type`, `action=answer` and `action=interact`.\", \"type\": \"string\"}, \"coordinate\": {\"description\": \"(x, y): The x (pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to. Required only by `action=mouse_move` and `action=left_click_drag`.\", \"type\": \"array\"}, \"pixels\": {\"description\": \"The amount of scrolling to perform. Positive values scroll up, negative values scroll down. Required only by `action=scroll` and `action=hscroll`.\", \"type\": \"number\"}, \"time\": {\"description\": \"The seconds to wait. Required only by `action=wait`.\", \"type\": \"number\"}, \"status\": {\"description\": \"The status of the task. Required only by `action=terminate`.\", \"type\": \"string\", \"enum\": [\"success\", \"failure\"]}}, \"required\": [\"action\"], \"type\": \"object\"}}}\n" +
"</tools>\n\n" +
"For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n" +
"<tool_call>\n" +
"{\"name\": <function-name>, \"arguments\": <args-json-object>}\n" +
"</tool_call>\n\n" +
"# Response format\n\n" +
"Response format for every step:\n" +
"1) Action: a short imperative describing what to do in the UI.\n" +
"2) A single <tool_call>...</tool_call> block containing only the JSON: {\"name\": <function-name>, \"arguments\": <args-json-object>}.\n\n" +
"Rules:\n" +
"- Output exactly in the order: Action, <tool_call>.\n" +
"- Be brief: one for Action.\n" +
"- Do not output anything else outside those two parts.\n" +
"- If finishing, use action=terminate in the tool call.";
MultiModalConversation conv = new MultiModalConversation();
MultiModalMessage systemMsg = MultiModalMessage.builder().role(Role.SYSTEM.getValue())
.content(Arrays.asList(
Collections.singletonMap("text",systemPrompt))).build();
MultiModalMessage userMessage = MultiModalMessage.builder().role(Role.USER.getValue())
.content(Arrays.asList(
Collections.singletonMap("image", "https://img.alicdn.com/imgextra/i2/O1CN016iJ8ob1C3xP1s2M6z_!!6000000000026-2-tps-3008-1758.png"),
Collections.singletonMap("text", "Help me open the browser."))).build();
MultiModalConversationParam param = MultiModalConversationParam.builder()
// If you have not configured the environment variable, replace the following line with your Model Studio API key: .apiKey("sk-xxx")
.apiKey(System.getenv("DASHSCOPE_API_KEY"))
.model("gui-plus")
.messages(Arrays.asList(userMessage,systemMsg))
.incrementalOutput(true)
.build();
Flowable<MultiModalConversationResult> result = conv.streamCall(param);
result.blockingForEach(item -> {
try {
var content = item.getOutput().getChoices().get(0).getMessage().getContent();
// Check if content exists and is not empty
if (content != null && !content.isEmpty()) {
System.out.println(content.get(0).get("text"));
}
} catch (Exception e) {
System.out.println(e.getMessage());
}
});
}
public static void main(String[] args) {
try {
streamCall();
} catch (ApiException | NoApiKeyException | UploadFileException e) {
System.out.println(e.getMessage());
}
System.exit(0);
}
}
# The following URL is for the China (Beijing) region. Replace {WorkspaceId} with your Workspace ID. URLs vary by region.
curl -X POST https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation \
-H "Authorization: Bearer $DASHSCOPE_API_KEY" \
-H 'Content-Type: application/json' \
-H "X-DashScope-SSE: enable" \
-d '{
"model": "gui-plus-2026-02-26",
"input": {
"messages": [
{
"role": "system",
"content": [
{
"text": "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"type\": \"function\", \"function\": {\"name\": \"computer_use\", \"description\": \"Use a mouse and keyboard to interact with a computer, and take screenshots.\\n* This is an interface to a desktop GUI. You do not have access to a terminal or applications menu. You must click on desktop icons to start applications.\\n* Some applications may take time to start or process actions, so you may need to wait and take successive screenshots to see the results of your actions. E.g. if you click on Firefox and a window doesn'\''t open, try wait and taking another screenshot.\\n* The screen'\''s resolution is 1000x1000.\\n* Make sure to click any buttons, links, icons, etc with the cursor tip in the center of the element. Don'\''t click boxes on their edges unless asked.\", \"parameters\": {\"properties\": {\"action\": {\"description\": \"The action to perform. The available actions are:\\n* `key`: Performs key down presses on the arguments passed in order, then performs key releases in reverse order.\\n* `type`: Type a string of text on the keyboard.\\n* `mouse_move`: Move the cursor to a specified (x, y) pixel coordinate on the screen.\\n* `left_click`: Click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `left_click_drag`: Click and drag the cursor to a specified (x, y) pixel coordinate on the screen.\\n* `right_click`: Click the right mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `middle_click`: Click the middle mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `double_click`: Double-click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `triple_click`: Triple-click the left mouse button at a specified (x, y) pixel coordinate on the screen (simulated as double-click since it'\''s the closest action).\\n* `scroll`: Performs a scroll of the mouse scroll wheel.\\n* `hscroll`: Performs a horizontal scroll (mapped to regular scroll).\\n* `wait`: Wait specified seconds for the change to happen.\\n* `terminate`: Terminate the current task and report its completion status.\\n* `answer`: Answer a question.\\n* `interact`: Resolve the blocking window by interacting with the user.\", \"enum\": [\"key\", \"type\", \"mouse_move\", \"left_click\", \"left_click_drag\", \"right_click\", \"middle_click\", \"double_click\", \"triple_click\", \"scroll\", \"hscroll\", \"wait\", \"terminate\", \"answer\", \"interact\"], \"type\": \"string\"}, \"keys\": {\"description\": \"Required only by `action=key`.\", \"type\": \"array\"}, \"text\": {\"description\": \"Required only by `action=type`, `action=answer` and `action=interact`.\", \"type\": \"string\"}, \"coordinate\": {\"description\": \"(x, y): The x (pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to. Required only by `action=mouse_move` and `action=left_click_drag`.\", \"type\": \"array\"}, \"pixels\": {\"description\": \"The amount of scrolling to perform. Positive values scroll up, negative values scroll down. Required only by `action=scroll` and `action=hscroll`.\", \"type\": \"number\"}, \"time\": {\"description\": \"The seconds to wait. Required only by `action=wait`.\", \"type\": \"number\"}, \"status\": {\"description\": \"The status of the task. Required only by `action=terminate`.\", \"type\": \"string\", \"enum\": [\"success\", \"failure\"]}}, \"required\": [\"action\"], \"type\": \"object\"}}}\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call>\n\n# Response format\n\nResponse format for every step:\n1) Action: a short imperative describing what to do in the UI.\n2) A single <tool_call>...</tool_call> block containing only the JSON: {\"name\": <function-name>, \"arguments\": <args-json-object>}.\n\nRules:\n- Output exactly in the order: Action, <tool_call>.\n- Be brief: one for Action.\n- Do not output anything else outside those two parts.\n- If finishing, use action=terminate in the tool call."
}
]
},
{
"role": "user",
"content": [
{
"image": "https://img.alicdn.com/imgextra/i2/O1CN016iJ8ob1C3xP1s2M6z_!!6000000000026-2-tps-3008-1758.png"
},
{
"text": "Help me open the browser"
}
]
}
]
},
"parameters": {
"vl_high_resolution_images": true
}
}'
Chat response object (same format for streaming and non-streaming output)
status_codestring
The status code of this request. A value of 200 indicates that the request was successful. Otherwise, the request failed.
The Java SDK does not return this parameter. If the call fails, an exception is thrown. The exception message contains the content of status_code and message.
request_idstring
The unique ID of this call.
The Java SDK returns this parameter as requestId
codestring
The error code. This is empty if the call is successful.
Only the Python SDK returns this parameter.
outputobject
Information about the call result.
Properties
textstring
This parameter is currently fixed to null.
finish_reasonstring
The reason the model stopped generating. The possible cases are:
null if generation is in progress.
The model output ends naturally with stop.
length if the generation ended because it was too long.
choicesarray
The model's output information.
Properties
finish_reasonstring
The possible cases are:
null if generation is in progress.
The model output naturally ends with stop.
length if the generation ended because it was too long.
messageobject
The message object output by the model.
Properties
rolestring
The role of the output message. The value is fixed to assistant.
contentstring
The model's output result
audiostring
This parameter is currently fixed to null.
usageobject
Token information used by this request.
Properties
input_tokensinteger
The number of input tokens.
output_tokensinteger
The number of output tokens.
image_tokensinteger
This field is returned when the input content includes an image. It is the length of the user's input image content after being converted to tokens.
charactersinteger
This parameter is currently fixed to null.
input_tokens_detailsobject
A fine-grained categorization of the input tokens.
Properties
image_tokensinteger
The number of tokens corresponding to the model's input image.
text_tokensinteger
The number of tokens corresponding to the model's input text.
output_tokens_detailsobject
A fine-grained categorization of the output tokens.
Properties
text_tokensinteger
The number of tokens corresponding to the model's input text.
total_tokensinteger
The total number of tokens used, which is the sum of input_tokens and output_tokens.
{
"status_code": 200,
"request_id": "b74b3a25-3968-4059-8c44-63d793c07f02",
"code": "",
"message": "",
"output": {
"text": null,
"finish_reason": null,
"choices": [
{
"finish_reason": "stop",
"message": {
"role": "assistant",
"content": [
{
"text": "```json\n{\"thought\": \"The user wants to open the browser. I see a Google Chrome icon in the screenshot. It is the last one in the top-right row. The next step should be to click this Chrome browser icon to launch it.\", \"action\": \"CLICK\", \"parameters\": {\"x\": 1086, \"y\": 127}}\n```"
}
]
}
}
],
"audio": null
},
"usage": {
"input_tokens": 2021,
"output_tokens": 78,
"characters": 0,
"image_tokens": 1244,
"input_tokens_details": {
"image_tokens": 1244,
"text_tokens": 777
},
"output_tokens_details": {
"text_tokens": 78
},
"total_tokens": 2099
}
}
Error codes
If the model call fails and an error message is returned, see Error messages to resolve the issue.
