Generate a conversation

Request syntax

POST /web-search-agent/chat/completions HTTP/1.1

Request parameters

• Note: If you provide dynamic parameters in the request, they override the corresponding status values in the application configuration.

Response parameters

Execution stage enumeration

• Due to model limitations, the step_change value may be absent. Use the persistent step flag whenever possible.

• For an empty packet, the step, step_change, and group fields may be absent.

• The plan and generation processes each consist of a *_start event and a *_end event.

• A tool_call consists of three events: tool_call_start, tool_calling, and tool_return.

• tool_call_start indicates the beginning of a tool call. The system emits the tool_calling event after gathering the complete tool call parameters; this event contains the parameters. tool_return contains the structured result from the tool.

Step value and tool name mapping

When a tool is called, the ["extra"]["step"] field contains a value, such as tool_calling_xx, that identifies the tool used. The following tables map each step value to its corresponding tool name.

Tool call status

"step_change": "tool_call_start", "step": "tool_calling_search","step_info": "Starting to call the web search tool"
"step_change": "tool_calling", "step": "tool_calling_search","step_info": "Calling the web search tool"
"step_change": "tool_return", "step": "tool_calling_search",  "step_info": "Call to the web search tool completed: retrieved 10 web pages"

Rich media message protocol

When rich media is enabled, the model interleaves text with images by embedding them in the message body with the HTML <img> tag, as shown in the example below:

<img src=\"xxx\" data-type=\"image\" data-url=\"xxx\" data-title=\"Hangzhou after the typhoon: Blue sky and white clouds as refreshing as a comic book️\" alt=\"Hangzhou clear day blue sky\" width=\"1080\" height=\"1410\">

The attributes in this tag are defined as follows:

• data-type: Specifies the data type. A value of image indicates image data.

• src: The URL of the image.

• data-url: The URL of the image's source content page.

• data-title: The image title.

• width/height: The image's width and height.

For messages containing an image, the additional_kwargs field contains the following two fields:

• data_type: For a rich media message containing an image, this value is set to image.

• data_json: A JSON array of all images related to the content. Only one of these images is displayed in the message body.

  • In data_json, each image object contains the following fields:

    • idx: The image ID.

    • url: The source URL of the image's content page.

    • title: The image title.

    • published: The publication date of the source content.

    • image_info["url"]: The URL of the image file.

    • image_info["width"]: The image width.

    • image_info["height"]: The image height.

{"idx": 15, "query": "Hangzhou clear day blue sky", "url": "xxx", "title": "xx", "published": "2024-11-04 03:04:04", "image_info": {"url": "xxx", "width": 1080, "height": 1410}}

The following example shows a rich media message delivered in three message packets:

data: {
    "status_code": 200,
    "code": "",
    "message": "",
    "output": {
        "choices": [
            {
                "finish_reason": "",
                "message": {
                    "content": "Sightseeing.\n\n",
                    "additional_kwargs": {},
                    "response_metadata": {},
                    "type": "ai",
                    "name": null,
                    "id": "run--019e63e4-e728-7421-b3e5-57b408d6d4b0",
                    "example": false,
                    "tool_calls": [],
                    "invalid_tool_calls": [],
                    "usage_metadata": null,
                    "tool_call_chunks": [],
                    "reasoning_content": "",
                    "role": "assistant",
                    "extra": {
                        "group": "generating",
                        "step_change": "",
                        "step": "generating"
                    }
                }
            }
        ]
    },
    "usage": null,
    "request_id": "chenwen-test-web-search-01"
}

data: {
    "status_code": 200,
    "code": "",
    "message": "",
    "output": {
        "choices": [
            {
                "finish_reason": "",
                "message": {
                    "content": "<img src=\"http://miaobi-lite.bj.bcebos.com/miaobi/5mao/b%275Y%2Bw6aOO6L%2BH5ZCO55qE5Zu%2B54mHXzE3MzA2NTY4MTMuMTQ0MDA1NQ%3D%3D%27/0.png\" data-type=\"image\" data-url=\"http://mbd.baidu.com/newspage/data/dtlandingsuper?nid=dt_4158884509151638702\" data-title=\"Hangzhou after the typhoon: Blue sky and white clouds as refreshing as a comic book\" alt=\"Hangzhou clear day blue sky\" width=\"1080\" height=\"1410\">\n\n\n",
                    "additional_kwargs": {
                        "data_type": "image",
                        "data_json": [
                            {
                                "idx": 15,
                                "query": "Hangzhou clear day blue sky",
                                "url": "http://mbd.baidu.com/newspage/data/dtlandingsuper?nid=dt_4158884509151638702",
                                "title": "Hangzhou after the typhoon: Blue sky and white clouds as refreshing as a comic book",
                                "published": "2024-11-04 03:04:04",
                                "image_info": {
                                    "url": "http://miaobi-lite.bj.bcebos.com/miaobi/5mao/b%275Y%2Bw6aOO6L%2BH5ZCO55qE5Zu%2B54mHXzE3MzA2NTY4MTMuMTQ0MDA1NQ%3D%3D%27/0.png",
                                    "width": 1080,
                                    "height": 1410
                                }
                            },
                            {
                                "idx": 16,
                                "query": "Hangzhou clear day blue sky",
                                "url": "http://m.dianping.com/ugcdetail/175828244?sceneType=0&bizType=29&msource=baiduappugc",
                                "title": "The sky in Hangzhou is so beautiful!",
                                "published": "2023-07-26 00:00:00",
                                "image_info": {
                                    "url": "http://qcloud.dpfile.com/pc/VWohABccM5j2sRMTT4YdH7qapT46U8bLlM3Wp-xKeMm3OVoihIkmurvV3052Y9Xt.jpg",
                                    "width": 2048,
                                    "height": 2731
                                }
                            },
                            {
                                "idx": 17,
                                "query": "Hangzhou clear day blue sky",
                                "url": "http://m.dianping.com/ugcdetail/175964222?sceneType=0&bizType=29&msource=baiduappugc",
                                "title": "The blue sky and white clouds in Hangzhou are beautiful, but you always get a tan",
                                "published": "2023-07-27 00:00:00",
                                "image_info": {
                                    "url": "http://qcloud.dpfile.com/pc/6ScKLghe1ZcE15OpzbomPigY1VkO6o_UTx9z6UB_BoNtf5OGAsnbF-AFIAJjozRl.jpg",
                                    "width": 2048,
                                    "height": 2731
                                }
                            }
                        ]
                    },
                    "response_metadata": {},
                    "type": "ai",
                    "name": null,
                    "id": "run--019e63e4-e728-7421-b3e5-57b408d6d4b0",
                    "example": false,
                    "tool_calls": [],
                    "invalid_tool_calls": [],
                    "usage_metadata": null,
                    "tool_call_chunks": [],
                    "reasoning_content": "",
                    "role": "assistant",
                    "extra": {
                        "group": "generating",
                        "step_change": "",
                        "step": "generating"
                    }
                }
            }
        ]
    },
    "usage": null,
    "request_id": "chenwen-test-web-search-01"
}

data: {
    "status_code": 200,
    "code": "",
    "message": "",
    "output": {
        "choices": [
            {
                "finish_reason": "",
                "message": {
                    "content": "**Summary**:",
                    "additional_kwargs": {},
                    "response_metadata": {},
                    "type": "ai",
                    "name": null,
                    "id": "run--019e63e4-e728-7421-b3e5-57b408d6d4b0",
                    "example": false,
                    "tool_calls": [],
                    "invalid_tool_calls": [],
                    "usage_metadata": null,
                    "tool_call_chunks": [],
                    "reasoning_content": "",
                    "role": "assistant",
                    "extra": {
                        "group": "generating",
                        "step_change": "",
                        "step": "generating"
                    }
                }
            }
        ]
    },
    "usage": null,
    "request_id": "chenwen-test-web-search-01"
}

Render local life POI cards

When local life is enabled, the model returns location-based Point of Interest (POI) search results. If rich text and image display is also enabled, the POI search results include images. Images in the message body use the standard HTML image format, as shown in the following example:

<img src=\"https://aos-comment.amap.com/B0FFGD1PB5/comment/content_media_external_file_9739_ss__1749454867961_64014666.jpg\" data-type=\"poi\" data-url=\"https://www.amap.com/detail/B0FFGD1PB5\" data-title=\"Wu山景区吴山广场-吴山文化公园\" alt=\"Wu山景区吴山广场-吴山文化公园\" width=\"400\" height=\"auto\"/>\n

The attributes in this tag are defined as follows:

• data-type: The data type. A value of poi indicates a map card.

• src: The URL of the first image for the card.

• data-url: The Amap URL that corresponds to the card.

• data-title: The card title.

• width/height: The width and height of the image.

In model messages that contain images, the additional_kwargs field includes the following fields:

• data_type: For map card images, the value is poi.

• data_json: A JSON object of all related map cards. Although the message body displays only one card, this field contains all relevant cards.

In data_json, each map card uses the following fields:

Custom fields in business

The following example shows the response when local life and rich text and image display are enabled.

"additional_kwargs": {
  "data_type": "poi",
  "data_json": [
    {
      "address": "Siyi Pavilion, Wushan Scenic Area (150m walk from Exit D of Wushan Square Metro Station)",
      "distance": "352",
      "business": {
        "opentime_today": "Open 24 hours",
        "keytag": "City Park",
        "rating": "3.8",
        "business_area": "Wushan",
        "tel": "0571-8703****",
        "rectag": "City Park",
        "opentime_week": "Monday to Sunday 00:00-24:00"
      },
      "pcode": "330000",
      "adcode": "330102",
      "pname": "Zhejiang Province",
      "cityname": "Hangzhou",
      "type": "Scenic Spot; Park and Square; Park",
      "photos": [
        {
          "title": "",
          "url": "https://aos-comment.amap.com/B0FFGD1PB5/comment/content_media_external_file_9739_ss__1749454867961_64014666.jpg"
        },
        {
          "title": "",
          "url": "https://aos-comment.amap.com/B0FFGD1PB5/comment/content_media_external_images_media_1000008441_ss__1751303656358_97699056.jpg"
        },
        {
          "title": "",
          "url": "https://aos-comment.amap.com/B0FFGD1PB5/comment/content_media_external_file_9746_ss__1749454867953_49866303.jpg"
        }
      ],
      "typecode": "110101",
      "adname": "Shangcheng District",
      "citycode": "0571",
      "navi": {
        "navi_poiid": "H51F022002_375744",
        "entr_location": "120.163399,30.239046",
        "gridcode": "4520218310"
      },
      "name": "Wushan Scenic Area Wushan Square - Wushan Cultural Park",
      "indoor": {
        "indoor_map": "0"
      },
      "location": "120.163789,30.238795",
      "id": "B0FFGD1PB5",
      "distance_text": "352m from Wushan Xiaoputuo Wushan Scenic Area, Ziyang Subdistrict",
      "from": "Wushan Xiaoputuo Wushan Scenic Area, Ziyang Subdistrict, Shangcheng District, Hangzhou, Zhejiang Province"
    }
  ]
}

Multimodal image Q&A

The Web Search Agent multimodal API lets you include images in your conversations. Provide an image URL with your question, and the Agent will interpret the image, call the necessary tools, and generate a response.

Examples

{
    "input": {
        "messages": [
            {
                "role": "user", 
                "content": "What is the current date?"
            }
        ]
    },
    "stream": true,
    "parameters": {
        "agent_options": {
            "agent_id": "aid-xxx",
            "agent_version": "beta"
        }
    }
}

{
    "input": {
        "messages": [
            {
                "role": "user",
                "content": [
                    {
                        "image_url": {
                            "url": "http://other-general-huabei2.oss-cn-beijing.aliyuncs.com/upload/36e553b350e98ba81f6b33b08833a784.png"
                        },
                        "type": "image_url"
                    },
                    {
                        "text": "What animal is this, and in which regions does it typically live?",
                        "type": "text"
                    }
                ]
            }
        ]
    },
    "stream": true,
    "parameters": {
        "agent_options": {
            "agent_version": "beta",
            "agent_id": "aid-xxx"
        }
    }
}

data: {
    "status_code": 200,
    "code": "",
    "message": "",
    "output": {
        "choices": [
            {
                "finish_reason": "",
                "message": {
                    "content": "xxx",
                    "additional_kwargs": {},
                    "response_metadata": {
                        "headers": {
                            "vary": "Origin",
                            "x-request-id": "ca7a41ad-3994-9dcf-adf6-f7aa56bec62b",
                            "content-type": "text/event-stream;charset=UTF-8",
                            "x-dashscope-call-gateway": "true",
                            "req-cost-time": "508",
                            "req-arrive-time": "1773800485527",
                            "resp-start-time": "1773800486035",
                            "x-envoy-upstream-service-time": "506",
                            "date": "Wed, 18 Mar 2026 02:21:25 GMT",
                            "server": "istio-envoy",
                            "transfer-encoding": "chunked"
                        }
                    },
                    "type": "ai",
                    "name": null,
                    "id": "run--019cfebf-7194-78b3-ad12-c3df6b6423b1",
                    "example": false,
                    "tool_calls": [],
                    "invalid_tool_calls": [],
                    "usage_metadata": null,
                    "tool_call_chunks": [],
                    "reasoning_content": "",
                    "role": "assistant",
                    "extra": {
                        "group": "generating",
                        "step_change": "generation_start",
                        "step": "generating"
                    }
                }
            }
        ]
    },
    "usage": null,
    "request_id": "xxxx-xxxx"
}

Code sample

# coding=utf-8

import os
import json
import requests

chat_completions_url = 'https://dashscope.aliyuncs.com/api/v2/apps/web-search-agent/chat/completions'

headers = {
    'Authorization': f'Bearer {os.getenv("DASHSCOPE_API_KEY", "")}',  # Configure the API key.
    'Content-Type': 'application/json'
}


if __name__ == "__main__":
    params = {
        "input": {
            "messages": [{"role": "user", "content": "What are the mainstream multimodal models in China? Please analyze their performance and effectiveness."}]  # Set the request message.
        },
        "parameters": {
            "agent_options": {  # Set agent options.
                "agent_id": "${agent_id}",  # The application ID, found on the Application Management page (e.g., aid-8fd***e00).
                "agent_version": "${agent_version}"  # The application version. Use 'beta' for the beta version or 'release' for the release version.
            }
        },
        "stream": True
    }
    
    response = requests.post(chat_completions_url, headers=headers, json=params, stream=True)
    
    resultlist = []
    stage = ''
    action = ''
    content = ''
    reasoning_content = ''
    for chunk in response.iter_lines():
        if chunk:
            chunk_str = chunk.decode('utf-8').strip()
            if chunk_str.startswith('data:'):
                json_str = chunk_str[len('data:'):].strip()
                try:
                    obj = json.loads(json_str)
                    # Check for a service error.
                    if obj.get('code') != '200':
                        print("Service error:", obj)
                    # Extract the message body.
                    msg = obj.get('output', {}).get('choices', [{}])[0].get('message', {})
                    extra_flags = msg.get('extra', {})  # Get the 'extra' field, which contains model state flags.
    
                    if stage != extra_flags.get('group', ''):  # Get the current model stage.
                        print(f"agent stage: {extra_flags.get('group', '')}")
                    stage = extra_flags.get('group', '')
    
                    if action != extra_flags.get('step', '') and extra_flags.get('step', ''):  # Get the current model step.
                        print(f"agent action: {extra_flags.get('step', '')}")
                    action = extra_flags.get('step', '')
    
                    role = msg.get('role', '')  # Get the role, e.g., 'assistant' or 'tool'.
                    content = msg.get('content')  # Get the generated content.
                    toolcalls = msg.get('tool_calls', [])  # Get the tool calls.
                    if toolcalls:
                        print(f'{toolcalls}')
    
                    if role == "tool":
                        print("\\n" + content + "\\n", end='')  # Wrap the tool output in newlines for readability.
                    else:
                        print(content, end='')  # Streaming output.
                    # Save the results as needed.
                    resultlist.append(obj)
                except Exception as e:
                    print("Parse error:", e)

import java.io.*;
import java.net.*;
import java.util.*;
import com.alibaba.fastjson.*;
import java.nio.charset.StandardCharsets;


public class WebSearchStreamDemo {

    // Configure the API key.
    public final static String CHAT_COMPLETIONS_URL = "https://dashscope.aliyuncs.com/api/v2/apps/web-search-agent/chat/completions";
    public final static String API_KEY = System.getenv("DASHSCOPE_API_KEY");


    public static void main(String[] args) throws Exception {
        // Build the parameters.
        Map<String, Object> params = new HashMap<>();
        // input.messages
        List<Map<String, Object>> messages = new ArrayList<>();
        Map<String, Object> msgObj = new HashMap<>();
        msgObj.put("role", "user");
        msgObj.put("content", "${prompt}");
        messages.add(msgObj);
        // input
        Map<String, Object> input = new HashMap<>();
        input.put("messages", messages);
        // parameters.agent_options
        Map<String, Object> agentOptions = new HashMap<>();
        agentOptions.put("agent_id", "${agent_id}");// The application ID, found on the Application Management page (e.g., aid-8fd***e00).
        agentOptions.put("agent_version", "${agent_version}"); // The application version. Use 'beta' for the beta version or 'release' for the release version.
        // parameters
        Map<String, Object> parameters = new HashMap<>();
        parameters.put("agent_options", agentOptions);

        params.put("input", input);
        params.put("parameters", parameters);
        params.put("stream", true);

        String body = JSON.toJSONString(params);

        // Create the HTTP request.
        URL apiUrl = new URL(CHAT_COMPLETIONS_URL);
        HttpURLConnection conn = (HttpURLConnection) apiUrl.openConnection();
        conn.setRequestMethod("POST");
        conn.setDoOutput(true);
        conn.setRequestProperty("Authorization", "Bearer " + API_KEY);
        conn.setRequestProperty("Content-Type", "application/json");

        // Send the request body.
        try (OutputStream os = conn.getOutputStream()) {
            os.write(body.getBytes(StandardCharsets.UTF_8));
        }

        // Handle the streaming response.
        InputStream inputStream = conn.getInputStream();
        BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream, StandardCharsets.UTF_8));
        String line;
        String stage = "";
        String action = "";
        List<JSONObject> resultList = new ArrayList<>();

        while ((line = reader.readLine()) != null) {
            if (!line.trim().isEmpty()) {
                String chunkStr = line.trim();
                if (chunkStr.startsWith("data:")) {
                    String jsonStr = chunkStr.substring(5).trim();
                    try {
                        JSONObject obj = JSON.parseObject(jsonStr);
                        // Check for a service error.
                        if (!"200".equals(obj.getString("code"))) {
                            System.out.print("Service error: " + obj);
                        }
                        // Extract the 'message' object from the response.
                        JSONObject msg = null;
                        if (obj.containsKey("output")) {
                            JSONObject output = obj.getJSONObject("output");
                            if (output != null && output.containsKey("choices")) {
                                JSONArray choices = output.getJSONArray("choices");
                                if (choices != null && !choices.isEmpty()) {
                                    JSONObject firstChoice = choices.getJSONObject(0);
                                    if (firstChoice.containsKey("message")) {
                                        msg = firstChoice.getJSONObject("message");
                                    }
                                }
                            }
                        }
                        if (msg == null) {
                            continue;
                        }

                        // Get the 'extra' field, which contains model state flags.
                        JSONObject extraFlags = msg.containsKey("extra") && msg.get("extra") != null
                                ? msg.getJSONObject("extra") : new JSONObject();

                        // Get the agent stage.
                        String stageNew = extraFlags.containsKey("group") && extraFlags.get("group") != null
                                ? extraFlags.getString("group") : "";
                        if (!stage.equals(stageNew)) {
                            System.out.println("agent stage: " + stageNew);
                        }
                        stage = stageNew;

                        // Get the agent action.
                        String actionNew = extraFlags.containsKey("step") && extraFlags.get("step") != null
                                ? extraFlags.getString("step") : "";
                        if (!action.equals(actionNew) && !actionNew.isEmpty()) {
                            System.out.println("agent action: " + actionNew);
                        }
                        action = actionNew;

                        String role = msg.containsKey("role") && msg.get("role") != null
                                ? msg.getString("role") : "";

                        Object contentObj = msg.get("content");
                        String content = null;
                        boolean isContentString = false;
                        // Check if the content is a string.
                        if (contentObj instanceof String) {
                            content = contentObj.toString();
                            isContentString = true;
                        }

                        // If 'content' is empty, use 'reasoning_content' as a fallback.
                        if (isContentString && content.isEmpty()) {
                            Object reasoningContentObj = msg.get("reasoning_content");
                            if (reasoningContentObj instanceof String) {
                                content = reasoningContentObj.toString();
                            }
                        }

                        // Process tool calls.
                        if (msg.containsKey("tool_calls") && msg.get("tool_calls") instanceof List) {
                            JSONArray toolCalls = msg.getJSONArray("tool_calls");
                            if (!toolCalls.isEmpty()) {
                                System.out.println(toolCalls);
                            }
                        }

                       if ("tool".equals(role)) {
                            System.out.print("\\n" + content + "\\n");
                        } else {
                            System.out.print(content);
                        }

                        // Save the results as needed.
                        resultList.add(obj);
                    } catch (Exception e) {
                        System.out.println("Parse error: " + e);
                    }
                }
            }
        }
        reader.close();
    }
}