Connect to the photo-based Q&A agent using the HTTP protocol

Updated at:

Terminology

Direct path: A request path that bypasses nodes such as automatic speech recognition (ASR), intent recognition, and text-to-speech (TTS). Instead, it sends the request directly to the agent and returns the agent's response.

Scenarios

You can send images directly to the agent for analysis without intent recognition or voice processing. This approach is ideal for camera-equipped products that require image understanding, such as photo-based learning devices or educational tablets with visual question-answering features.

Prerequisites

Activate Alibaba Cloud Model Studio and obtain an API key

For more information, see Obtain an API key. The API key is your authentication credential for Alibaba Cloud Model Studio.

Enable the photo-based Q&A direct path in the console

  1. In the Multimodal Development Suite, create a multimodal interactive application. Select the Full-Featured template (do not select the Vision-only template) and disable interactive voice response.

⚠️ Note: Only disable interactive voice response. Keep intent recognition and text model configurations enabled.

  1. Disable conversation carryover, knowledge base, web search, and long-term memory configurations.

  2. Clear all skill configurations and keep only the photo-based Q&A agent in the agent configuration.

  3. Configure the photo-based Q&A agent. Leave the launch instruction empty. We recommend that you select the “Balanced Visual Understanding” model. You can customize the prompt as needed.

For testing image description scenarios, you can use the following prompt. This prompt is for demonstration purposes only and should be customized for your scenario.

你是一个专业的图像分析与描述助手,请基于图片内容生成简洁、准确且信息丰富的文字描述。请确保描述涵盖以下要点:
主要物体和场景:明确指出画面中的核心元素及其位置、颜色、形状、数量,并说明场景类型(如室内、室外、城市、自然等)。
人物特征与动作(如有):描述人物的衣着、表情、姿势、动作,互动关系,以及可能的身份或角色。
文字信息(如有):描述图片中出现的文字、字体样式,以及文字的含义或用途(如标牌、广告、标题)。
环境与氛围:说明背景细节(如天气、光线、季节)、整体色调,以及画面传递的情绪或故事感。
视角与构图:可补充拍摄角度(俯视、平视、仰视)、景深效果、焦点位置等。
不进行主观评价,只提供客观、可观察的事实。
描述要求:
使用简洁清晰的中文,避免冗长和模糊词汇。
确保突出关键信息,同时覆盖细节。
如果画面元素之间存在明显关系(如因果、动作链),请在描述中说明。
不加入个人主观猜测或情绪判断(除非氛围明显)。
  1. After you complete the configuration, click Publish in the upper-right corner. You must publish the application before you can test it.

Connect using the HTTP protocol

Request parameters

Top-level parameter

Secondary parameters

Type

Required

Description

model

string

Yes

Alibaba Cloud Model Studio model name. Always use "multimodal-dialog". Copy this value directly.

input

directive

string

Yes

Directive name: Request

app_id

string

Yes

Your application ID (see Obtain an app ID). You can find it on the My Applications page in the Multimodal Interactive Development Suite console.

dialog_id

string

No

Dialog ID. If left blank, a new dialog starts. The server generates a dialog ID automatically and returns it in the response. Example format: "12345678-1234-1234-1234-1234567890ab" (36 characters total). To continue a previous dialog, pass the dialog_id previously returned by the server.

text

string

Yes

Text to process. In this scenario, leave it as an empty string ("").

parameters

client_info

object

Yes

See the table below for parameters under parameters.client_info.

images

list[]

No

Image data for analysis. Only multimodal applications support image-based Q&A. See the table below for parameters under parameters.images.

biz_params

object

No

Configure as needed. See the table below for parameters under parameters.biz_params.

parameters.client_info parameters

Top-level parameter

Secondary parameters

Type

Required

Description

user_id

string

Yes

End user ID. Generate this ID according to your business rules to enable customized features per end user. Maximum length: 36 characters.

device

uuid

string

No

Globally unique client device ID. Generate and pass this ID into the SDK yourself. Maximum length: 40 characters. One end user can have multiple devices, each with a different uuid but the same user_id.

parameters.images parameters

Top-level parameters

Type

Required

Description

type

string

Yes

Image type. Supported values: base64 or url.

value

string

Yes

Image content.

  • If type is base64, this field contains the image’s base64-encoded string.

  • If type is url, this field contains the image’s URL.

parameters.biz_params parameters

Top-level parameter

Secondary Parameters

Type

Required

Description

commands[i]

name

string

Yes

Indicates direct routing to the agent. In this scenario, always use “agent_command”. Copy this value directly.

exec_params

object

Yes

Specifies the target agent. In this scenario, this value is fixed. See the commands.exec_params parameter table below for details.

commands.exec_params parameters

exec_params

app_id

string

Yes

Specifies the target agent. In this scenario, always use "visual_qa". Copy this value directly.

intent

string

Yes

Specifies the agent action. In this scenario, always use "open_visual_qa". Copy this value directly.

Sample request

{
    "model": "multimodal-dialog",
    "input": {
        "directive": "Request",
        "app_id": "xxxxxx",
        "text": ""
    },
    "parameters": {
        "images": [
            {
                "type": "url",
                "value": "img_url"
            }
        ],
        "client_info": {
            "user_id": "test-251222",
            "device": {
                "uuid": "test-251222-123"
            }
        },
        "biz_params": {
            "commands": [
                {
                    "name": "agent_command",
                    "exec_params": {
                        "app_id": "visual_qa",
                        "intent": "open_visual_qa"
                    }
                }
            ]
        }
    }
}

curl request example

curl --location "https://dashscope.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation" \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer your_api_key' \
  --header 'X-DashScope-SSE: enable' \
  --data '{
    "model": "multimodal-dialog",
    "input": {
      "directive": "Request",
      "app_id": "xxxxxx",
      "text": ""
    },
    "parameters": {
      "images": [
        {
          "type": "url",
          "value": "img_url"
        }
      ],
      "client_info": {
        "user_id": "test-251222",
        "device": {
          "uuid": "test-251222-123"
        }
      },
      "biz_params": {
        "commands": [
          {
            "name": "agent_command",
            "exec_params": {
              "app_id": "visual_qa",
              "intent": "open_visual_qa"
            }
          }
        ]
      }
    }
  }'

Important

  1. Replace your_api_key (API key), app_id, and img_url in the request.
  2. Set the text field to an empty string (""). Otherwise, the system will trigger intent recognition.

Text response event fields

Top-level parameter

Secondary Parameters

Type

Required

Description

output

event

string

Yes

Event name: RespondingContent

dialog_id

string

Yes

Dialog ID

round_id

string

Yes

ID of the current conversation round

llm_request_id

string

Yes

Request ID for the LLM call

text

string

Yes

Text output from the system. Streamed as incremental updates.

spoken

string

Yes

Text used for speech synthesis. Streamed as incremental updates.

finished

bool

Yes

Indicates whether output has finished

finish_reason

string

No

Reason for completion. Currently supports only one value:

  • stop: Indicates normal completion

extra_info

object

No

Additional extended information. Currently supports:

  • agent_info: Agent information. See below.

extra_info.agent_info parameters

Top-level parameter

Secondary parameters

Type

Required

Description

round

string

Yes

Conversation round

device

device_id

string

Yes

The device.uuid used in the request

intent_infos

intent

string

Yes

Agent used. In this scenario, always "visual_qa".

domain

string

Yes

Agent used. In this scenario, always "visual_qa".

Sample response

{
    "output": {
        "round_id": "xxx",
        "llm_request_id": "xxx",
        "extra_info": {
            "agent_info": {
                "round": 1,
                "device": {
                    "device_id": "test-251222-123"
                },
                "intent_infos": [
                    {
                        "intent": "visual_qa",
                        "domain": "visual_qa"
                    }
                ]
            },
            "query": ""
        },
        "dialog_id": "6c4340eb-xxx-4055-8d66-0794df7986e0",
        "spoken": "你好",
        "finished": false,
        "text": "你好",
        "event": "RespondingContent"
    },
    "request_id": "xxx"
}