IPC video summarization and retrieval
For IPC camera scenarios, connect to the physical world perception agent over HTTP to generate video summaries, structured tags, and natural language video retrieval.
Overview
For consumer IP cameras (IPC), this feature provides intelligent understanding and semantic retrieval of surveillance data. The system uses a multimodal large model to automatically summarize and structurally tag video footage, and uses vector storage and retrieval to enable natural language queries over historical surveillance, helping users quickly locate key moments.
- Video summarization: Automatically generates scene descriptions and structured tags for surveillance footage, supports custom tags, and persists them as semantic vectors.
- Video retrieval: Supports natural language queries over historical events, combines vector similarity matching with LLM semantic reranking, and returns results by similarity threshold or Top-K.
ImportantThis topic covers two invocation modes: vector_mode=store generates and stores a summary, and vector_mode=retrieve retrieves historical video. Both use the same HTTP endpoint and are distinguished by vector_mode.
Prerequisites
- Alibaba Cloud Model Studio is activated.
- An API key is available as the credential for Model Studio. See Obtain and configure an API key.
Console configuration
Step 1: Create an application
- Open the Multimodal Interaction Development Suite and log on. Click Activate for free in the upper-right corner and complete service activation.
- Click Custom create to open the application configuration page.
Step 2: Configure the application
- Turn off Voice interaction and keep Intent recognition and Text model turned on. Turn off Conversation carryover, Knowledge base, Web search, and Long-term memory. Delete all instructions, plugins, and MCP services.
- Click the add button for Model Studio applications and add the Physical World Perception application. If you use it for the first time, activate the service.
- After activation, refresh the application configuration page, select Physical World Perception, and click OK. Do not select other applications.
- The application provides IPC scene visual analysis by default. You can switch it with the configuration button.
NoteKeep only Physical World Perception in the application configuration. Do not select other applications, otherwise the invocation path in this topic may be affected.
Step 3: Publish the application
After configuration, click Publish in the upper-right corner. The application must be published before testing.
Return to the previous page, click Copy, and save the application ID for later invocation.
HTTP protocol integration
Endpoint: https://dashscope.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation. Set the X-DashScope-SSE: enable header. Results are returned as an SSE event stream.
Request parameters
| Parameter | Type | Required | Description |
|---|---|---|---|
model | string | Yes | Model name. Fixed to multimodal-dialog. |
input.directive | string | Yes | Instruction name. Fixed to Request. |
input.app_id | string | Yes | Application ID created in the console. You can view it on the My Apps page of the Multimodal Interaction Development Suite console. See Obtain an App ID and workspace ID. |
input.text | string | Yes | Text to process. In this topic, set it to an empty string "". |
parameters.client_info | object | Yes | Client information. See below. |
parameters.biz_params | object | Yes | Business parameters. See below. |
parameters.client_info
| Parameter | Type | Required | Description |
|---|---|---|---|
user_id | string | Yes | End-user ID generated by your business logic to provide customized features for different end users. Maximum length: 36 characters. Note: This field is required for summarization and retrieval. Within the same app_id, each user's user_id must be unique. Videos with the same user_id are placed in the same retrieval space. If two end users use the same user_id, they share retrieval resources. |
device.uuid | string | Yes | Device ID generated by your business logic and passed to the SDK. Maximum length: 40 characters. One end user (user_id) can have multiple devices. Each device has a different uuid, but the user_id is the same.Note: This field is required for summarization and retrieval. Within the same app_id, different devices of the same user_id must have different uuid values; otherwise, the devices cannot be distinguished. |
parameters.biz_params
| Parameter | Type | Required | Description |
|---|---|---|---|
commands[].name | string | Yes | Fixed to agent_command. |
commands[].exec_params | object | Yes | Specifies the agent. See below. |
user_defined_params.physical_sense | object | Yes | Parameters passed to the physical world perception model. |
commands[].exec_params
| Parameter | Type | Required | Description |
|---|---|---|---|
app_id | string | Yes | Fixed to physical_sense. |
intent | string | Yes | Fixed to open_physical_sense. |
slots | list[] | Yes | Agent slot information. |
slots
| Parameter | Type | Required | Description |
|---|---|---|---|
name | string | Yes | Slot name. |
norm_value | string | Yes | Slot value. |
To connect to the physical world perception agent, fill in the scene slot. scene is required and supports the following values:
embodied: Embodied intelligenceipc: IPC scenario (used in this topic)auto_driving: Autonomous driving
{
"slots": [
{
"name": "scene",
"norm_value": "ipc"
}
]
}
parameters.biz_params.user_defined_params.physical_sense.user_prompt_params
| Parameter | Type | Required | Description |
|---|---|---|---|
param.sub_scene | string | Yes | Fixed to vector in this scenario. |
param.vector_mode | string | Yes | Summary or retrieval. store: summary; retrieve: retrieval. Other values return an error. |
param.vector_store_config | object | No | Required when vector_mode=store. Summary configuration. See below. |
param.vector_retrieve_config | object | No | Required when vector_mode=retrieve. Retrieval configuration. See below. |
param.format | string | Yes | Resource mode. Valid values: image and video. |
param.prompt | string | Yes | Set to an empty string when vector_mode=store. When vector_mode=retrieve, specify the natural language query. |
param.images | array | Conditional | Resource array. Required for summarization. See below. |
param.oss | object | No | OSS configuration. See below. |
param.frame_enabled | boolean | No | Whether to use an image sequence. Takes effect only when format=video. |
param.high_resolution | boolean | No | Whether to enable high resolution. Default: off. |
param.max_pixels | integer | No | Maximum number of pixels. Valid when vector_mode=store. Use the example value 230400. |
param.fps | double | Conditional | Frame extraction rate. Value range: [0.1, 10.0]. Required when vector_mode=store and format=video. Use the example value 0.2. We recommend that you do not change it. A higher value increases costs. |
param.max_frames | integer | No | Maximum number of extracted frames. |
param.max_tokens | integer | No | Maximum number of output tokens. |
param.temperature | float | No | Temperature. Default: 0.0. |
param.version | string | No | Version number. Leave empty if no special convention applies. |
param.images
| Parameter | Type | Required | Description |
|---|---|---|---|
type | string | Yes | Resource type. Valid values: base64 and url. |
value | string | Yes | When type=base64, the base64 string of the resource. When type=url, the resource URL. |
Note
- The recommended resolution range is 640×480 to 1920×1080.
- When
type=url, the URL must start withhttps.httpis not supported.
param.oss
| Parameter | Type | Required | Description |
|---|---|---|---|
internal | boolean | No | Whether to use an internal network URL. Internal URLs can be downloaded only from the same region. |
region | string | No | Valid when internal=true. The OSS region. |
vector_store_config
| Parameter | Type | Required | Description |
|---|---|---|---|
tags | object | No | Custom tags. They are injected into the prompt. |
tags.tags_object | list<string> | No | Object tags. |
tags.tags_action | list<string> | No | Action tags. |
extra_data | object | No | Custom data returned with retrieval results. |
pre_process | object | No | Preprocessing configuration. |
pre_process.type | string | No | Only dts_to_mp4 is supported. Other values return an error. |
vector_retrieve_config
| Parameter | Type | Required | Description |
|---|---|---|---|
pipeline | string | Yes | Pipeline used for vector retrieval. See below. |
similarity_threshold | string | Conditional | Required when pipeline=threshold. Must be parseable as a double in (0,1]; otherwise, an error is returned. |
similarity_top_size | string | Conditional | Required when pipeline=top_size. Must be parseable as a positive integer ≤ 1000; otherwise, an error is returned. |
vector_retrieval_time | object | No | Retrieval time range. If empty, the current day is used. |
vector_retrieval_time.start_time | string | No | Start time in yyyy-MM-dd HH:mm:ss format. |
vector_retrieval_time.end_time | string | No | End time in yyyy-MM-dd HH:mm:ss format. |
Time range rules:
start_timeandend_timemust be specified together. Specifying only one returns an error.end_timemust be later thanstart_time.- The default time zone is
Asia/Shanghai. - Data is retained for 7 days by default.
pipeline
| Value | Description |
|---|---|
threshold | Performs retrieval by similarity threshold and returns vectors whose similarity meets the threshold. |
top_size | Returns the top size vectors by similarity. |
default_topk | Default pipeline, equivalent to top_size=10. |
llm_rerank | Retrieves the top 10 vectors, performs LLM semantic reranking, and returns the top 3. This pipeline filters out semantically contradictory results and returns only 3 results. |
Request examples
Video summarization
curl --request POST \
--url https://dashscope.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation \
--header "Authorization: Bearer $DASHSCOPE_API_KEY" \
--header "Content-Type: application/json" \
--header "X-DashScope-SSE: enable" \
--data '{
"model": "multimodal-dialog",
"input": {
"directive": "Request",
"app_id": "mm_xxxxxxxx",
"text": ""
},
"parameters": {
"client_info": {
"user_id": "user_001",
"device": {
"uuid": "device_001"
}
},
"biz_params": {
"commands": [
{
"name": "agent_command",
"exec_params": {
"app_id": "physical_sense",
"intent": "open_physical_sense",
"slots": [
{
"name": "scene",
"norm_value": "ipc"
}
]
}
}
],
"user_defined_params": {
"physical_sense": {
"user_prompt_params": {
"param": {
"sub_scene": "vector",
"format": "video",
"prompt": "",
"fps": 0.2,
"max_pixels": 230400,
"images": [
{
"type": "url",
"value": "https://example-bucket.oss-cn-hangzhou.aliyuncs.com/ipc/camera-01.dts"
}
],
"oss": {
"internal": false,
"region": "hangzhou"
},
"vector_mode": "store",
"vector_store_config": {
"tags": {
"tags_object": ["car", "pet", "flame", "smoke", "food delivery", "package", "person"],
"tags_action": ["fall", "sitting", "lying"]
},
"extra_data": {
"oss_path": "/video/00"
},
"pre_process": {
"type": "dts_to_mp4"
}
}
}
}
}
}
}
}
}'
Video retrieval (top_size)
curl --request POST \
--url https://dashscope.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation \
--header "Authorization: Bearer $DASHSCOPE_API_KEY" \
--header "Content-Type: application/json" \
--header "X-DashScope-SSE: enable" \
--data '{
"model": "multimodal-dialog",
"input": {
"directive": "Request",
"app_id": "mm_xxxxxxxx",
"text": ""
},
"parameters": {
"client_info": {
"user_id": "user_001",
"device": {
"uuid": "device_001"
}
},
"biz_params": {
"commands": [
{
"name": "agent_command",
"exec_params": {
"app_id": "physical_sense",
"intent": "open_physical_sense",
"slots": [
{
"name": "scene",
"norm_value": "ipc"
}
]
}
}
],
"user_defined_params": {
"physical_sense": {
"user_prompt_params": {
"param": {
"sub_scene": "vector",
"format": "image",
"prompt": "What time was someone playing ball?",
"vector_mode": "retrieve",
"vector_retrieve_config": {
"pipeline": "top_size",
"similarity_top_size": "5",
"vector_retrieval_time": {
"start_time": "2026-09-21 00:00:00",
"end_time": "2026-09-21 23:00:00"
}
}
}
}
}
}
}
}
}'
Video retrieval (llm_rerank)
curl --request POST \
--url https://dashscope.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation \
--header "Authorization: Bearer $DASHSCOPE_API_KEY" \
--header "Content-Type: application/json" \
--header "X-DashScope-SSE: enable" \
--data '{
"model": "multimodal-dialog",
"input": {
"directive": "Request",
"app_id": "mm_xxxxxxxx",
"text": ""
},
"parameters": {
"client_info": {
"user_id": "user_001",
"device": {
"uuid": "device_001"
}
},
"biz_params": {
"commands": [
{
"name": "agent_command",
"exec_params": {
"app_id": "physical_sense",
"intent": "open_physical_sense",
"slots": [
{
"name": "scene",
"norm_value": "ipc"
}
]
}
}
],
"user_defined_params": {
"physical_sense": {
"user_prompt_params": {
"param": {
"sub_scene": "vector",
"format": "image",
"prompt": "What time was someone playing ball?",
"vector_mode": "retrieve",
"vector_retrieve_config": {
"pipeline": "llm_rerank",
"vector_retrieval_time": {
"start_time": "2026-09-21 00:00:00",
"end_time": "2026-09-21 23:00:00"
}
}
}
}
}
}
}
}
}'
Response
The API returns an SSE event stream. During waiting, only the last data packet with finished=true matters, which contains the final result. Events contain the event: result and data fields. The following examples show the JSON structure of the data field.
Summary response example
{
"output": {
"round_id": "385edc5b707b413faf6b399f0894a73c",
"llm_request_id": "7ea9521fbb1a48d796588213ceb84900",
"extra_info": {
"tool_infos": [
{
"output": "",
"tool_name": "physical_sense",
"arguments": {},
"type": "agent",
"success": true
}
],
"agent_info": {
"ext_params": {
"vector_store_result": {
"summary": "No one is active in the indoor office area; the environment is quiet.",
"tags_action": [],
"tags_object": [],
"is_success": true,
"info": "stored 1 embedding successfully"
},
"usage": [
{
"inputTokens": 1340,
"modelType": "type_ipc_flash",
"outputTokens": 91,
"totalTokens": 1431
},
{
"inputTokens": 69,
"modelType": "event_retrieval",
"outputTokens": 0,
"totalTokens": 69
}
]
},
"round": 1,
"device": {
"device_id": "device_001"
},
"intent_infos": [
{
"intent": "open_physical_sense",
"domain": "physical_sense"
}
]
},
"query": ""
},
"finish_reason": "stop",
"dialog_id": "2dd901ad-d061-449b-a6eb-8652fccf4f53",
"spoken": "",
"finished": true,
"text": "summary and vector store",
"event": "RespondingContent"
},
"request_id": "66207dc3-2b37-9ace-a5b8-b8df609ef4eb"
}
Retrieval response example
{
"output": {
"round_id": "e582d5e2a59d4f889468027d18eca40f",
"llm_request_id": "4d9ecf97e8b845e695ea66fb1aa30f73",
"extra_info": {
"tool_infos": [
{
"output": "",
"tool_name": "physical_sense",
"arguments": {},
"type": "agent",
"success": true
}
],
"agent_info": {
"ext_params": {
"vector_retrieve_result": {
"pipeline": "top_size",
"total_candidates": 1,
"is_success": true,
"matches": [
{
"capturedAt": "2026-09-21 19:45:14",
"description": "The surveillance footage shows an indoor office area during the day with ample lighting. A white reception desk is on the left, and a rectangular table with several chairs is on the right. Potted plants and an ashtray are on the table. A tall green plant stands in the background. No people, vehicles, or pets appear in the scene, and the environment remains still.",
"extraData": "{\"oss_path\":\"/video/00\"}",
"mediaUrls": [
"https://example-bucket.oss-cn-hangzhou.aliyuncs.com/ipc/camera-01.dts"
],
"similarity": 0.4510287801329569
}
],
"info": "retrieved 1 matches"
},
"usage": [
{
"inputTokens": 5,
"modelType": "event_retrieval",
"outputTokens": 0,
"totalTokens": 5
}
]
},
"round": 1,
"device": {
"device_id": "device_001"
},
"intent_infos": [
{
"intent": "open_physical_sense",
"domain": "physical_sense"
}
]
},
"query": ""
},
"finish_reason": "stop",
"dialog_id": "03874c54-e385-49b3-a534-7856cc4d3533",
"spoken": "",
"finished": true,
"text": "vector retrieve",
"event": "RespondingContent"
},
"request_id": "5c0858af-a642-9303-89a1-407338afe19e"
}
Response parameters
| Parameter | Type | Description |
|---|---|---|
request_id | string | Request ID of the Model Studio gateway. |
output.event | string | Event name. Fixed to RespondingContent. |
output.dialog_id | string | Dialog ID. |
output.round_id | string | ID of the current interaction round. |
output.llm_request_id | string | Request ID of the large model call. |
output.text | string | Response content. In this topic, the business result is returned in output.extra_info.agent_info.ext_params. |
output.spoken | string | Content used for TTS. You can ignore this field when speech synthesis is not enabled. |
output.finished | boolean | Whether the output is complete. |
output.finish_reason | string | Reason for completion. Only stop is supported, which indicates a normal completion. |
output.extra_info | object | Extended information. |
output.extra_info.tool_infos | array | Tool call information, including fields such as tool_name, type, success, arguments, and output. |
output.extra_info.agent_info | object | Agent information. |
output.extra_info.query | string | Query content. |
output.extra_info.agent_info.ext_params
| Parameter | Type | Description |
|---|---|---|
vector_store_result | object | Returned when vector_mode=store. Indicates the summary storage result. |
vector_store_result.summary | string | Video summary. |
vector_store_result.tags_object | list<string> | Matched object tags. |
vector_store_result.tags_action | list<string> | Matched action tags. |
vector_store_result.is_success | boolean | Whether the operation succeeded. |
vector_store_result.info | string | Prompt information. |
vector_retrieve_result | object | Returned when vector_mode=retrieve. Indicates the retrieval result. |
vector_retrieve_result.pipeline | string | Same as the pipeline specified in the request. |
vector_retrieve_result.total_candidates | integer | Number of vectors involved in similarity comparison. |
vector_retrieve_result.is_success | boolean | Whether the operation succeeded. |
vector_retrieve_result.matches | array | Details of the returned vectors, where size is the actual number of returned vectors. |
vector_retrieve_result.matches[].mediaUrls | array | Array of resource URLs used during storage. |
vector_retrieve_result.matches[].description | string | Text summary of the resource. |
vector_retrieve_result.matches[].similarity | double | Similarity. |
vector_retrieve_result.matches[].capturedAt | string | Time when the vector was stored. |
vector_retrieve_result.matches[].extraData | string | Custom information bound to the vector, as a JSON string. |
vector_retrieve_result.info | string | Prompt information. |
usage | array | Token consumption information, including all models that were called. |
usage[].modelType | string | Model type. |
usage[].inputTokens | integer | Number of input tokens. |
usage[].outputTokens | integer | Number of output tokens. |
usage[].totalTokens | integer | Total number of tokens. |