IPC video summarization and retrieval

Updated at:

For IPC camera scenarios, connect to the physical world perception agent over HTTP to generate video summaries, structured tags, and natural language video retrieval.

Overview

For consumer IP cameras (IPC), this feature provides intelligent understanding and semantic retrieval of surveillance data. The system uses a multimodal large model to automatically summarize and structurally tag video footage, and uses vector storage and retrieval to enable natural language queries over historical surveillance, helping users quickly locate key moments.

  • Video summarization: Automatically generates scene descriptions and structured tags for surveillance footage, supports custom tags, and persists them as semantic vectors.
  • Video retrieval: Supports natural language queries over historical events, combines vector similarity matching with LLM semantic reranking, and returns results by similarity threshold or Top-K.

ImportantThis topic covers two invocation modes: vector_mode=store generates and stores a summary, and vector_mode=retrieve retrieves historical video. Both use the same HTTP endpoint and are distinguished by vector_mode.

Prerequisites

Console configuration

Step 1: Create an application

  1. Open the Multimodal Interaction Development Suite and log on. Click Activate for free in the upper-right corner and complete service activation.
Activate the Multimodal Interaction service
  1. Click Custom create to open the application configuration page.
Create a Multimodal Interaction application Custom create an application

Step 2: Configure the application

  1. Turn off Voice interaction and keep Intent recognition and Text model turned on. Turn off Conversation carryover, Knowledge base, Web search, and Long-term memory. Delete all instructions, plugins, and MCP services.
Turn off voice interaction and keep intent recognition and text model Turn off conversation carryover, knowledge base, web search, and long-term memory Delete all instructions, plugins, and MCP services
  1. Click the add button for Model Studio applications and add the Physical World Perception application. If you use it for the first time, activate the service.
Add the Physical World Perception application Activate the Physical World Perception service
  1. After activation, refresh the application configuration page, select Physical World Perception, and click OK. Do not select other applications.
Select the Physical World Perception application
  1. The application provides IPC scene visual analysis by default. You can switch it with the configuration button.
Switch the IPC scene visual analysis capability

NoteKeep only Physical World Perception in the application configuration. Do not select other applications, otherwise the invocation path in this topic may be affected.

Step 3: Publish the application

After configuration, click Publish in the upper-right corner. The application must be published before testing.

Publish the application

Return to the previous page, click Copy, and save the application ID for later invocation.

Copy the application ID

HTTP protocol integration

Endpoint: https://dashscope.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation. Set the X-DashScope-SSE: enable header. Results are returned as an SSE event stream.

Request parameters

ParameterTypeRequiredDescription
modelstringYesModel name. Fixed to multimodal-dialog.
input.directivestringYesInstruction name. Fixed to Request.
input.app_idstringYesApplication ID created in the console. You can view it on the My Apps page of the Multimodal Interaction Development Suite console. See Obtain an App ID and workspace ID.
input.textstringYesText to process. In this topic, set it to an empty string "".
parameters.client_infoobjectYesClient information. See below.
parameters.biz_paramsobjectYesBusiness parameters. See below.

parameters.client_info

ParameterTypeRequiredDescription
user_idstringYesEnd-user ID generated by your business logic to provide customized features for different end users. Maximum length: 36 characters.
Note: This field is required for summarization and retrieval. Within the same app_id, each user's user_id must be unique. Videos with the same user_id are placed in the same retrieval space. If two end users use the same user_id, they share retrieval resources.
device.uuidstringYesDevice ID generated by your business logic and passed to the SDK. Maximum length: 40 characters. One end user (user_id) can have multiple devices. Each device has a different uuid, but the user_id is the same.
Note: This field is required for summarization and retrieval. Within the same app_id, different devices of the same user_id must have different uuid values; otherwise, the devices cannot be distinguished.

parameters.biz_params

ParameterTypeRequiredDescription
commands[].namestringYesFixed to agent_command.
commands[].exec_paramsobjectYesSpecifies the agent. See below.
user_defined_params.physical_senseobjectYesParameters passed to the physical world perception model.

commands[].exec_params

ParameterTypeRequiredDescription
app_idstringYesFixed to physical_sense.
intentstringYesFixed to open_physical_sense.
slotslist[]YesAgent slot information.
slots
ParameterTypeRequiredDescription
namestringYesSlot name.
norm_valuestringYesSlot value.

To connect to the physical world perception agent, fill in the scene slot. scene is required and supports the following values:

  • embodied: Embodied intelligence
  • ipc: IPC scenario (used in this topic)
  • auto_driving: Autonomous driving
{
  "slots": [
    {
      "name": "scene",
      "norm_value": "ipc"
    }
  ]
}

parameters.biz_params.user_defined_params.physical_sense.user_prompt_params

ParameterTypeRequiredDescription
param.sub_scenestringYesFixed to vector in this scenario.
param.vector_modestringYesSummary or retrieval. store: summary; retrieve: retrieval. Other values return an error.
param.vector_store_configobjectNoRequired when vector_mode=store. Summary configuration. See below.
param.vector_retrieve_configobjectNoRequired when vector_mode=retrieve. Retrieval configuration. See below.
param.formatstringYesResource mode. Valid values: image and video.
param.promptstringYesSet to an empty string when vector_mode=store. When vector_mode=retrieve, specify the natural language query.
param.imagesarrayConditionalResource array. Required for summarization. See below.
param.ossobjectNoOSS configuration. See below.
param.frame_enabledbooleanNoWhether to use an image sequence. Takes effect only when format=video.
param.high_resolutionbooleanNoWhether to enable high resolution. Default: off.
param.max_pixelsintegerNoMaximum number of pixels. Valid when vector_mode=store. Use the example value 230400.
param.fpsdoubleConditionalFrame extraction rate. Value range: [0.1, 10.0]. Required when vector_mode=store and format=video. Use the example value 0.2. We recommend that you do not change it. A higher value increases costs.
param.max_framesintegerNoMaximum number of extracted frames.
param.max_tokensintegerNoMaximum number of output tokens.
param.temperaturefloatNoTemperature. Default: 0.0.
param.versionstringNoVersion number. Leave empty if no special convention applies.
param.images
ParameterTypeRequiredDescription
typestringYesResource type. Valid values: base64 and url.
valuestringYesWhen type=base64, the base64 string of the resource. When type=url, the resource URL.

Note

  • The recommended resolution range is 640×480 to 1920×1080.
  • When type=url, the URL must start with https. http is not supported.
param.oss
ParameterTypeRequiredDescription
internalbooleanNoWhether to use an internal network URL. Internal URLs can be downloaded only from the same region.
regionstringNoValid when internal=true. The OSS region.

vector_store_config

ParameterTypeRequiredDescription
tagsobjectNoCustom tags. They are injected into the prompt.
tags.tags_objectlist<string>NoObject tags.
tags.tags_actionlist<string>NoAction tags.
extra_dataobjectNoCustom data returned with retrieval results.
pre_processobjectNoPreprocessing configuration.
pre_process.typestringNoOnly dts_to_mp4 is supported. Other values return an error.

vector_retrieve_config

ParameterTypeRequiredDescription
pipelinestringYesPipeline used for vector retrieval. See below.
similarity_thresholdstringConditionalRequired when pipeline=threshold. Must be parseable as a double in (0,1]; otherwise, an error is returned.
similarity_top_sizestringConditionalRequired when pipeline=top_size. Must be parseable as a positive integer ≤ 1000; otherwise, an error is returned.
vector_retrieval_timeobjectNoRetrieval time range. If empty, the current day is used.
vector_retrieval_time.start_timestringNoStart time in yyyy-MM-dd HH:mm:ss format.
vector_retrieval_time.end_timestringNoEnd time in yyyy-MM-dd HH:mm:ss format.

Time range rules:

  • start_time and end_time must be specified together. Specifying only one returns an error.
  • end_time must be later than start_time.
  • The default time zone is Asia/Shanghai.
  • Data is retained for 7 days by default.

pipeline

ValueDescription
thresholdPerforms retrieval by similarity threshold and returns vectors whose similarity meets the threshold.
top_sizeReturns the top size vectors by similarity.
default_topkDefault pipeline, equivalent to top_size=10.
llm_rerankRetrieves the top 10 vectors, performs LLM semantic reranking, and returns the top 3. This pipeline filters out semantically contradictory results and returns only 3 results.

Request examples

Video summarization

curl --request POST \
  --url https://dashscope.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation \
  --header "Authorization: Bearer $DASHSCOPE_API_KEY" \
  --header "Content-Type: application/json" \
  --header "X-DashScope-SSE: enable" \
  --data '{
  "model": "multimodal-dialog",
  "input": {
    "directive": "Request",
    "app_id": "mm_xxxxxxxx",
    "text": ""
  },
  "parameters": {
    "client_info": {
      "user_id": "user_001",
      "device": {
        "uuid": "device_001"
      }
    },
    "biz_params": {
      "commands": [
        {
          "name": "agent_command",
          "exec_params": {
            "app_id": "physical_sense",
            "intent": "open_physical_sense",
            "slots": [
              {
                "name": "scene",
                "norm_value": "ipc"
              }
            ]
          }
        }
      ],
      "user_defined_params": {
        "physical_sense": {
          "user_prompt_params": {
            "param": {
              "sub_scene": "vector",
              "format": "video",
              "prompt": "",
              "fps": 0.2,
              "max_pixels": 230400,
              "images": [
                {
                  "type": "url",
                  "value": "https://example-bucket.oss-cn-hangzhou.aliyuncs.com/ipc/camera-01.dts"
                }
              ],
              "oss": {
                "internal": false,
                "region": "hangzhou"
              },
              "vector_mode": "store",
              "vector_store_config": {
                "tags": {
                  "tags_object": ["car", "pet", "flame", "smoke", "food delivery", "package", "person"],
                  "tags_action": ["fall", "sitting", "lying"]
                },
                "extra_data": {
                  "oss_path": "/video/00"
                },
                "pre_process": {
                  "type": "dts_to_mp4"
                }
              }
            }
          }
        }
      }
    }
  }
}'

Video retrieval (top_size)

curl --request POST \
  --url https://dashscope.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation \
  --header "Authorization: Bearer $DASHSCOPE_API_KEY" \
  --header "Content-Type: application/json" \
  --header "X-DashScope-SSE: enable" \
  --data '{
  "model": "multimodal-dialog",
  "input": {
    "directive": "Request",
    "app_id": "mm_xxxxxxxx",
    "text": ""
  },
  "parameters": {
    "client_info": {
      "user_id": "user_001",
      "device": {
        "uuid": "device_001"
      }
    },
    "biz_params": {
      "commands": [
        {
          "name": "agent_command",
          "exec_params": {
            "app_id": "physical_sense",
            "intent": "open_physical_sense",
            "slots": [
              {
                "name": "scene",
                "norm_value": "ipc"
              }
            ]
          }
        }
      ],
      "user_defined_params": {
        "physical_sense": {
          "user_prompt_params": {
            "param": {
              "sub_scene": "vector",
              "format": "image",
              "prompt": "What time was someone playing ball?",
              "vector_mode": "retrieve",
              "vector_retrieve_config": {
                "pipeline": "top_size",
                "similarity_top_size": "5",
                "vector_retrieval_time": {
                  "start_time": "2026-09-21 00:00:00",
                  "end_time": "2026-09-21 23:00:00"
                }
              }
            }
          }
        }
      }
    }
  }
}'

Video retrieval (llm_rerank)

curl --request POST \
  --url https://dashscope.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation \
  --header "Authorization: Bearer $DASHSCOPE_API_KEY" \
  --header "Content-Type: application/json" \
  --header "X-DashScope-SSE: enable" \
  --data '{
  "model": "multimodal-dialog",
  "input": {
    "directive": "Request",
    "app_id": "mm_xxxxxxxx",
    "text": ""
  },
  "parameters": {
    "client_info": {
      "user_id": "user_001",
      "device": {
        "uuid": "device_001"
      }
    },
    "biz_params": {
      "commands": [
        {
          "name": "agent_command",
          "exec_params": {
            "app_id": "physical_sense",
            "intent": "open_physical_sense",
            "slots": [
              {
                "name": "scene",
                "norm_value": "ipc"
              }
            ]
          }
        }
      ],
      "user_defined_params": {
        "physical_sense": {
          "user_prompt_params": {
            "param": {
              "sub_scene": "vector",
              "format": "image",
              "prompt": "What time was someone playing ball?",
              "vector_mode": "retrieve",
              "vector_retrieve_config": {
                "pipeline": "llm_rerank",
                "vector_retrieval_time": {
                  "start_time": "2026-09-21 00:00:00",
                  "end_time": "2026-09-21 23:00:00"
                }
              }
            }
          }
        }
      }
    }
  }
}'

Response

The API returns an SSE event stream. During waiting, only the last data packet with finished=true matters, which contains the final result. Events contain the event: result and data fields. The following examples show the JSON structure of the data field.

Summary response example

{
  "output": {
    "round_id": "385edc5b707b413faf6b399f0894a73c",
    "llm_request_id": "7ea9521fbb1a48d796588213ceb84900",
    "extra_info": {
      "tool_infos": [
        {
          "output": "",
          "tool_name": "physical_sense",
          "arguments": {},
          "type": "agent",
          "success": true
        }
      ],
      "agent_info": {
        "ext_params": {
          "vector_store_result": {
            "summary": "No one is active in the indoor office area; the environment is quiet.",
            "tags_action": [],
            "tags_object": [],
            "is_success": true,
            "info": "stored 1 embedding successfully"
          },
          "usage": [
            {
              "inputTokens": 1340,
              "modelType": "type_ipc_flash",
              "outputTokens": 91,
              "totalTokens": 1431
            },
            {
              "inputTokens": 69,
              "modelType": "event_retrieval",
              "outputTokens": 0,
              "totalTokens": 69
            }
          ]
        },
        "round": 1,
        "device": {
          "device_id": "device_001"
        },
        "intent_infos": [
          {
            "intent": "open_physical_sense",
            "domain": "physical_sense"
          }
        ]
      },
      "query": ""
    },
    "finish_reason": "stop",
    "dialog_id": "2dd901ad-d061-449b-a6eb-8652fccf4f53",
    "spoken": "",
    "finished": true,
    "text": "summary and vector store",
    "event": "RespondingContent"
  },
  "request_id": "66207dc3-2b37-9ace-a5b8-b8df609ef4eb"
}

Retrieval response example

{
  "output": {
    "round_id": "e582d5e2a59d4f889468027d18eca40f",
    "llm_request_id": "4d9ecf97e8b845e695ea66fb1aa30f73",
    "extra_info": {
      "tool_infos": [
        {
          "output": "",
          "tool_name": "physical_sense",
          "arguments": {},
          "type": "agent",
          "success": true
        }
      ],
      "agent_info": {
        "ext_params": {
          "vector_retrieve_result": {
            "pipeline": "top_size",
            "total_candidates": 1,
            "is_success": true,
            "matches": [
              {
                "capturedAt": "2026-09-21 19:45:14",
                "description": "The surveillance footage shows an indoor office area during the day with ample lighting. A white reception desk is on the left, and a rectangular table with several chairs is on the right. Potted plants and an ashtray are on the table. A tall green plant stands in the background. No people, vehicles, or pets appear in the scene, and the environment remains still.",
                "extraData": "{\"oss_path\":\"/video/00\"}",
                "mediaUrls": [
                  "https://example-bucket.oss-cn-hangzhou.aliyuncs.com/ipc/camera-01.dts"
                ],
                "similarity": 0.4510287801329569
              }
            ],
            "info": "retrieved 1 matches"
          },
          "usage": [
            {
              "inputTokens": 5,
              "modelType": "event_retrieval",
              "outputTokens": 0,
              "totalTokens": 5
            }
          ]
        },
        "round": 1,
        "device": {
          "device_id": "device_001"
        },
        "intent_infos": [
          {
            "intent": "open_physical_sense",
            "domain": "physical_sense"
          }
        ]
      },
      "query": ""
    },
    "finish_reason": "stop",
    "dialog_id": "03874c54-e385-49b3-a534-7856cc4d3533",
    "spoken": "",
    "finished": true,
    "text": "vector retrieve",
    "event": "RespondingContent"
  },
  "request_id": "5c0858af-a642-9303-89a1-407338afe19e"
}

Response parameters

ParameterTypeDescription
request_idstringRequest ID of the Model Studio gateway.
output.eventstringEvent name. Fixed to RespondingContent.
output.dialog_idstringDialog ID.
output.round_idstringID of the current interaction round.
output.llm_request_idstringRequest ID of the large model call.
output.textstringResponse content. In this topic, the business result is returned in output.extra_info.agent_info.ext_params.
output.spokenstringContent used for TTS. You can ignore this field when speech synthesis is not enabled.
output.finishedbooleanWhether the output is complete.
output.finish_reasonstringReason for completion. Only stop is supported, which indicates a normal completion.
output.extra_infoobjectExtended information.
output.extra_info.tool_infosarrayTool call information, including fields such as tool_name, type, success, arguments, and output.
output.extra_info.agent_infoobjectAgent information.
output.extra_info.querystringQuery content.

output.extra_info.agent_info.ext_params

ParameterTypeDescription
vector_store_resultobjectReturned when vector_mode=store. Indicates the summary storage result.
vector_store_result.summarystringVideo summary.
vector_store_result.tags_objectlist<string>Matched object tags.
vector_store_result.tags_actionlist<string>Matched action tags.
vector_store_result.is_successbooleanWhether the operation succeeded.
vector_store_result.infostringPrompt information.
vector_retrieve_resultobjectReturned when vector_mode=retrieve. Indicates the retrieval result.
vector_retrieve_result.pipelinestringSame as the pipeline specified in the request.
vector_retrieve_result.total_candidatesintegerNumber of vectors involved in similarity comparison.
vector_retrieve_result.is_successbooleanWhether the operation succeeded.
vector_retrieve_result.matchesarrayDetails of the returned vectors, where size is the actual number of returned vectors.
vector_retrieve_result.matches[].mediaUrlsarrayArray of resource URLs used during storage.
vector_retrieve_result.matches[].descriptionstringText summary of the resource.
vector_retrieve_result.matches[].similaritydoubleSimilarity.
vector_retrieve_result.matches[].capturedAtstringTime when the vector was stored.
vector_retrieve_result.matches[].extraDatastringCustom information bound to the vector, as a JSON string.
vector_retrieve_result.infostringPrompt information.
usagearrayToken consumption information, including all models that were called.
usage[].modelTypestringModel type.
usage[].inputTokensintegerNumber of input tokens.
usage[].outputTokensintegerNumber of output tokens.
usage[].totalTokensintegerTotal number of tokens.