Film and television media video understanding

Application details

On the Application Details tab of the Application Details application, you can view the feature description, target audience, best practices, usage instructions, billing rules, related Quanmiao application recommendations, and related video understanding solutions.

1. Configure the Video Processing options.

   1. Upload a video file: Click or drag a local video file into the designated area.

      Supports Chinese and English videos in .flv, .ts, .avi, .wmv, .mpg, .mkv, .mov, and .mp4 formats.

      For online tasks, each video must be smaller than 512 MB, have a resolution below 1080p, and be no longer than 60 minutes.

      For offline tasks, each video must be smaller than 1 GB and have a resolution below 1080p. You can adjust the duration limit by using the UpdateVideoAnalysisConfig API.

      If your video exceeds 60 minutes, you can use the asynchronous API. The functionality is the same, but result previews are not currently supported on the page.

      For details on how to call the API, see the API page:

      package com.aliyun.bailian.bailian_sdk_demo;
      
      import com.aliyun.bailian.bailian_sdk_demo.client.xxx;
      import com.aliyun.bailian.bailian_sdk_demo.entity.xxx;
      import com.aliyun.bailian.bailian_sdk_demo.utils.xxx;
      import com.aliyun.bailian.sdk.xxx;
      
      import java.util.xxx;
      import java.net.xxx;
      import java.nio.xxx;
      
      public class VideoUnderstandingDemo {
          public static void main(String[] args) {
              // xxx Asynchronous API call example. For detailed parameters, see the API page in the console.
          }
      }

   2. Optional: Enter supplementary text related to the video to help the large model understand its content.

2. Configure the Visual Language Analysis (VL Analysis) options. This step uses a multimodal large model to analyze detailed information in each video segment, such as characters, actions, and on-screen content. The generated content is an intermediate result for Step 3 and is not the primary output.

   1. Select a VLM: Currently, Qwen-VL-Max-Latest, Qwen-VL-Max, Qwen-VL-Plus-Latest, and Qwen-VL-Plus are supported.

   2. Enter a VLM prompt: Fill in the prompt for the VLM according to the template.

3. Optional: Configure the Text Processing options. Based on your specific goals, enter an appropriate prompt below for tasks such as video content summarization, tag extraction, and video classification.

   1. Select a large language model. Currently, Qwen-Max-Latest, Qwen-Max, Qwen-Plus-Latest, Qwen-Plus, DeepSeek-R1, Tongyi Qianwen 2.5-7B-1M, and Qwen3-235B-A22B (thinking and non-thinking modes) are supported.

   2. Enter a large language model prompt: You can use the application scenarios in the Reference Examples section. Ten common applications are supported, including content description, structure analysis, tag classification, Q&A scenarios, content mining, video retrieval, analysis scenarios, and marketing scenarios. Multiple subtask templates are built-in for these applications.

      You can click Add Text Model Task to add up to three tasks.

      The video understanding feature supports the following capabilities:

      The main feature categories in Reference Examples include Content Description, Structure Analysis, Tag Classification, Q&A Scenarios, Content Mining, Video Retrieval, Analysis Scenarios, Marketing Scenarios, Learning Samples, and Game & E-sports Scenarios. Each category provides multiple prompt templates (such as for video summaries, key characters, key events, key moments, key phrases, and key frames) that you can use directly or fine-tune to fit your specific business needs.

4. Configure Other optional outputs (built-in capabilities), including Video Timestamps + Subtitles, Summarize Video Content into a Mind Map, and Generate Video Titles from Summary.

5. Configure Advanced features, which include frame extraction strategy configuration, multi-language recognition, and character identity recognition.

   • Frame extraction strategy configuration: Supports fast, standard, and custom frame extraction methods. A longer interval between extracted frames reduces the granularity of video information captured but also consumes fewer tokens per video.

   • Automatic speech recognition (ASR) settings.

   • Multi-language recognition: Supports video analysis for five languages, including Chinese, English, French, Japanese, and mixed Chinese-English speech.

   • Character identity recognition: Supports the identification of up to three characters in a video. You can upload one image for each character.

   • Face similarity threshold: The value ranges from 0.1 to 1. The recommended range is 0.45 to 0.65. The default value is 0.5.

   • Number of frames to extract per video segment for character matching: The value ranges from 1 to 5. The recommended value is 3, which is also the default.

6. Click Start Analysis. The large model then generates content based on your input. You can view the results from Step 4 (Video Timestamps + Subtitles, Summarize Video Content into a Mind Map, Generate Video Titles from Summary), as well as the Visual language analysis results and Text processing results on the right.

   After the video understanding task runs, the results pane on the right displays three sections from top to bottom: Video Subtitles shows the subtitle text with timestamps; Visual language analysis results presents descriptions of scenes and plots for each video segment; and Text processing results outputs the video summary, tags, and classifications generated according to the prompts.

The service supports Chinese and English videos in .flv, .ts, .avi, .wmv, .mpg, .mkv, .mov, and .mp4 formats. Video requirements vary by processing mode. For real-time tasks, videos must be smaller than 200 MB, shorter than 30 minutes, and have a resolution below 1080p. For asynchronous tasks, they must be smaller than 450 MB and shorter than 60 minutes.

The video processing API uses the SSE protocol for real-time, streaming processing. If you need to process a large number of videos, you can improve efficiency by making parallel API calls. We recommend keeping the concurrency within 20 QPM. If you require higher concurrency, contact us for assistance.

• The video processing workflow generally includes: ASR, segmentation, frame extraction, and visual language analysis (which consumes VLM tokens), followed by text processing (which consumes LLM tokens).

• The consumption of text processing tokens is not significantly affected by video duration. However, for visual language analysis, token consumption increases almost linearly with longer video durations and shorter frame extraction intervals.

• If you need to control token consumption, consider increasing the frame extraction interval (snapshotInterval[1–10]). However, a longer interval may reduce the detail and accuracy of the video understanding.

Use the Video Understanding - Get Configuration and Video Understanding - Update Configuration API operations to view and increase your concurrency quota.

To reduce costs while maintaining acceptable performance, you can try the following methods:

1. Use smaller VLM and LLM models.

2. Customize the frame extraction configuration by adjusting the extraction interval and reducing the image resolution.

3. Try disabling ASR if the video already has subtitles.

4. Keep concurrency at 2 or below, as there is no charge for up to 2 concurrent requests.

During the video understanding process, the system synchronizes audio and video to the same timestamp as follows:

• Automatic speech recognition (ASR) is performed in conjunction with the video frames. The visual and audio information is already segmented during the initial parsing.

• When the visual language model (VLM) analyzes the video, it records the corresponding timestamps for the frames in different segments.

• The system then aligns the VLM frame timestamps with the ASR speech timestamps to ensure precise audio-video synchronization.