Caption extraction-ApsaraVideo VOD(VOD)-阿里云帮助中心

ApsaraVideo for VOD workflow 2.0 integrates with Intelligent Media Service (IMS) to automatically recognize and extract hard subtitles from videos and output standard SRT files.

This feature (CaptionExtraction) uses OCR technology to identify and extract visible subtitle text directly from video frames. It is applicable to videos that already contain embedded (burned-in) hard subtitles. This feature does not support generating subtitles from video audio or speech. If you need to generate subtitles from audio or speech using ASR recognition, use the AI Video Translation feature instead.

Note

Subtitle extraction is available in the following regions:

China (Shanghai), Asia Pacific SE 1 (Singapore), and US West 1 (Silicon Valley).

Prerequisites

To use the subtitle extraction node in a workflow, you must activate ApsaraVideo for VOD and Intelligent Media Service (IMS) and subscribe to the Standard Edition of IMS. For more information, see Subscription-based billing.

Billing

Subtitle extraction is powered by Intelligent Media Service (IMS). You must purchase the IMS subscription service and pay usage fees. For more information, see Intelligent Production.

Usage

Step 1: Configure a subtitle extraction workflow

Workflows can be created only in the ApsaraVideo for VOD console.

Log on to the ApsaraVideo for VOD console.
In the left-side navigation pane, choose Configuration Management > Media Processing > Workflows to open the Workflows page.
Click Add Workflow and enter a name for the workflow.
On the workflow editing page, click the + icon to the right of the Start node and add a Subtitle Extraction node.
In the right-side panel, configure the following parameters:
- Node Name: A custom name for the node.
- Subtitle Language: Supports Chinese, English, and mixed Chinese-English.
- Frame Rate (fps): Higher frame rates improve recognition accuracy but increase processing time. Valid values: 2 to 10. Recommended: 5 for standard videos and 10 for fast-paced videos.
- Sample Material: A sample video for configuring subtitle settings. The workflow does not process this video. Supported formats: MP4, WebM, MOV, and M3U8.
- Recognition Area: Defaults to the bottom quarter of the video. You can manually select a custom area.
- Main Subtitle Track Extraction: When enabled, only the main subtitle track is extracted, which is suitable for TV series and movies. When disabled, all subtitles in the specified area are extracted, which is suitable for live streams and user-generated content (UGC).
After you complete the configuration, click OK to submit the workflow template. The system generates a workflow ID. Record this ID to use for specifying the workflow during uploads. The created Subtitle Extraction template appears in the workflow management list with a status of Activating. In the Actions column, you can click Details or Deactivate.

Step 2: Trigger the workflow

You can trigger a workflow job either during or after the video upload.

Console

During upload

Log on to the ApsaraVideo for VOD console.
In the left-side navigation pane, choose Media Files > Audio/Video, and then click Upload Media.
On the Upload Audio/Video page, click to add an audio or video file. In the Add Audio/Video dialog box that appears, set the upload method to Local Upload and click Add Audio/Video to add the files that you want to upload. Select Use Workflow and specify the workflow that you created in the previous step. In the Transcoding Template Group/Workflow Template column, select the target workflow, such as Subtitle Extraction test, and then click Start Upload.

After upload

Log on to the ApsaraVideo for VOD console.
In the left-side navigation pane, choose Media Files > Audio/Video to open the audio and video list.
In the Actions column for the target media file, click Processing. Then, select Process with Workflow and choose the workflow that you created.

API

During upload

CreateUploadVideo only obtains the upload URL and credentials and creates basic media asset information. You must implement the file upload logic separately. For a complete example, see Upload media files by using ApsaraVideo for VOD APIs.
When you call the CreateUploadVideo or UploadMediaByURL operation to upload a media file, set the WorkflowId parameter to the ID of the workflow you created. After the upload is complete, the system automatically processes the media file using the specified workflow.

After upload

Call the SubmitWorkflowJob operation and set WorkflowId to the ID of the workflow you created. The system immediately processes the media file with the specified workflow.

Step 3: Query the results

Console

Log on to the ApsaraVideo for VOD console.
In the left-side navigation pane, choose Media Files > Short Video Material to open the Short-video materials page.
Find the subtitle file generated by the workflow. You can filter by source video name, creation time, or other criteria.
After processing is complete, you can view the following information:
- The media ID of the subtitle extraction output.
- The storage address of the output file.
- Details such as file size and file URL. You can copy the URL to download the file.

Workflow callbacks

Configure HTTP or Lightweight Message Queue (formerly MNS) callbacks.
When a workflow job completes, the system triggers a workflow task completion event and sends the results through the configured callback. The key fields in the callback message are:
1. Status: The job status. Valid values are Succeed and Failed.
2. ActivityResults: A JSON string that contains execution details for each node, such as the SubtitleExtraction node. The Result field contains key information about the output, such as MediaId and JobId.
3. TaskInput: The original input media information, such as the Media ID and filename.
Parse the ActivityResults field in the callback message to extract the MediaId of the generated subtitle file for subsequent playback or distribution.

API

The workflow submits a subtitle extraction job by calling SubmitAIJob, which returns a job ID. Use this job ID with GetAIJob to check the job status.

Request parameters:

JobId: The job ID returned when the workflow calls SubmitAIJob to submit the subtitle extraction job.

Sample response:

{
  "createTime": "2024-09-24T06:17:09Z",
  "finishTime": "2024-09-24T06:17:31Z",
  "functionName": "CaptionExtraction",
  "input": {
    "type": "OSS",
    "media": "oss://example-bucket/example-object.mp4"
  },
  "jobId": "270d****0d1a",
  "jobParams": "{\"fps\":10,\"roi\":[[0.5,1],[0,1]],\"sep\":false}",
  "output": {
    "type": "OSS",
    "media": "oss://example-bucket/output.srt"
  },
  "outputFiles": [
    "output.srt"
  ],
  "outputUrls": [
    "http://example-bucket.oss-region.aliyuncs.com/output.srt"
  ],
  "requestId": "********-****-****-****-************",
  "result": "{}",
  "status": "Success"
}

Note

A Status of Success indicates that the job is complete.
You can use the URL in OutputUrls to download the extracted SRT file.