Subtitle removal

Updated at:

ApsaraVideo VOD workflow 2.0 integrates the subtitle removal feature from Intelligent Media Services (IMS). It identifies and removes subtitles from videos, then restores the background using smart inpainting for automated post-production.

Note

This feature is available in the following regions:

  • China (Shanghai), China (Beijing), Asia Pacific SE 1 (Singapore), and US West 1 (Silicon Valley).

Prerequisites

Benefits

Two versions of subtitle removal are available:

  • Advanced (Recommended): Seamlessly removes subtitles and resolves mosaic and shadow artifacts for a more natural-looking video. To enable this version, use the ModelId parameter. For more information, see ModelId parameter.

  • The advanced version uses a superior algorithm, but it processes videos more slowly and costs more than the basic version. Choose the version that best fits your business needs.

Billing

Subtitle removal in ApsaraVideo VOD workflows is powered by Intelligent Media Services (IMS). You must purchase an IMS subscription service and pay usage fees. For more information, see Subtitle removal.

Usage

Step 1: Configure the workflow

You can create workflows only in the ApsaraVideo VOD console.

  1. Log on to the ApsaraVideo VOD console.

  2. In the left-side navigation pane, choose Configuration Management > Media Processing > Workflows.

  3. Click Add Workflow Template and enter a name for the workflow.

  4. On the workflow editor page, click the + icon to the right of the Start node to add a subtitle removal node.

  5. Configure the following parameters in the right-side panel:

    • Node Name: Enter a custom name for the subtitle removal node.

    • Sample Material: A sample video used to configure subtitle settings. This video is not processed by the workflow. Supported formats are MP4, WebM, MOV, and M3U8.

    • Recognition Area: The default area is the bottom 1/4 of the video. You can also manually select an area.

    • Algorithm Version: Two versions are available: Basic and Advanced.

      • Advanced (Recommended): Seamlessly removes subtitles across multiple time ranges and resolves mosaic and shadow artifacts for a more natural-looking result. To enable this version, set the ModelId parameter. For more information, see ModelId parameter.

      • Basic: The output may contain mosaic or shadow artifacts, and the restoration quality is average.

    • Time Range: You can specify a custom start time and end time.

      Note
      • You must specify the start and end times as a pair.

      • The start time cannot be later than the end time.

      • If you do not specify a time range, the entire video is processed by default.

      • The advanced version supports up to five time ranges, while the basic version supports only one.

  6. After you finish the configuration, click OK. The system generates a workflow ID. Record this ID for use when uploading videos. The new Subtitle Removal workflow template appears in the template list on the Workflow Management page with a status of Enabled. In the Actions column, you can click Details to view the configuration or Disable to disable the template.

Step 2: Trigger the workflow

You can trigger this workflow during or after video upload.

Console

During upload

  1. Log on to the ApsaraVideo VOD console.

  2. In the left-side navigation pane, choose Media Files > Audio/Video, and then click Upload Audio/Video.

  3. On the Upload Audio/Video page, select Use Workflow as the upload method and add your video. In the processing options for the file, select Process with Workflow, choose the Subtitle Removal workflow from the Transcoding Template Group/Workflow Template list, and then click Start Upload.

After upload

  1. Log on to the ApsaraVideo VOD console.

  2. In the left-side navigation pane, choose Media Files > Audio/Video.

  3. Find the target video and click Processing in the Actions column. In the Media Processing dialog box, set Processing Type to Process with Workflow, select the Subtitle Removal workflow, and then click OK.

API

During upload

  • The CreateUploadVideo API operation obtains an upload URL and credentials and creates basic media asset information but does not upload the media file itself. You must implement the complete upload logic. For a full example of uploading by using an API, see Upload media files by using the ApsaraVideo VOD API.

  • When you call the CreateUploadVideo or UploadMediaByUrl API operation to upload an audio or video file, set the WorkflowId parameter to the ID of the workflow that you created in the previous step. After the upload is complete, ApsaraVideo VOD automatically processes the file based on the specified workflow.

After upload

Call the SubmitWorkflowJob API operation and set the WorkflowId parameter to your workflow ID. This immediately starts processing for the specified media file.

Step 3: Query results

VOD console

  1. Log on to the ApsaraVideo VOD console.

  2. In the left-side navigation pane, choose Media Files > Audio/Video.

  3. On the Audio/Video page, find the video generated by the workflow. You can filter by the source video name or creation time.

IMS console

  1. Log on to the Intelligent Media Services console.

  2. In the left-side navigation pane, choose VOD Media Processing > Task Management, and then click the Smart Removal tab.

  3. On the Smart Removal tab, find the removal task generated by the workflow. You can filter by name or creation time.

  4. When the task status is Success, click View to see the details:

    • Basic parameters and configuration of the removal task.

    • Input information for the removal video.

    • Output information and output file of the removal video.

Callback

  1. Configure HTTP or Message Service (MNS) callbacks.

  2. When a workflow task is complete, the system triggers a WorkflowTaskComplete event and pushes the results through the configured HTTP or Message Service (MNS) callback. Key fields in the callback include:

    1. Status: The overall status of the task (Succeed/Failed).

    2. ActivityResults: A JSON string containing the execution details of each node, such as VideoDetext. The Result field contains key output information, such as the MediaId and JobId.

    3. TaskInput: The original input media information, such as the Media ID and filename.

  3. Parse ActivityResults in the callback payload to extract the MediaId of the generated video for playback or distribution.

API

Call the GetAIMediaAuditJob API operation to query the task result. For the JobId parameter, use the ID from ActivityResults.Result.JobId in the workflow task callback. The API response contains the detailed status and output of the job.

  • JobId: The job ID returned when the workflow internally calls the SubmitAIMediaAuditJob operation to submit the subtitle removal task.

Sample response:

{
  "RequestId": "****20b48fb04483915d4f2cd8ac****",
  "JobId": "****20b48fb04483915d4f2cd8ac****",
  "FunctionName": "VideoDetext",
  "Input": {
    "Type": "OSS",
    "Media": "oss://example-bucket/input.mp4"
  },
  "Output": {
    "Type": "OSS",
    "Media": "oss://example-bucket/output.mp4",
    "OutputUrl": "http://example-bucket.oss-cn-shanghai.aliyuncs.com/output.mp4"
  },
  "Status": "Success",
  "CreateTime": "2024-09-24T06:17:09Z",
  "FinishTime": "2024-09-24T06:17:31Z",
  "OutputFiles": ["output.mp4"],
  "OutputUrls": ["http://example-bucket.oss-cn-shanghai.aliyuncs.com/output.mp4"],
  "Result": {}
}
Note
  • A Status of Success indicates that the subtitle removal is complete.

  • You can access the processed video using the URLs in the OutputUrls field.