Intelligent highlight editing

Background

With the rapid growth of content industries such as gaming, education, short drama, and live streaming, the production efficiency of promotional materials has become a core bottleneck in content delivery. Traditional manual highlight editing relies on heavy repetitive labor and cannot meet the demands of multiple languages, versions, and platforms.

The highlight editing capability is built on a large model that jointly understands plot, shots, and audio. It provides an end-to-end automated video production solution that supports two modes: continuous mode (CONTINUOUS) and mixed cut mode (STORY_CUT). You only need to provide source videos and content preferences. The server automatically completes candidate recall, timeline planning, boundary verification, and video rendering to produce final videos ready for distribution.

Limits

• A single input video can be up to 3,600 seconds in length. The total input duration for a single task can be up to 14,400 seconds.

• Input videos must contain both decodable video and audio streams. Silent videos fail with MEDIA_INVALID.

• Time fields use seconds. Both index and media_index start from 0.

• The mixed cut mode does not support reverse narration, free segment reordering, or character library capabilities in this release.

Features

Highlight editing tasks are asynchronous tasks under the operator service path /api/v1/operators/highlight-editing. After a task is submitted, a task ID is returned. You must poll the task status through the query API and retrieve the final video results. Both modes use the large model for planning, and the server validates the timeline before exporting with FFmpeg.

Usage

{
  "media_inputs": [
    {"uri": "https://media.example.com/episode-01.mp4"},
    {"uri": "https://media.example.com/episode-02.mp4"}
  ],
  "highlight_description": "Prioritize scenes with strong visual impact and character tension",
  "editing": {
    "mode": "STORY_CUT",
    "output_count": 2,
    "target_duration": 120,
    "per_media_target_duration": 60,
    "require_all_media": false,
    "opening_hook": {
      "description": "Prioritize moments of escalating action",
      "target_duration": 10
    }
  },
  "output": {
    "need_export": true,
    "oss": {
      "region": "cn-shanghai",
      "bucket": "customer-bucket",
      "object_key": "highlight-edits/{task_id}/output_{index}.mp4"
    }
  }
}

{
  "status": "SUCCESS",
  "message": null,
  "data": {"task_id": "1720000000000_efgh", "task_status": "PENDING"}
}

curl -X POST 'http://{endpoint}/api/v1/operators/highlight-editing/tasks' \
  -H 'Content-Type: application/json' \
  -d '{
    "media_inputs": [{"uri": "https://media.example.com/episode-01.mp4"}],
    "editing": {
      "mode": "CONTINUOUS",
      "output_count": 1,
      "target_duration": 120,
      "opening_hook": {"target_duration": 10}
    },
    "output": {"need_export": false}
  }'

curl 'http://{endpoint}/api/v1/operators/highlight-editing/tasks/1720000000000_efgh'

General constraints and error codes

• Single input video: up to 3,600s. Single task total: up to 14,400s.

• Input must have video and audio streams. Silent videos fail with MEDIA_INVALID.

• The server validates time boundaries based on shot, VAD, and ASR results.

• When need_export=true, FFmpeg rendering and OSS upload are performed. Platform URLs are valid for 7,200s by default.

• Default output key: highlight-outputs/{task_id}/output_{index}.mp4.

On failure, data.error.code returns: MEDIA_UNAVAILABLE, MEDIA_INVALID, MEDIA_LIMIT_EXCEEDED, ASR_FAILED, MODEL_FAILED, MODEL_OUTPUT_INVALID, PLAN_INVALID, RENDER_FAILED, UPLOAD_FAILED, or INTERNAL_ERROR.

Failed task query example

{
  "status": "SUCCESS",
  "message": null,
  "data": {
    "task_id": "1720000000000_efgh",
    "task_status": "FAILED",
    "error": {
      "code": "MODEL_FAILED",
      "message": "Media analysis or model recall failed."
    }
  }
}

Use case: Short drama overseas distribution

Short drama overseas platforms need to produce promotional materials at scale with strong opening hooks. The highlight editing capability automates this process:

1. Submit: Place episodes in media_inputs by plot order, select STORY_CUT, and specify opening_hook preferences.

2. Model planning: The server performs joint understanding of shots, audio, and plot, recalls highlights across episodes, and plans the timeline.

3. Rendering: The server validates the timeline, renders with FFmpeg, and uploads to customer OSS.

4. Retrieve: Poll until task_status=SUCCESS, then get the URL from outputs[i].output_url.