The LVM-Caption-Frames Mapper (DLC) component automatically generates text captions for MP4 videos. It samples frames at even intervals, runs each frame through the Bootstrapped Language Image Pretraining (BLIP) model, and writes the resulting captions to a JSONL file in Object Storage Service (OSS). Use this component in a Machine Learning Designer pipeline to produce caption datasets for large vision model (LVM) training.
Only MP4 videos are supported.
Supported computing resources
How it works
-
Sample a fixed number of frames from the MP4 video, distributed evenly across the video duration.
-
Run each frame through the BLIP model to generate one or more candidate captions.
-
Write all captions to an output JSONL file in the specified OSS directory.
Inputs and outputs
Input
Connect one of the following upstream sources to this component:
| Option | When to use |
|---|---|
| Read File Data component | Use this component to read the OSS path where your training data is stored. Configure the data path using its OSS Data Path parameter. |
| Any LVM Data Processing (DLC) component | Pass processed video data directly from another component in the same pipeline. |
| Video Data OSS Path parameter (no upstream) | On the first run when no upstream component is connected, specify the OSS directory manually. See the parameter description below. |
Output
The component writes results to the OSS directory specified by Output File OSS Path.
| File | Description |
|---|---|
{name}.jsonl |
Caption output file. The file name is set by the Output Filename parameter. |
dj_run_yaml.yaml |
Parameter configuration file used during the run. |
Configure the component
Configure parameters in Machine Learning Designer. The following table describes all parameters.
| Tab | Parameter | Required | Description | Default |
|---|---|---|---|---|
| Field Settings | Video Data OSS Path | No | The OSS directory containing your video data. See the usage note below the table. | — |
| Field Settings | Output File OSS Path | Yes | The OSS directory where output files are saved. | — |
| Field Settings | Output Filename | Yes | The file name of the caption output file. | result.jsonl |
| Parameter Settings | Number of Candidate Captions | Yes | The number of caption candidates generated per frame. | 1 |
| Parameter Settings | Number of Sampled Frames | Yes | The number of frames sampled from each video. Frames are selected at even intervals based on video duration. | 3 |
| Execution Tuning | Select Resource Group – Public Resource Group | No | The instance type (CPU or GPU) and virtual private cloud (VPC) to use. Select a GPU instance type. | — |
| Execution Tuning | Select Resource Group – Dedicated resource group | No | The number of vCPUs, memory, shared memory, and GPUs to allocate. | — |
| Execution Tuning | Maximum Running Duration (seconds) | No | The maximum time the component is allowed to run. If this limit is exceeded, the job is terminated. | — |
Video Data OSS Path: first run vs. subsequent runs
The Video Data OSS Path parameter behaves differently depending on whether you have run the component before:
| Scenario | What to specify |
|---|---|
| First run (no upstream component) | Specify the OSS directory where your video data is stored. |
| Subsequent runs | Point directly to the video_meta.jsonl file that was generated during the first run. |
On the first run, the component creates a video_meta.jsonl metadata file in the parent directory of the path you specify — not in the path itself.