DocsICP FilingConsole
官方文档
首页

Submit DataJuicer jobs

更新时间：
复制 MD 格式
产品详情
我的收藏

This document describes how to use the open-source tool DataJuicer on DLC for large-scale multimodal data processing.

Background

As artificial intelligence and large models advance, data quality has become a critical factor limiting model accuracy and reliability. This creates a greater need for unified processing and collaborative modeling of multimodal data, such as text, images, and videos. Systematically collecting, cleaning, enhancing, and synthesizing large-scale data to ensure its accuracy, diversity, and representativeness remains a core challenge in large model development.

DataJuicer is an open-source tool that specializes in processing large-scale multimodal data, such as text, images, audio, and video. It helps researchers and developers efficiently clean, filter, transform, and enhance large datasets, providing higher-quality, richer, and more "digestible" data for large language models (LLMs).

PAI has launched a new task type, DataJuicer on DLC, to provide users with an out-of-the-box, high-performance, stable, and efficient data processing capability.

Features

DataJuicer on DLC is a data processing service from PAI and Tongyi Lab. It allows you to submit DataJuicer framework jobs in the cloud with a single click, enabling efficient cleaning, filtering, transformation, and enhancement of large-scale data for multimodal data processing in LLM workloads.

  • Rich operators: Provides over 100 core operators, including aggregators, duplicators, filters, formatters, groupers, mappers, and selectors. These cover the entire data processing lifecycle, from data loading and normalization to editing, transformation, filtering, deduplication, and high-quality sample selection. You can compose operator chains to meet your business needs.

  • Excellent performance: Delivers excellent linear scalability and data processing speed. For multimodal data processing at the scale of tens of millions of samples, it reduces processing time by 24.8% compared to native nodes.

  • Resource estimation: Supports resource estimation to intelligently balance resource constraints with runtime efficiency. It automatically tunes operator parallelism to significantly reduce task failures caused by Out of Memory (OOM) errors. By analyzing dataset, operator, and resource quota information, the system automatically estimates the optimal resource configuration, which lowers the barrier to entry and ensures efficient and stable task execution.

  • Large-scale processing: Built on PAI DLC's distributed computing framework and deep hardware acceleration optimizations (CUDA/operator fusion), it supports efficient processing of workloads ranging from experimental datasets with thousands of samples to production data with tens of billions of samples.

  • Automatic fault tolerance: PAI DLC provides fault tolerance and self-healing at the node, task, and container levels. In addition, DataJuicer offers operator-level fault tolerance to handle interruptions caused by infrastructure failures, such as server or network issues.

  • Ease of use: You can submit DataJuicer tasks for AI data processing with a single click, eliminating the need for deployment, O&M, and infrastructure management.

Instructions

1. Select an image and framework

The image for a DataJuicer task must have the DataJuicer environment pre-installed and include the dj-process command. We recommend using an official image from the data-juicer repository or a custom image based on an official data-juicer image.

image

Set Framework to DataJuicer.

image

2. Configure run mode

When you create a DLC task, you can choose between single-node and distributed run modes. Ensure that your choice matches the executor_type in your configuration file.

  • Single-node mode:

    • DataJuicer configuration file: In the configuration file, set executor_type to default or omit this field.

    • DLC configuration:

      • Run Mode: Select Single node.

      • Nodes: Set to 1.

  • Distributed mode:

    • DataJuicer configuration file: In the configuration file, you must set executor_type to ray.

    • DLC configuration:

      • Run Mode: Select Distributed.

      • Resource Estimation: This option is available only when you use a Resource Quota. When enabled, the system intelligently estimates the optimal resource configuration based on dataset, operator, and quota information, and then runs the task automatically to ensure efficient execution. You can also set a maximum resource limit for the task if needed.

        • Maximum Task Resource Limit: If you need to set an upper limit on the resources requested by the DataJuicer task, configure this option. The total resources requested by the task will not exceed the configured limit. If you leave this field blank, the system automatically requests resources based on the estimation result.

      • Task Resources: If you do not enable resource estimation, you must manually specify the task resources.

        • Nodes: The number of head nodes must be 1, and the number of worker nodes must be at least 1.

        • Instance Type: The head node requires a resource specification with more than 8 GB of memory. Configure the worker resource specification based on your needs.

      • Fault Tolerance and Diagnosis (Optional): You can configure head node fault tolerance by selecting a Redis instance within the same Virtual Private Cloud (VPC).

3. Enter startup command

DLC supports startup commands in Shell and YAML formats, with Shell as the default. The Shell format works the same as it does for other DLC tasks. When using the YAML format, you can enter the DataJuicer configuration directly in the command input field.

image

You can use DataJuicer by creating a configuration file. For details, see Build a Configuration File. For a complete configuration reference, see config_all.yaml. The following figure shows a sample DataJuicer configuration:

image.png

The key parameters are as follows:

  • dataset_path: The path to the input data. For a DLC task, this path must be where your data storage, such as Object Storage Service (OSS), is mounted inside the container.

  • export_path: The output path for the processed results. For a distributed task, this path must be a directory, not a specific file.

  • executor_type: The executor type.

    • default indicates that DefaultExecutor will be used to run on a single node.

    • ray indicates that RayExecutor will be used. RayExecutor supports distributed processing. For details, see Data-Juicer Distributed Data Processing.

The following are examples of how to configure the startup command on DLC:

  • Shell format example 1: Write the configuration to a temporary file and start the task by using the dj-process command.

    Command example 1

    set -ex

cat > /tmp/run_config.yaml <<EOL
# Process config example for dataset

# global parameters
project_name: 'ray-demo'
dataset_path: '/mnt/data/process_on_ray/data/demo-dataset2.jsonl'  # path to your dataset directory or file
export_path: '/mnt/data/data-juicer-outputs/20250728/01/process_on_ray/result.jsonl'

executor_type: 'ray'
ray_address: 'auto'                     # change to your ray cluster address, e.g., ray://<hostname>:<port>
np: 12

# process schedule
# a list of several process operators with their arguments
process:
  # Filter ops
  - alphanumeric_filter:                                    # filter text with alphabet/numeric ratio out of specific range.
      tokenization: false                                     # Whether to count the ratio of alphanumeric to the total number of tokens.
      min_ratio: 0.0                                          # the min ratio of filter range
      max_ratio: 0.9                                          # the max ratio of filter range
  - average_line_length_filter:                             # filter text with the average length of lines out of specific range.
      min_len: 10                                             # the min length of filter range
      max_len: 10000                                          # the max length of filter range
  - character_repetition_filter:                            # filter text with the character repetition ratio out of specific range
      rep_len: 10                                             # repetition length for char-level n-gram
      min_ratio: 0.0                                          # the min ratio of filter range
      max_ratio: 0.5                                          # the max ratio of filter range
  - flagged_words_filter:                                   # filter text with the flagged-word ratio larger than a specific max value
      lang: en                                                # consider flagged words in what language
      tokenization: false                                     # whether to use model to tokenize documents
      max_ratio: 0.0045                                       # the max ratio to filter text
      flagged_words_dir: ./assets                             # directory to store flagged words dictionaries
      use_words_aug: false                                    # whether to augment words, especially for Chinese and Vietnamese
      words_aug_group_sizes: [2]                              # the group size of words to augment
      words_aug_join_char: ""                                 # the join char between words to augment
  - language_id_score_filter:                               # filter text in specific language with language scores larger than a specific max value
      lang: en                                                # keep text in what language
      min_score: 0.8                                          # the min language scores to filter text
  - maximum_line_length_filter:                             # filter text with the maximum length of lines out of specific range
      min_len: 10                                             # the min length of filter range
      max_len: 10000                                          # the max length of filter range
  - perplexity_filter:                                      # filter text with perplexity score out of specific range
      lang: en                                                # compute perplexity in what language
      max_ppl: 1500                                           # the max perplexity score to filter text
  - special_characters_filter:                              # filter text with special-char ratio out of specific range
      min_ratio: 0.0                                          # the min ratio of filter range
      max_ratio: 0.25                                         # the max ratio of filter range
  - stopwords_filter:                                       # filter text with stopword ratio smaller than a specific min value
      lang: en                                                # consider stopwords in what language
      tokenization: false                                     # whether to use model to tokenize documents
      min_ratio: 0.3                                          # the min ratio to filter text
      stopwords_dir: ./assets                                 # directory to store stopwords dictionaries
      use_words_aug: false                                    # whether to augment words, especially for Chinese and Vietnamese
      words_aug_group_sizes: [2]                              # the group size of words to augment
      words_aug_join_char: ""                                 # the join char between words to augment
  - text_length_filter:                                     # filter text with length out of specific range
      min_len: 10                                             # the min length of filter range
      max_len: 10000                                          # the max length of filter range
  - words_num_filter:                                       # filter text with number of words out of specific range
      lang: en                                                # sample in which language
      tokenization: false                                     # whether to use model to tokenize documents
      min_num: 10                                             # the min number of filter range
      max_num: 10000                                          # the max number of filter range
  - word_repetition_filter:                                 # filter text with the word repetition ratio out of specific range
      lang: en                                                # sample in which language
      tokenization: false                                     # whether to use model to tokenize documents
      rep_len: 10                                             # repetition length for word-level n-gram
      min_ratio: 0.0                                          # the min ratio of filter range
      max_ratio: 0.5                                          # the max ratio of filter range
EOL

dj-process --config /tmp/run_config.yaml

  • Shell format example 2: Save the configuration file to cloud storage, such as Object Storage Service (OSS), mount it to the DLC container, and run the task by directly specifying the mounted configuration file with dj-process.

    dj-process --config /mnt/data/process_on_ray/config/demo.yaml

  • YAML format example: Enter the DataJuicer configuration directly in the command field.

    YAML format command example

    # Process config example for dataset

# global parameters
project_name: 'ray-demo'
dataset_path: '/mnt/data/process_on_ray/data/demo-dataset2.jsonl'  # path to your dataset directory or file
export_path: '/mnt/data/data-juicer-outputs/20250728/01/process_on_ray/result.jsonl'

executor_type: 'ray'
ray_address: 'auto'                     # change to your ray cluster address, e.g., ray://<hostname>:<port>
np: 12

# process schedule
# a list of several process operators with their arguments
process:
  # Filter ops
  - alphanumeric_filter:                                    # filter text with alphabet/numeric ratio out of specific range.
      tokenization: false                                     # Whether to count the ratio of alphanumeric to the total number of tokens.
      min_ratio: 0.0                                          # the min ratio of filter range
      max_ratio: 0.9                                          # the max ratio of filter range
  - average_line_length_filter:                             # filter text with the average length of lines out of specific range.
      min_len: 10                                             # the min length of filter range
      max_len: 10000                                          # the max length of filter range
  - character_repetition_filter:                            # filter text with the character repetition ratio out of specific range
      rep_len: 10                                             # repetition length for char-level n-gram
      min_ratio: 0.0                                          # the min ratio of filter range
      max_ratio: 0.5                                          # the max ratio of filter range
  - flagged_words_filter:                                   # filter text with the flagged-word ratio larger than a specific max value
      lang: en                                                # consider flagged words in what language
      tokenization: false                                     # whether to use model to tokenize documents
      max_ratio: 0.0045                                       # the max ratio to filter text
      flagged_words_dir: ./assets                             # directory to store flagged words dictionaries
      use_words_aug: false                                    # whether to augment words, especially for Chinese and Vietnamese
      words_aug_group_sizes: [2]                              # the group size of words to augment
      words_aug_join_char: ""                                 # the join char between words to augment
  - language_id_score_filter:                               # filter text in specific language with language scores larger than a specific max value
      lang: en                                                # keep text in what language
      min_score: 0.8                                          # the min language scores to filter text
  - maximum_line_length_filter:                             # filter text with the maximum length of lines out of specific range
      min_len: 10                                             # the min length of filter range
      max_len: 10000                                          # the max length of filter range
  - perplexity_filter:                                      # filter text with perplexity score out of specific range
      lang: en                                                # compute perplexity in what language
      max_ppl: 1500                                           # the max perplexity score to filter text
  - special_characters_filter:                              # filter text with special-char ratio out of specific range
      min_ratio: 0.0                                          # the min ratio of filter range
      max_ratio: 0.25                                         # the max ratio of filter range
  - stopwords_filter:                                       # filter text with stopword ratio smaller than a specific min value
      lang: en                                                # consider stopwords in what language
      tokenization: false                                     # whether to use model to tokenize documents
      min_ratio: 0.3                                          # the min ratio to filter text
      stopwords_dir: ./assets                                 # directory to store stopwords dictionaries
      use_words_aug: false                                    # whether to augment words, especially for Chinese and Vietnamese
      words_aug_group_sizes: [2]                              # the group size of words to augment
      words_aug_join_char: ""                                 # the join char between words to augment
  - text_length_filter:                                     # filter text with length out of specific range
      min_len: 10                                             # the min length of filter range
      max_len: 10000                                          # the max length of filter range
  - words_num_filter:                                       # filter text with number of words out of specific range
      lang: en                                                # sample in which language
      tokenization: false                                     # whether to use model to tokenize documents
      min_num: 10                                             # the min number of filter range
      max_num: 10000                                          # the max number of filter range
  - word_repetition_filter:                                 # filter text with the word repetition ratio out of specific range
      lang: en                                                # sample in which language
      tokenization: false                                     # whether to use model to tokenize documents
      rep_len: 10                                             # repetition length for word-level n-gram
      min_ratio: 0.0                                          # the min ratio of filter range
      max_ratio: 0.5                                          # the max ratio of filter range

Use case

Massive video data processing

With the breakthrough applications of multimodal large language models (MLLMs) in fields such as autonomous driving and embodied intelligence, the fine-grained processing of massive video data has become a key technology for gaining a competitive edge. In autonomous driving scenarios, models must analyze complex road conditions, traffic signs, and pedestrian behavior in real time from continuous video streams from in-vehicle cameras. In embodied intelligence, systems rely on video data to build dynamic representations of the physical world for high-level tasks such as robot motion planning and environmental interaction. However, traditional data processing solutions face three core challenges:

  • Modality separation: Video data contains heterogeneous information from multiple sources, including visuals, audio, time series, and text descriptions. This requires a toolchain for cross-modal feature fusion, as traditional pipeline-style tools struggle to perform global correlation analysis.

  • Quality bottlenecks: Data cleaning involves multiple stages, such as deduplication, annotation repair, keyframe extraction, and noise filtering. Traditional multi-stage processing can lead to information loss and redundant computation.

  • Engineering inefficiency: Processing large-scale video data (at the terabyte or petabyte scale) places high demands on distributed computing power scheduling and heterogeneous hardware adaptation. Self-built systems often have long development cycles and low resource utilization.

The PAI-DLC DataJuicer framework provides an end-to-end solution to these challenges, with the following technical advantages:

  • Multimodal collaborative processing engine: Built-in operators for text, images, video, and audio support the joint cleaning and enhancement of visual, textual, and temporal data, avoiding the fragmented processing of traditional toolchains.

  • Cloud-native elastic architecture: Deeply integrates PAI's distributed storage acceleration (at hundreds of GB/s) and heterogeneous GPU/CPU resource pooling capabilities, supporting automatic scaling for tasks with thousands of nodes.

Procedure

This use case demonstrates a video processing workflow for autonomous driving and embodied intelligence. It shows how to use DataJuicer to:

  1. Filter out video clips from the raw data that are too short.

  2. Filter out low-quality data based on NSFW scores.

  3. Extract frames from the videos and generate text captions.

Prepare the data

This example uses 2,000 video data entries from the Youku-AliceMind dataset, which have been uploaded to Object Storage Service (OSS).

Create a DLC task

Create a DLC task and configure the following key parameters. You can leave the other parameters at their default values.

  • Image Configuration: Select Alibaba Cloud Image, then search for and select data-juicer:1.4.3-pytorch2.6-gpu-py310-cu121-ubuntu22.04.

  • Mount storage: Select OSS

    • Uri: Select the OSS directory where the dataset is located.

    • Mount Path: The default is /mnt/data/.

  • Startup Command: Select YAML and enter the following command:

    Startup command example

    # global parameters
project_name: 'dj-video-demo'
# Dataset mount path
dataset_path: '/mnt/data/data/Youku-AliceMind/caption/validation/youku_alice_mind_dj_2k.jsonl' 

executor_type: 'ray' 
skip_op_error: false  # For debugging
export_type: 'jsonl'
export_path: '/mnt/data/outputs/video_demo/v1'
video_key: 'videos'
video_special_token: '<__dj__video>'

eoc_special_token: '<|__dj__eoc|>'

# process schedule
# a list of several process operators with their arguments
process:
  - video_duration_filter:
      min_duration: 0
      max_duration: 3600
      any_or_all: any
  - video_nsfw_filter:
      hf_nsfw_model: Falconsai/nsfw_image_detection
      max_score: 0.5
      frame_sampling_method: all_keyframes
      frame_num: 3
      reduce_mode: avg
      any_or_all: any
  - video_captioning_from_frames_mapper:
      hf_img2seq: 'Salesforce/blip2-opt-2.7b'
      caption_num: 1
      keep_candidate_mode: 'random_any'
      keep_original_sample: true
      frame_sampling_method: 'all_keyframes'
      frame_num: 3

  • Source: Select Public Resources.

  • Framework: Select DataJuicer.

  • Running Mode: Select Distributed.

  • Job Resource: Configure the nodes and resources as shown in the figure:

    image

Click OK to create the task.

Previous：Submit a Ray jobNext：Submit Slurm Training Jobs Quickly