A hands-on guide to multimodal data processing and large model training

Step 1: Prepare data

This solution uses the open source LLaVa dataset. It uses Data-Juicer to pre-process the data and Pai-Megatron-Patch to convert the data to the Megatron-Energon format. The Megatron-Energon format is an efficient data storage format suitable for distributed training.

• About the LLaVa dataset: This dataset contains text and image data for training cross-modal models.

• About the Data-Juicer data processing flow: Data-Juicer uses data recipes in YAML format to coordinate and simplify the entire data processing flow. This solution uses the data processing recipe for LLaVA data (llava-pretrain-refine.yaml) as an example. A data recipe consists of the following two parts:

  • Global parameters:

    Parameter	Description
    project_name	The project name.
    dataset_path	The data path. This can be a file or a folder.
    export_path	The path where the processed data is saved.
    np	The number of processes for processing. You can also configure the number of processes for a single operator. The general parameter for a single operator is `num_proc`.
    text_keys	The names of the text fields to process. For example: text,instruction,output,...
    Global parameters for multimodal data
    image_key	The name of the field that stores the image paths. This field corresponds to list-type data and stores the paths of multiple images.
    image_special_token	The special marker character for an image. This character is identified from the text to determine if there is image data.
    eoc_special_token	End character token. The default is <|__dj__eoc|>.

  • List of processing operators: A data recipe must contain the registered names of the operators and their corresponding parameters. Data-Juicer processes data in the order of the operator list in the recipe. For a list of all operators in Data-Juicer, see Operators.md. For the configurations supported by the YAML file, see config_all.yaml.

  Additionally, Data-Juicer supports the distributed Ray framework. After you start a Ray cluster, you only need to configure executor_type: ray in the global parameters. PAI-DLC supports the Ray framework, which lets you create a distributed Ray job. For more information, see Create a training task.

Complete the following steps in a Notebook in the DSW development environment:

!pip uninstall -y cmake && apt-get update -y && apt-get install cmake -y && pip install py-data-juicer==1.3.3

!mkdir -p ~/.cache/data_juicer/models
!wget http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release_pai-designer/lm_data_processor/pretrained_models/paiflow_merge_ops/en.sp.model -O ~/.cache/data_juicer/models/en.sp.model
!wget http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release_pai-designer/lm_data_processor/pretrained_models/paiflow_merge_ops/en.arpa.bin -O ~/.cache/data_juicer/models/en.arpa.bin
!wget http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release_pai-designer/lm_data_processor/pretrained_models/Salesforce.zip && unzip Salesforce.zip
!wget http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release_pai-designer/lm_data_processor/pretrained_models/openai.zip && unzip openai.zip

5. Process the data

1. By default, Data-Juicer supports operator fault tolerance. It skips errors from abnormal samples and continues running, which prevents the process from being interrupted by individual abnormal samples. However, this may also cause other errors to be ignored and lead to abnormal processing. During the debugging phase, we recommend that you add the skip_op_error: false parameter to each operator configuration in the llava-pretrain-refine.yaml file to disable this feature. For example, consider the last two operators:

   ```yaml
   - image_text_similarity_filter:
       hf_clip: openai/clip-vit-base-patch32
       min_score: 0.20315419
       skip_op_error: false
   - image_text_matching_filter:
       hf_blip: Salesforce/blip-itm-base-coco
       min_score: 0.44930778
       skip_op_error: false
   ```

2. Execute the following command. The system uses the dj-process tool to process the data according to the sequence of operators in the data recipe. The processed data is output to ./data_output/dj_llava_refined.jsonl.

   !pip install -U transformers
   !dj-process --config ./llava-pretrain-refine.yaml

3. Check whether the data processing was successful and count the number of output data entries.

   !wc -l data_output/dj_llava_refined.jsonl

   The following is an example result:

   81 data_output/dj_llava_refined.jsonl

6. Convert the data format

1. Run the following command to convert the data from the Data-Juicer format to the original format of the LLaVa dataset. For tools to convert between the Data-Juicer format and some open source data formats, see fmt_conversion.

   1. Download the data conversion script to the current directory.

      !curl -O https://raw.githubusercontent.com/modelscope/data-juicer/main/tools/fmt_conversion/multimodal/data_juicer_format_to_target_format/dj_to_llava.py

   2. Run the following command to update the data conversion script. Change the output field name for the image to images and the field storage type to List. This meets the format requirements for the PAI-Megatron-Patch training dataset.

      Note

      After the code is executed, confirm that the file has been modified. If the file was not modified, manually change new_sample['image'] = image_path to new_sample['images'] = [image_path].

      !sed -i.bak "s/new_sample\['image'\] = image_path/new_sample['images'] = [image_path]/g" dj_to_llava.py

   3. Transform the processed data file ./data_output/dj_llava_refined.jsonl into the original LLaVa format and save it to ./data/llava_part/refined_llava_format.json.

      !python dj_to_llava.py data_output/dj_llava_refined.jsonl data/llava_part/refined_llava_format.json

2. Convert the open source LLaVa dataset format to the Megatron-Energon format for training.

   1. Run the following code to clone Pai-Megatron-Patch.

      !git clone --recurse-submodules https://github.com/alibaba/Pai-Megatron-Patch.git

   2. Install the transformers library.

      !pip install -U transformers fsspec

   3. Convert the open source LLaVa dataset to the Megatron-Energon format and output the results to .data/llava_part/wds.

      Note

      Because the demo uses a small amount of data, set max-samples-per-tar to 5, train-split to 6, and val-split to 4. This ensures that data is allocated to both the training and validation sets. Adjust these values based on your actual data volume.

      !python Pai-Megatron-Patch/toolkits/multimodal_data_preprocessing/convert_custom_dataset_to_wds_chatml.py \
      --dataset-root data/llava_part/ \
      --json refined_llava_format.json \
      --max-samples-per-tar 5 \
      --train-split 6 \
      --val-split 4

   4. Execute the following code and verify that a .tar file and its corresponding .tar.idx file are output, the .nv-meta folder is created, and the training dataset and validation set files are assigned in .nv-meta/split.yaml.

      !ls -a data/llava_part/wds/
      !cat data/llava_part/wds/.nv-meta/split.yaml

Step 2: Prepare the pre-trained model

Complete the following steps in a DSW Notebook:

1. Run the following code to download the Qwen2-VL-7B-Instruct model as the foundation model.

   !apt-get install git-lfs -y
   !mkdir -p qwen2-vl-ckpts/Qwen2-VL-7B-Instruct && git clone https://www.modelscope.cn/Qwen/Qwen2-VL-7B-Instruct.git qwen2-vl-ckpts/Qwen2-VL-7B-Instruct

   If the download is slow, use the ModelScope tool for a faster download:

   pip install modelscope && modelscope download --model qwen/Qwen2-VL-7B-Instruct --local_dir qwen2-vl-ckpts/Qwen2-VL-7B-Instruct

2. Transform the Checkpoint model format to MCore-Dense and output the result to ./qwen2-vl-ckpts/Qwen2-VL-7B-Instruct-tp2pp2.

   !current_dir=$PWD && \
   cd ${current_dir}/Pai-Megatron-Patch/toolkits/model_checkpoints_convertor/qwen && \
   bash hf2mcore_qwen2_vl_convertor.sh \
   7B \
   "${current_dir}/qwen2-vl-ckpts/Qwen2-VL-7B-Instruct" \
   "${current_dir}/qwen2-vl-ckpts/Qwen2-VL-7B-Instruct-tp2pp2" \
   2  \
   2  \
   false \
   bf16

   The hf2mcore_qwen2_vl_convertor.sh script requires the following parameters:

   Parameter	Description
   MODEL_SIZE=$1	The parameter size of the model structure.
   SOURCE_CKPT_PATH=$2	The path to the downloaded Qwen2-VL-7B-Instruct model.
   TARGET_CKPT_PATH=$3	The path to save the transformed model.
   TP=$4	The number of tensor segments must match the number used during training.
   PP=$5	The pipeline parallelism degree. This value must be the same as the one used for training.
   MG2HF=$6	Specifies whether to transform the model from Megatron format to Hugging Face format. Valid values are as follows: false: The format is not transformed. true: The format is transformed.
   PR=$7	The training precision. Valid values are fp16 or bf16.

Step 3: Pre-train the multimodal model

1. Pre-training and debugging

After you prepare the data and the pre-trained model, execute the following code in a DSW Notebook to debug the pre-training. After the code is executed, the system outputs the pre-trained model to ./output_mcore_qwen2vl_pretrain.

!pip install -U transformers && current_dir=$PWD && \
cd ${current_dir}/Pai-Megatron-Patch/examples/qwen2_vl && \
sh run_mcore_qwen.sh  \
dsw  \
7B   \
1    \
32 \
1e-5   \
1e-6   \
2048  \
2048  \
bf16  \
2   \
2  \
1 \
true \
true \
true   \
false \
false \
20  \
"${current_dir}/data/llava_part/wds" \
"${current_dir}/data/llava_part/wds" \
"${current_dir}/qwen2-vl-ckpts/Qwen2-VL-7B-Instruct-tp2pp2" \
20  \
5  \
"${current_dir}/output_mcore_qwen2vl_pretrain"

The parameter values in the preceding code are for debugging with the small demo dataset. They are used for quick feature validation and do not guarantee accuracy.

2. Create a distributed training task

After successful debugging on DSW, submit a distributed training task with multi-GPU servers in the Deep Learning Containers (DLC) environment. The procedure is as follows:

1. Go to the create task page.

   a. Log on to the PAI console. At the top of the page, select the destination region. In the right part of the page, select the destination workspace, and then click Enter DLC.

   b. On the Deep Learning Containers (DLC) page, click Create Task.

2. On the Create Task page, configure the following key parameters. Use the default values for other parameters.

   Parameter		Description
   Basic Information	Task Name	Enter a custom task name. This solution uses `test_qwen2_vl_dlc`.
   Environment Context	Node Image	Select Registry Address, and enter dsw-registry.cn-wulanchabu.cr.aliyuncs.com/pai/pai-megatron-patch:25.01 in the text box.
   	Dataset	Click Custom Dataset and configure the following parameters: Custom Dataset: Select the created NAS dataset. Mount Path: Set to /mnt/data/nas.
   	Start Command	Configure the start command. The start parameters for the run_mcore_qwen.sh script are the same as for single-node training in DSW. Simply set ENV=$1 to `dlc`. !pip install -U transformers && current_dir=$PWD && \ cd ${current_dir}/Pai-Megatron-Patch/examples/qwen2_vl && \ sh run_mcore_qwen.sh \ dlc \ 7B \ 1 \ 32 \ 1e-5 \ 1e-6 \ 2048 \ 2048 \ bf16 \ 2 \ 2 \ 1 \ true \ true \ true \ false \ false \ 100000 \ "${current_dir}/data/llava_part/wds" \ "${current_dir}/data/llava_part/wds" \ "${current_dir}/qwen2-vl-ckpts/Qwen2-VL-7B-Instruct-tp2pp2" \ 20000 \ 200 \ "${current_dir}/output_mcore_qwen2vl_pretrain"
   Resource Information	Resource Type	Select Lingjun AI Computing.
   	Resource Source	Select Resource Quota.
   	Resource Quota	Select the created Lingjun AI Computing resource quota.
   	Framework	Select PyTorch.
   	Task Resources	Configure the following parameters for the Worker node: Number of Nodes: 2. For multi-node training, set this to the required number of machines. Set GPU (Cards) to 8. Set CPU (Cores) to 90. It cannot be greater than 96. Memory (GiB): 1024

3. Click OK. You are redirected to the Deep Learning Containers (DLC) page. When the status changes to Succeeded, the training task is successful.

3. Convert the model format

After pre-training is complete, run the following command in the DSW development environment Notebook to convert the pre-trained model from MCore-Dense format to HuggingFace format.

!current_dir=$PWD && \
cd ${current_dir}/Pai-Megatron-Patch/toolkits/model_checkpoints_convertor/qwen && \
bash hf2mcore_qwen2_vl_convertor.sh \
7B \
"${current_dir}/output_mcore_qwen2vl_pretrain/checkpoint/finetune-mcore-qwen2-vl-7B-lr-1e-5-minlr-1e-6-bs-1-gbs-32-seqlen-2048-pr-bf16-tp-2-pp-2-cp-1-ac-false-do-true-sp-true-ti-20-wi-5/iter_0000020" \
"${current_dir}/qwen2-vl-ckpts/Qwen2-VL-7B-Instruct-mc2hf" \
2  \
2  \
true \
bf16 \
"${current_dir}/qwen2-vl-ckpts/Qwen2-VL-7B-Instruct/"

The key parameters for the hf2mcore_qwen2_vl_convertor.sh script are described as follows:

Step 4: Deploy the service

Deploy the pre-trained model that was converted to HuggingFace format as an EAS service. The procedure is as follows:

1. Log on to the PAI console. Select a region on the top of the page. Then, select the desired workspace and click Elastic Algorithm Service (EAS).

2. On the Elastic Algorithm Service (EAS) page, click Deploy Service. Then, in the Deploy Service by Using Custom Images section, click Custom Deployment.

3. On the Custom Deployment page, configure the following key parameters, and then click Deploy.

   Parameter		Description
   Basic Information	Service Name	Enter a custom service name. For example, `test_qwen2_vl_7b_instruct`.
   Environment Context	Deployment Method	Select Image Deployment.
   	Image Configuration	On the Registry Address tab, enter the registry address eas-registry-vpc.cn-wulanchabu.cr.aliyuncs.com/pai-eas/pai-quickstart:mllm-deploy-swift3.0.1.
   	Storage Mounting	This solution mounts the pre-trained model using NAS. Click General-purpose NAS and configure the following parameters: Select File System: Select the NAS file system that was used when creating the dataset. File System Mount Target: Select the mount target corresponding to the NAS file system. File system path: The path to the converted hugging face format model stored in NAS. For this solution, set the path to /qwen2-vl-ckpts/Qwen2-VL-7B-Instruct-mc2hf. Mount path: Specify the path for reading the transformed model files. In this solution, the path is /qwen2-vl-7b-instruct.
   	Run Command	Set to swift deploy --model /qwen2-vl-7b-instruct --model_type qwen2_vl --infer_backend pt --stream true --temperature 0 --max_new_tokens 2048 --served_model_name Qwen2-VL-7B-Instruct. The --model parameter must match the mount path configured for the storage mount. The model file is read from this path.
   	Port Number	Set this to 8000.
   Resource Information	Resource Type	Select Public Resources.
   	Deployment Resources	Select a resource specification. This solution sets it to `ecs.gn8v.6xlarge` based on the model's VARM size.
   	Configure System Disk	Set this to 100 GiB.
   Service Registration	Virtual Private Cloud (VPC)	After you mount the NAS file system, the system automatically configures the VPC and vSwitch to be the same as those of the NAS file system. Simply select the security group name.
   	vSwitch
   	Security Group Name

Step 5: Call the service

Online debugging service

1. On the Elastic Algorithm Service (EAS) page, in the Actions column of the destination service, click > Debug Online.

2. On the debug page, in the Online Debugging Request Parameters section, enter the prepared request body in the Body field, add /v1/chat/completions to the request URL text box, and click Send Request.

   • API endpoint: /v1/chat/completions

   • The following is an example request body:

     {
         "model": "Qwen2-VL-7B-Instruct",
         "messages": [
             {
                 "role": "user",
                 "content": [
                     {
                         "type": "image",
                         "image": "<image URL>"
                     },
                     {
                         "type": "text",
                         "text": "Describe the image"
                     }
                 ]
             }
         ]
     }

     Set image to the image URL. If the image is stored on the public network, the EAS service cannot access the public network by default. You must access public or private network resources from EAS to ensure that the EAS service can access the image URL.

Call the service through an API

After the model is deployed, you can use the OpenAI API for inference.

1. View endpoint information.

   On the Elastic Algorithm Service (EAS) page, click the name of the destination service. In the Basic Information section, click View Endpoint Information. In the Endpoint Information panel, you can view the service endpoint and token.

   Note

   Use a public endpoint or a VPC endpoint. If you use a VPC endpoint, the client that calls the service must be in the same VPC as the EAS service.

2. Run code in a terminal to call the service. The following is an example of Python code for image inference:

   Note

   The deployed service can access the public network only if it is configured with a VPC that has public network access. Therefore, when you call the service for inference, the image in the request data must be in Base64-encoded format, not an image URL.

   import base64
   import requests
   from openai import OpenAI
   # Modify OpenAI's API key and API base to use the EAS API server.
   openai_api_key = "<EAS API KEY>"
   openai_api_base = "<EAS API Endpoint>/v1"
   client = OpenAI(
       api_key=openai_api_key,
       base_url=openai_api_base,
   )
   models = client.models.list()
   model = models.data[0].id
   def encode_base64_content_from_url(content_url: str) -> str:
       """Encode a content retrieved from a remote url to base64 format."""
       with requests.get(content_url) as response:
           response.raise_for_status()
           result = base64.b64encode(response.content).decode("utf-8")
       return result
   def main():
       image_url = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"
       stream = True
       image_base64 = encode_base64_content_from_url(image_url)
       chat_completion_from_base64 = client.chat.completions.create(
           messages=[
               {
                   "role": "user",
                   "content": [
                       {
                           "type": "image",
                           "image": f"data:image/jpeg;base64,{image_base64}",
                       },
                       {
                           "type": "text",
                           "text": "Describe the image",
                       },
                   ],
               }
           ],
           model=model,
           max_completion_tokens=1024,
           stream=stream,
       )
       if stream:
           for chunk in chat_completion_from_base64:
               print(chunk.choices[0].delta.content, end="")
       else:
           result = chat_completion_from_base64.choices[0].message.content
           print(result)
   if __name__ == "__main__":
       main()

   Where:

   • openai_api_key: Replace <EAS API KEY> with the EAS service token.

   • openai_api_base: Replace <EAS API Endpoint> with the service endpoint.

   • image_url: Set this to a publicly accessible image URL.