This guide helps large language model (LLM) developers use the Lingjun AI Computing platform. It describes the complete development workflow for the sparse LLM, Mixtral. The workflow includes efficient distributed training, three-stage instruction fine-tuning, offline model inference, and online service deployment. This topic uses the Mixtral-8x7B model as an example to describe this process in detail.
Prerequisites
This topic uses the Mixtral-8x7B model as an example. Before you begin, complete the following preparations:
Activate Platform for AI (PAI), including Data Science Workshop (DSW), Deep Learning Containers (DLC), and Elastic Algorithm Service (EAS), and create a default workspace. For more information, see Activate PAI and create a default workspace.
Purchase Lingjun resources and create a resource quota. The following table lists the supported resource specifications for different model parameter sizes. Select the appropriate resources based on your model's parameter size. For more information about the node specifications of Lingjun resources, see Lingjun Serverless pricing details. For more information about how to purchase resources, see Create a resource group and purchase Lingjun resources and Create a resource quota.
Model parameters
Full-parameter training resources
Inference resources (minimum)
Megatron training model slicing
7B, 8 experts
2 servers with 8 GPUs each (gu7xf, gu7ef)
4 × V100 (32 GB VRAM), 4 × A10 (22 GB VRAM)
TP2, PP1
Create a General-purpose NAS file system dataset to store training and result files. Configure the default mount path to
/mnt/data/nas. For more information, see Create and manage datasets.Create a DSW instance with the following key parameter settings. For more information, see Create a DSW instance.
Resource Quota: Select the resource quota for the Lingjun resources that you created.
Resource Specifications: Configure the following resource specifications.
CPU (Cores): 90
Memory (GiB): 1024
Shared Memory (GiB): 1024
GPU (Cards): At least 8
Dataset Mounting: Click Custom Dataset, select the dataset that you created, and use the default mount path.
Image: On the Registry Address tab, set the image to
pai-image-manage-registry.cn-wulanchabu.cr.aliyuncs.com/pai/pytorch-training:1.12-ubuntu20.04-py3.10-cuda11.3-megatron-patch-llm.
If you use a Resource Access Management (RAM) user to perform these operations, you must grant the required permissions for DSW, DLC, or EAS to the RAM user. For more information, see Cloud product dependencies and authorization: DSW, Cloud product dependencies and authorization: DLC, or Cloud product dependencies and authorization: EAS.
Limits
This best practice is available only in the China (Ulanqab) region.
Step 1: Prepare Mixtral-8x7B-v0.1
This topic provides two ways to download the model. Choose one of the following methods:
Go to the development environment of DSW.
Log on to the PAI console.
In the upper-left corner of the page, select the China (Ulanqab) region.
In the left-side navigation pane, click Workspaces. On the Workspaces page, click the name of the workspace that you want to manage.
In the left-side navigation pane, choose .
Find the DSW instance that you want to manage and click Open in the Actions column.
In the top menu bar, click Terminal, and then click Create Terminal in the new tab.
Download the Mixtral-8x7B-v0.1 model.
Download the model from the ModelScope community
Run the following command in the Terminal to install ModelScope.
Run the following command to enter the Python environment.
The following code is an example of how to download the Mixtral-8x7B-v0.1 model file. To download the Mixtral-8x7B-Instruct-v0.1 model file, click the corresponding model link in the table below to view the code.
Press
Ctrl+Dto exit the Python environment.Run the following command to move the downloaded Mixtral-8x7B-v0.1 model to the appropriate folder.
pip install modelscopepython# ### Loading Model and Tokenizer from modelscope.hub.snapshot_download import snapshot_download model_dir = snapshot_download('AI-ModelScope/Mixtral-8x7B-v0.1', 'v1.1.4') # Get the download path print(model_dir) # model_dir = snapshot_download('AI-ModelScope/Mixtral-8x7B-Instruct-v0.1', 'v1.1.4')Model type
Model link
Model name
Version
Mixtral-8x7B-v0.1
AI-ModelScope/Mixtral-8x7B-v0.1
v1.1.4
Mixtral-8x7B-Instruct-v0.1
AI-ModelScope/Mixtral-8x7B-Instruct-v0.1
# mkdir -p /mnt/workspace/mixtral-ckpts/${your_hf_ckpt_folder} mkdir -p /mnt/workspace/mixtral-ckpts/Mixtral-8x7B-v0.1 # cp -r ${your_downloaded_model_path}/* /mnt/workspace/mixtral-ckpts/${your_hf_ckpt_folder} cp -r /root/.cache/modelscope/hub/mixtral/Mixtral-8x7B-v0.1/* /mnt/workspace/mixtral-ckpts/Mixtral-8x7B-v0.1Download a model from the Hugging Face community
Run the following commands in the DSW Terminal to download the model file. This guide uses the Mixtral-8x7B-v0.1 model as an example. To download the Mixtral-8x7B-Instruct-v0.1 model file, you must modify the following code.
mkdir /mnt/workspace/mixtral-ckpts cd /mnt/workspace/mixtral-ckpts git clone https://huggingface.co/mistralai/Mixtral-8x7B-v0.1 # git clone https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1
Step 2: Prepare pre-training data
Prepare the pre-training data in a DSW instance. This topic uses the WuDaoCorpora2.0 dataset (for research purposes only) as an example to demonstrate the data pre-processing workflow for Megatron training. You can download the small sample dataset prepared by PAI or follow these steps to prepare your own pre-training data.
Use the small sample dataset processed by PAI
To help you try this example, PAI provides a processed small sample dataset. You can run the following commands in the DSW Terminal to download the sample dataset.
mkdir /mnt/workspace/mixtral-datasets/
cd /mnt/workspace/mixtral-datasets
wget https://atp-modelzoo-wlcb-pai.oss-cn-wulanchabu.aliyuncs.com/release/models/pai-megatron-patch/mistral-datasets/alpaca_zh-mistral-train.json
wget https://atp-modelzoo-wlcb-pai.oss-cn-wulanchabu.aliyuncs.com/release/models/pai-megatron-patch/mistral-datasets/alpaca_zh-mistral-valid.json
mkdir -p /mnt/workspace/mixtral-datasets/wudao
cd /mnt/workspace/mixtral-datasets/wudao
wget https://atp-modelzoo-wlcb-pai.oss-cn-wulanchabu.aliyuncs.com/release/models/pai-megatron-patch/mistral-datasets/wudao_mistralbpe_content_document.idx
wget https://atp-modelzoo-wlcb-pai.oss-cn-wulanchabu.aliyuncs.com/release/models/pai-megatron-patch/mistral-datasets/wudao_mistralbpe_content_document.binProcess the data on your own
Download the open source WuDaoCorpora2.0 dataset to the
/mnt/workspace/mixtral-datasetsworking directory. In this example, the decompressed folder is named wudao_200g.PAI provides a sample dataset. You can run the following commands in the DSW Terminal to download and decompress the dataset.
mkdir /mnt/workspace/mixtral-datasets cd /mnt/workspace/mixtral-datasets wget https://atp-modelzoo.oss-cn-hangzhou.aliyuncs.com/release/datasets/WuDaoCorpus2.0_base_sample.tgz tar zxvf WuDaoCorpus2.0_base_sample.tgz mv WuDaoCorpus2.0_base_sample wudao_200gRun the following commands in the Terminal to clean the Wudao dataset and convert the file format. This process generates a merged file named merged_wudao_cleaned.json.
#! /bin/bash set -ex # Set the path where the raw data is stored. data_dir=/mnt/workspace/mistral-datasets/wudao_200g # Start the data cleaning process. dataset_dir=$(dirname $data_dir) mkdir -p ${dataset_dir}/cleaned_wudao_dataset cd ${dataset_dir}/cleaned_wudao_dataset wget https://atp-modelzoo-wlcb-pai.oss-cn-wulanchabu.aliyuncs.com/release/models/pai-megatron-patch/llama2-codes/preprocess_wudao2.py # Unlike the previous section, the key parameter is added and set to text. python preprocess_wudao2.py -i ${data_dir} -o ${dataset_dir}/cleaned_wudao_dataset -k text -p 32 # Merge the cleaned data. mkdir ${dataset_dir}/wudao cd ${dataset_dir}/wudao find ${dataset_dir}/cleaned_wudao_dataset -name "*.json" -exec cat {} + > ${dataset_dir}/wudao/merged_wudao_cleaned.json rm -rf ${dataset_dir}/cleaned_wudao_datasetAfter the commands are run, the file structure of the
mixtral-datasetsdirectory is as follows. A `wudao` folder is created.mixtral-datasets ├── wudao_200g └── wudao └── merged_wudao_cleaned.jsonRun the following commands in the Terminal to split the data in the merged_wudao_cleaned.json file into groups and compress them. This facilitates multithreading in subsequent steps.
apt-get update apt-get install zstd # The number of chunks is set to 10. You can set a larger number if data processing is slow. NUM_PIECE=10 # Process the merged_wudao_cleaned.json file. mkdir -p ${dataset_dir}/cleaned_zst/ # Query the total data length and split the data. NUM=$(sed -n '$=' ${dataset_dir}/wudao/merged_wudao_cleaned.json) echo "total line of dataset is $NUM, data will be split into $NUM_PIECE pieces for processing" NUM=`expr $NUM / $NUM_PIECE` echo "each group is processing $NUM sample" split_dir=${dataset_dir}/split mkdir $split_dir split -l $NUM --numeric-suffixes --additional-suffix=.jsonl ${dataset_dir}/wudao/merged_wudao_cleaned.json $split_dir/ # Compress the data. o_path=${dataset_dir}/cleaned_zst/ mkdir -p $o_path files=$(ls $split_dir/*.jsonl) for filename in $files do f=$(basename $filename) zstd -z $filename -o $o_path/$f.zst & done rm -rf $split_dir rm ${dataset_dir}/wudao/merged_wudao_cleaned.jsonAfter the commands are run, the file structure of the
mixtral-datasetsdirectory is as follows. Acleaned_zstfolder is created, which contains 10 compressed files.mixtral-datasets ├── wudao_200g ├── wudao └── cleaned_zst ├── 00.jsonl.zst │ ... └── 09.jsonl.zstCreate a pre-training dataset in MMAP format.
MMAP is a data format in which tokenization is performed in advance. This format reduces the data read latency during training and fine-tuning, which is especially useful for large-scale datasets. The procedure is as follows:
Run the following command in the DSW Terminal to copy the source code of the Megatron model training tool, PAI-Megatron-Patch, to the
/mnt/workspace/DSW working directory.cd /mnt/workspace/ # Method 1: Obtain the training code from the open source website. git clone --recurse-submodules https://github.com/alibaba/Pai-Megatron-Patch.git # Method 2: Obtain the training code using wget. You need to run tar zxvf Pai-Megatron-Patch.tgz to decompress it. wget https://atp-modelzoo.oss-cn-hangzhou.aliyuncs.com/release/models/Pai-Megatron-Patch.tgzRun the following command in the Terminal to convert the data to MMAP format.
After the command is run,
.binand.idxfiles are generated in the/mnt/workspace/mixtral-datasets/wudaodirectory.# Set the dataset path and working directory. export dataset_dir=/mnt/workspace/mixtral-datasets export WORK_DIR=/mnt/workspace # Generate MMAP-formatted pre-training datasets for the training and validation sets respectively. cd ${WORK_DIR}/Pai-Megatron-Patch/toolkits/pretrain_data_preprocessing bash run_make_pretraining_dataset.sh \ ../../Megatron-LM-23.04 \ ${WORK_DIR}/Pai-Megatron-Patch/ \ ${dataset_dir}/cleaned_zst/ \ mistralbpe \ ${dataset_dir}/wudao/ \ ${WORK_DIR}/mixtral-ckpts/Mixtral-8x7B-v0.1 rm -rf ${dataset_dir}/cleaned_zstThe following table describes the six startup parameters for running run_make_pretraining_dataset.sh.
Parameter
Description
MEGATRON_PATH=$1
The code path of the open source Megatron.
MEGATRON_PATCH_PATH=$2
The code path of Megatron Patch.
input_data_dir=$3
The folder path of the packaged Wudao dataset.
tokenizer=$4
Specifies the tokenizer type as mistralbpe.
output_data_dir=$5
The path to save the output
.binand.idxfiles.load_dir=$6
The path of the generated tokenizer_config.json file.
After the script is run, the file structure of the
mixtral-datasetsdirectory is as follows.mixtral-datasets ├── wudao_200g └── wudao ├── wudao_mixtralbpe_content_document.bin └── wudao_mixtralbpe_content_document.idx
Step 3: Megatron training
Follow this procedure to perform Megatron training.
Model format conversion
Convert the Hugging Face model file to the Megatron format.
Download the converted Megatron model
To help you try this example, PAI provides a model in the converted format. You can run the following command in the Terminal to download the model.
cd /mnt/workspace/
mkdir mixtral-ckpts
cd mixtral-ckpts
wget https://atp-modelzoo-wlcb-pai.oss-cn-wulanchabu.aliyuncs.com/release/models/pai-megatron-patch/mistral-ckpts/Mixtral-8x7B-v0.1-to-mcore-tp4-ep4.tar.zst
tar -zxf Mixtral-8x7B-v0.1-to-mcore-tp4-ep4.tar.zstConvert the Hugging Face model to Megatron format on your own
Run the following command in the Terminal to use the PAI-provided model conversion tool to convert the Hugging Face model file to the Megatron format:
# Convert the model.
cd /mnt/workspace/Pai-Megatron-Patch/toolkits/model_checkpoints_convertor/mixtral
sh hf2mcore_convertor.sh \
../../../Megatron-LM-231221 \
/mnt/workspace/mixtral-ckpts/Mixtral-8x7B-v0.1 \
/mnt/workspace/mixtral-ckpts/Mixtral-8x7B-v0.1-to-mcore-tp4-ep4 \
4 \
1 \
4 \
16 \
mixtral-8x7b \
falseThe following table describes the parameters for running hf2mcore_convertor.sh.
Parameter | Description |
MEGATRON_PATH=$1 | The code path of the open source Megatron. |
SOURCE_CKPT_PATH=$2 | The path of the Hugging Face model file. |
TARGET_CKPT_PATH=$3 | The path to save the model after it is converted to the Megatron format. |
TP=$4 | The number of tensor slices, which must be the same as in training. The number of slices varies with the number of parameters and must be modified accordingly during model conversion: Mixtral-8x7B-v0.1: 2 or 4. |
PP=$5 | The number of pipeline slices, which must be the same as in training. The number of slices varies with the number of parameters and must be modified accordingly during model conversion: Mixtral-8x7B-v0.1: 1. |
MN=$6 | Model name: mixtral-8x7b. |
mg2hf=$8 | Specifies whether to convert from Megatron format to Hugging Face format. |
Pre-trained model
You can pre-train the model in a single-node DSW environment or submit a multi-GPU distributed training task in a DLC environment. The training process takes about 2 hours. After the task is successfully completed, the model files are saved to the /mnt/workspace/output_megatron_mixtral/ directory.
DSW standalone pre-trained model
The following code shows an example for the Mixtral-8x7B-v0.1 model. Run the code in the Terminal.
export WORK_DIR=/mnt/workspace
cd ${WORK_DIR}/Pai-Megatron-Patch/examples/mixtral
sh run_pretrain_megatron_mixtral.sh \
dsw \
${WORK_DIR}/Pai-Megatron-Patch \
7B \
1 \
8 \
1e-5 \
1e-6 \
2048 \
2048 \
0 \
bf16 \
4 \
1 \
sel \
true \
false \
false \
false \
100000 \
${WORK_DIR}/mixtral-datasets/wudao/wudao_mistralbpe_content_document \
${WORK_DIR}/mixtral-ckpts/Mixtral-8x7B-v0.1-to-mcore-tp4-ep4 \
100000000 \
10000 \
${WORK_DIR}/output_megatron_mixtral/ The following table describes the parameters for running run_pretrain_megatron_mixtral.sh.
Parameter | Description |
ENV=$1 | Configure the runtime environment:
|
MEGATRON_PATH=$2 | The code path of the open source Megatron. |
MODEL_SIZE=$3 | The parameter size of the model structure: 7B. |
BATCH_SIZE=$4 | The number of samples for one iteration per GPU card: 4 or 8. |
GLOBAL_BATCH_SIZE=$5 | The global batch size. |
LR=$6 | The learning rate: 1e-5 or 5e-5. |
MIN_LR=$7 | The minimum learning rate: 1e-6 or 5e-6. |
SEQ_LEN=$8 | The sequence length. |
PAD_LEN=${9} | The padding length. |
EXTRA_VOCAB_SIZE=${10} | The vocabulary expansion size: Mixtral-8x7B-v0.1: 0. |
PR=${11} | The training precision: fp16 or bf16. |
TP=${12} | The degree of model parallelism. |
PP=${13} | The degree of pipeline parallelism. |
AC=${14} | The activation checkpointing mode:
|
DO=${15} | Specifies whether to use the Megatron version of the Zero-1 optimizer to reduce VRAM usage:
|
FL=${16} | Specifies whether to enable Flash Attention:
|
SP=${17} | Specifies whether to use sequence parallelism:
|
TE=${18} | Specifies whether to enable Transformer-engine acceleration. This requires H800 GPUs. |
SAVE_INTERVAL=${19} | The interval for saving checkpoint files. |
DATASET_PATH=${20} | The training dataset path. |
PRETRAIN_CHECKPOINT_PATH=${21} | The pre-trained model path. |
TRAIN_TOKENS=${22} | The training tokens. |
WARMUP_TOKENS=${23} | The number of warm-up tokens. |
OUTPUT_BASEPATH=${24} | The path to save the output model files from training. |
DLC distributed pre-trained model
After you develop and test on a single node, you can configure a multi-GPU distributed task in the DLC environment. The procedure is as follows:
Go to the Create Job page.
Log on to the PAI console. select the target region and workspace at the top of the page, and then click Deep Learning Containers (DLC).
On the Deep Learning Containers (DLC) page, click Create Job.
On the Create Job page, configure the following key parameters. Use the default values for other parameters. For more information, see Create a training task.
Parameter
Description
Basic Information
Job Name
Enter a custom job name. In this example, set it to test_mixtral_dlc.
Environment Information
Node Image
Select Registry Address and enter
dsw-registry.cn-wulanchabu.cr.aliyuncs.com/pai/pytorch-training:23.12-gpu-py310-cu123-ubuntu22.04-megatron-patch-llmin the text box.Dataset
Click Custom Dataset and configure the following parameters:
Custom Dataset: Select the created NAS dataset.
Mount Path: Set it to
/mnt/workspace/.
Start Command
Configure the following command. The start parameters for the run_pretrain_megatron_mixtral.sh script are the same as those for pre-training the model on a single DSW node.
export WORK_DIR=/mnt/workspace cd ${WORK_DIR}/Pai-Megatron-Patch/examples/mixtral sh run_pretrain_megatron_mixtral.sh \ dlc \ ${WORK_DIR}/PAI-Megatron-Patch \ 7B \ 1 \ 8 \ 1e-5 \ 1e-6 \ 2048 \ 2048 \ 0 \ bf16 \ 1 \ 1 \ sel \ true \ false \ false \ false \ 100000 \ ${WORK_DIR}/mixtral-datasets/wudao/wudao_mistralbpe_content_document \ ${WORK_DIR}/mixtral-ckpts/Mixtral-8x7B-v0.1-to-mcore-tp4-ep4 \ 100000000 \ 10000 \ ${WORK_DIR}/output_megatron_mixtral/Resource Information
Resource Type
Select Lingjun AI Computing.
Resource Source
Select Resource Quota.
Resource Quota
Select the resource quota for the created Lingjun resources.
Framework
Select PyTorch.
Job Resources
On the Worker node configuration tab, configure the following parameters:
Number of Nodes: 2. For multi-node training, set Number of Nodes to the required number of machines.
GPU (Cards): 8
CPU (Cores): 90
NoteThe number of CPU cores cannot exceed 96.
Memory (GiB): 1024
Shared Memory (GiB): 1024
Click OK. You are redirected to the Deep Learning Containers (DLC) page. When the Status changes to Succeeded, the training task is complete.
Supervised fine-tuning of the model
You can fine-tune the model in a single-node DSW environment or submit a multi-GPU distributed task in a DLC environment. The fine-tuning process takes about 2 hours. After the task is successfully completed, the model files are saved to the /mnt/workspace/output_megatron_mixtral/ directory.
Before you fine-tune the model, go to the Step 2: Prepare pre-training data section. In the "Use the small sample dataset provided by PAI" subsection, download the JSON file as instructed.
Fine-tune the model.
Fine-tune the model on a single DSW node
The following code shows an example for the Mixtral-8x7B-v0.1 model. Run the code in the Terminal.
export WORK_DIR=/mnt/workspace cd ${WORK_DIR}/Pai-Megatron-Patch/examples/mixtral sh run_finetune_megatron_mixtral.sh \ dsw \ ${WORK_DIR}/Pai-Megatron-Patch \ 7B \ 1 \ 1e-5 \ 1e-6 \ 2048 \ 2048 \ 0 \ bf16 \ 1 \ 1 \ sel \ true \ false \ false \ false \ ${WORK_DIR}/mixtral-datasets/alpaca_zh-mistral-train.json \ ${WORK_DIR}/mixtral-datasets/alpaca_zh-mistral-valid.json \ ${WORK_DIR}/mixtral-ckpts/Mixtral-8x7B-v0.1-to-mcore-tp4-ep4 \ 2 \ ${WORK_DIR}/output_megatron_mixtral/The following table describes the parameters for running run_finetune_megatron_mixtral.sh.
Parameter
Description
ENV=$1
The running environment:
dlc
dsw
MEGATRON_PATH=$2
The code path of the open source Megatron.
MODEL_SIZE=$3
The parameter size of the model structure: 7B, 14B, or 72B.
BATCH_SIZE=$4
The number of samples for one iteration per GPU card: 1, 2, 4, or 8.
LR=$6
The learning rate: 1e-5 or 5e-5.
MIN_LR=$7
The minimum learning rate: 1e-6 or 5e-6.
SEQ_LEN=$8
The sequence length.
PAD_LEN=$9
The padding sequence length.
EXTRA_VOCAB_SIZE=${10}
The vocabulary expansion size:
Mixtral-8x7B-v0.1: 0.
PR=${11}
The training precision: fp16 or bf16.
TP=${12}
The degree of model parallelism.
PP=${13}
The degree of pipeline parallelism.
AC=${14}
The activation checkpointing mode: full or sel.
DO=${15}
Specifies whether to use the Megatron version of the Zero-1 optimizer to reduce VRAM usage:
true
false
FL=${16}
Specifies whether to enable Flash Attention:
true
false
SP=${17}
Specifies whether to use sequence parallelism:
true
false
TE=${18}
Specifies whether to enable Transformer-engine acceleration. This requires H800 GPUs.
DATASET_PATH=${20}
The training dataset path.
VALID_DATASET_PATH=${21}
The validation set path.
PRETRAIN_CHECKPOINT_PATH=${22}
The pre-trained model path.
EPOCH=${23}
The number of training epochs.
OUTPUT_BASEPATH=${25}
The path to save the output model files from training.
Fine-tune the model with distributed DLC
After you develop and test in a single-node DSW environment, you can configure a multi-GPU distributed task in the DLC environment. When you submit the DLC training task, configure the Start Command as follows. For information about other parameter settings, see the Step 2: Pre-train the model section.
export WORK_DIR=/mnt/workspace cd ${WORK_DIR}/Pai-Megatron-Patch/examples/mixtral sh run_finetune_megatron_mixtral.sh \ dlc \ ${WORK_DIR}/Pai-Megatron-Patch \ 7B \ 1 \ 1e-5 \ 1e-6 \ 2048 \ 2048 \ 0 \ bf16 \ 1 \ 1 \ sel \ true \ false \ false \ false \ ${WORK_DIR}/mixtral-datasets/alpaca_zh-mistral-train.json \ ${WORK_DIR}/mixtral-datasets/alpaca_zh-mistral-valid.json \ ${WORK_DIR}/mixtral-ckpts/Mixtral-8x7B-v0.1-to-mcore-tp4-ep4 \ 2 \ ${WORK_DIR}/output_megatron_mixtral/The parameters for running run_finetune_megatron_mixtral.sh are the same as those for fine-tuning the model on a single DSW node.
Step 4: Model format conversion
You can convert the trained Megatron-format model to the Hugging Face format. You can then use the converted Hugging Face model for online service deployment. The procedure is as follows:
Run the following command in the Terminal to convert the trained Megatron-format model to the Hugging Face format.
export WORK_DIR=/mnt/workspace cd /mnt/workspace/Pai-Megatron-Patch/toolkits/model_checkpoints_convertor/mixtral sh hf2mcore_convertor.sh \ ../../../Megatron-LM-231221 \ ${WORK_DIR}/output_megatron_mixtral/checkpoint/${path}/iter_******* \ /mnt/workspace/mixtral-ckpts/mixtral-8x7b-mg-to-hf-tp1-pp1/ \ 1 \ 1 \ mixtral-8x7b \ trueThe following table describes the parameters for running the hf2mcore_convertor.sh script.
Parameter
Description
MEGATRON_PATH=$1
The code path of the open source Megatron.
SOURCE_CKPT_PATH=$2
The path of the Megatron-format model obtained from training, specific to
iter_*. For example:${WORK_DIR}/output_megatron_mixtral/checkpoint/dsw-pretrain-megatron-mixtral-7B-lr-1e-5-bs-1-seqlen-2048-pr-bf16-tp-1-pp-1-ac-sel-do-true-sp-false-tt--wt-/iter_*******.ImportantReplace this with your own model path.
If you convert a pre-trained model, you must delete all distrib_optim.pt files from the model path.
TARGET_CKPT_PATH=$3
The path to save the model after it is converted to the Hugging Face format.
TP=$4
The number of tensor slices, which must be the same as in training.
PP=$5
The number of pipeline slices, which must be the same as in training.
MN=$6
Model name: mixtral-8x7b.
mg2hf=$8
Specifies whether to convert from Megatron format to Hugging Face format.
Copy the
.jsonand.pyfiles from the open source Hugging Face model directory/mnt/workspace/mixtral-ckpts/Mixtral-8x7B-v0.1to the/mnt/workspace/mixtral-ckpts/Mixtral-8x7B-v0.1-to-mcore-tp4-ep4directory. This step ensures that the model can be used as expected.ImportantDo not copy the pytorch_model.bin.index.json file.
Step 5: Deploy and call the model service
After you perform offline inference and evaluate the model's performance, you can deploy the converted Hugging Face model as an online service. You can then call the service in a production environment for inference. The procedure is as follows:
Deploy the model service
Log on to the PAI console. Select a region on the top of the page. Then, select the desired workspace and click Elastic Algorithm Service (EAS).
Click Deploy Service. In the Custom Model Deployment section, click Custom Deployment.
On the Custom Deployment page, configure the following key parameters. Use the default values for the other parameters.
Parameter
Description
Basic Information
Service Name
Enter a custom model service name that is unique within the region. In this example, set it to test_mixtral.
Environment Information
Deployment Method
In this example, select Image Deployment and select Enable Web App.
Image Configuration
Select Registry Address and enter the image address
pai-image-manage-registry.cn-wulanchabu.cr.aliyuncs.com/pai/llm-inference:vllm-0.2.1-v4in the text box.Direct Mount
Select NAS as the mount type. Click General-purpose NAS and configure the following parameters:
Select File System: Select the NAS file system used to create the dataset.
File System Mount Target: Select the mount target used to create the dataset.
File System Path: The path of the converted Hugging Face model stored in NAS. In this example, set it to
/mixtral-ckpts/mixtral-8x7b-mg-to-hf-tp1-pp1.Mount Path: The path after mounting. In this example, set it to
/mixtral-8x7b.
Run Command
Set it to
nohup python -m fastchat.serve.controller > tmp1.log 2>&1 & python -m fastchat.serve.gradio_web_server_pai --model-list-mode reload > tmp2.log 2>&1 & python -m fastchat.serve.vllm_worker --model-path /mixtral-8x7b --tensor-parallel-size 1 --trust-remote-code.Where:
--model-path: Must be the same as the mount path in the model configuration.
--tensor-parallel-size: The number of model tensor slices. This needs to be adjusted based on the number of GPUs. For a 7B model on a single GPU, you can set this to 1. For a 72B model, you need to set it to 2.
Port Number
Set it to 7860.
Resource Information
Resource Type
Select Resource Quota.
Resource Quota
Select the resource quota for the created Lingjun resources.
Number of Instances
Configure this based on the model and selected resources. For a 7B model, set Number of Instances to 1.
Deployment Resources
For a 7B model, configure the resources used by each instance as follows:
CPU (Cores): 16.
Memory (GB): 64.
GPU (Cards): 1.
Service Access
Virtual Private Cloud (VPC)
After you configure the NAS mount target, the system automatically matches the VPC, vSwitch, and security group with the preset NAS file system.
vSwitch
Security Group Name
Click Deploy.
When the Service Status changes to Running, the service is deployed.
Call the service
After the service is deployed, you can call it for inference. The procedure is as follows:
Log on to the PAI console. Select a region on the top of the page. Then, select the desired workspace and click Elastic Algorithm Service (EAS).
In the service list, click the name of the target service. In the upper-right corner of the page, click View Web App.
On the WebUI page, perform model inference.