General distillation-Platform For AI(PAI)-阿里云帮助中心

Classic three-step distillation: basic configuration → distillation data generation → student model training. Choose any teacher and student models and datasets, and configure distillation and training parameters to fit most general-purpose scenarios.

Template overview

The general distillation template is the base template in ModelDistill. It uses black-box distillation — generative data augmentation followed by SFT training.

Unlike scenario-specific templates, general distillation follows the standard three-step pipeline without preset operator orchestration for a specific task type. You control every stage. Scenario-specific templates already cover common task types such as instruction distillation, chain-of-thought distillation, multimodal distillation, and Agent tool-call distillation. First, evaluate whether a scenario-specific template fits your task. Use general distillation when none of them match.

General distillation follows a fixed three-stage pipeline. Each stage is configurable in the creation wizard. Distillation data and training artifacts are saved separately for reuse.

Basic configuration: Define the task name, output model name, and path.
Distillation data generation: Call the teacher model to generate high-quality responses from the source data and build a distillation SFT dataset.
Student model training: Fine-tune the student model on the distillation data and produce a deployable distilled model.

Quick start

The following walkthrough uses a PAI public dataset and default configurations to demonstrate a complete general distillation task.

Log on to the Model Application > Model Distillation (ModelDistill) section in PAI console.
Click Create task to go to the Distillation Template tab, or go to the Distillation Template tab directly.
Select General Distillation, click Use Template, and go to the Create Distillation Task page. Configure the key parameters described below and keep the default values for other settings.
Complete the Basic Configuration:
- Task name and Model Name: pre-filled by the system.
- Model output path: specify an OSS path that you have access to, such as oss://mybucket.oss-cn-hangzhou-internal.aliyuncs.com/model-distill/model.
  Note
  Create a separate output directory for each distillation task to prevent model files from being overwritten. The OSS bucket must be in the same region as the PAI service.
Click Next to go to the Build distillation data step:
- Teacher model: Teacher Model Source — select Token Service and choose a model from the drop-down list, such as qwen3.8-max.
- Dataset: select Public Dataset and choose Chinese-medical-dialogue-data from the drop-down list. You can also select Custom Dataset to use your own data.
- Confirm distillation data: specify an OSS path that you have access to as the Distillation output path, such as oss://mybucket.oss-cn-hangzhou-internal.aliyuncs.com/model-distill/dist-data.
- Compute Resource: set Resource Type to General Computing and Source to Public Resources. Then select an appropriate Instance Type in Job Resource.
Click Next to go to the Student model training step:
- Student Model: select Public Model and choose a smaller model from the drop-down list, such as Qwen3-4B.
- Training Mode: select LoRA.
- Compute Resource: set Resource Type to General Computing and Source to Public Resources. Then select an appropriate Instance Type in Job Resource.
Click OK, and then click OK in the billing prompt. The page redirects to the task details page where you can track the task status. The estimated duration is 20–40 minutes for a small dataset (100–1,000 entries).

Configuration reference

Distillation data generation

Teacher model: supports the following sources:
- PAI Token Service.
- PAI-EAS Service: only EAS services deployed from Model Gallery models — either directly or after fine-tuning — are supported.
Dataset: supports both public distillation datasets and custom datasets. Data must be in JSON format and include the instruction field (question column). The output field (output column) isn't required.
```
[
    {
        "instruction": "What is the capital of France?"
    },
    {
        "instruction": "Explain what artificial intelligence is."
    }
]
```
Confirm distillation data: choose how the system handles the generated distillation data after data generation is complete.
- Auto-confirm (recommended): the system automatically validates and approves the data, then starts student model training. Suitable for public datasets or when you're confident in data quality.
- Manually confirm: requires manual review of data quality before training begins. Suitable for high data-quality requirements or when using a custom dataset for the first time. The procedure is as follows:
  1. After distillation data generation finishes, the task status changes to Waiting for distillation data confirmation.
  2. Click Confirm distillation data in the Actions column. A confirmation dialog box appears.
  3. In the dialog box, click Click to view to view the distillation dataset details.
    - If the data quality meets your expectations, click Confirm to proceed with student model training.
    - If the data doesn't meet your expectations, click Cancel to stop the task at the current stage.

Student model training

The student model can be a Model Gallery public model or a custom model (currently limited to LLMs fine-tuned in Model Gallery). Make sure the student model has fewer parameters than the teacher model.

Training mode

Select the SFT method for the student model.

LoRA (recommended): a parameter-efficient fine-tuning method that maintains good performance while significantly reducing GPU memory requirements.
QLoRA: a quantized variant of LoRA that further reduces GPU memory usage. Suitable for training in more resource-constrained environments.
Full-parameter fine-tuning: updates all model parameters. Produces the best results in theory, but requires substantial compute resources and time at high cost.

Validation set

Select the validation dataset used to evaluate model performance during training.

Don't configure: no validation is performed during training. Suitable for quick experiments or small datasets.
Auto-split distillation data (recommended): the system automatically splits the generated distillation dataset into training and validation sets. The default validation set ratio is 5%.
Add validation dataset: specify an independent OSS dataset as the validation set. Suitable when you have a standard validation dataset.

Hyperparameter

These hyperparameters control the SFT training process for the student model.

Parameter	Description	Tuning tips
`learning_rate`	Learning rate. Controls the step size for model parameter updates. Default: 5e-5.	• If training loss decreases slowly, try increasing the learning rate. • If loss oscillates sharply or fails to converge, reduce the learning rate.
`num_train_epochs`	Number of training epochs — how many times the entire dataset is repeated during training. Default: 1.	• For large datasets, 1–3 epochs are usually sufficient. • Too many epochs may cause overfitting.
`per_device_train_batch_size`	Number of samples processed per GPU device in a single training step. Default: 1.	• Increase this value when GPU memory allows to speed up training and potentially stabilize the process. • If you encounter out-of-memory (OOM) errors, reduce this value first.
`seq_length`	Maximum sequence length (in tokens) fed into the student model. Default: 128.	• Set this based on your data characteristics and use case. Longer texts require a larger `seq_length`. • Increasing this value significantly raises GPU memory consumption.

Best practices

Cost optimization

Resource type:
- Tests and small tasks: use public resources for on-demand, pay-as-you-go access.
- Production and high-priority tasks: use resource quotas for stable resources and predictable costs.
- Cost-sensitive and fault-tolerant tasks: try preemptible resources for lower-cost compute, at the risk of task interruption.
Start with a small dataset: before running large-scale distillation, run the full pipeline on a small subset (100–1,000 entries) to verify your configuration. This avoids wasting compute resources on misconfigured jobs.
Optimize your dataset: smaller, higher-quality seed datasets reduce teacher model inference costs and student model training time.
Use QLoRA training: QLoRA trains on a quantized model, which significantly lowers GPU memory requirements and lets you use lower-spec (and cheaper) GPUs.
Set a reasonable timeout: configure a maximum runtime to prevent tasks from running indefinitely due to unexpected issues and incurring unnecessary charges.

Performance tuning

Dataset quality is the key factor: the teacher model sets the upper bound for distillation quality, but the distillation dataset sets the lower bound. Make sure the source dataset covers your business scenarios, has clear instructions, and contains diverse content.
Hyperparameter tuning:
- If the student model loss doesn't decrease or diverges, reduce the learning rate (learning_rate) first.
- If you encounter OOM errors, reduce the batch size (per_device_train_batch_size) or sequence length (seq_length) first.
Monitor GPU utilization: check GPU utilization in task monitoring. Low utilization means your GPUs aren't fully used. Try increasing per_device_train_batch_size to improve throughput.

Evaluation

After distillation is complete, evaluate the student model.

Deploy the model: on the task details page, click Deploy in the upper-right corner to deploy the distilled student model as an online inference service.
Comparative evaluation: use the PAI model evaluation feature to compare teacher and student model performance metrics (such as accuracy and BLEU) on the same evaluation dataset and quantify the performance gap.
Production validation: before going live, run a canary test with a small traffic percentage. Compare inference speed, resource usage, and business metrics between the distilled model and the original model in real business scenarios.