Model Distillation (ModelDistill)

How it works

ModelDistill uses black-box distillation, a form of generative data augmentation: the teacher model generates high-quality labeled data, and that data is then used to fine-tune the student model through supervised fine-tuning (SFT).

The workflow has two stages:

1. Distillation data generation: select a source dataset, expand it through data synthesis (such as instruction expansion and refinement), and then use the teacher model to produce high-quality distillation data. The teacher model can come from PAI Token Service or PAI-EAS.

2. Student model training: train the student model on the distillation dataset through SFT. Three training methods are supported: LoRA, QLoRA, and full-parameter fine-tuning.

Use cases

When to use model distillation:

• Edge deployment: compress a large model into a lightweight model for resource-constrained environments such as mobile devices and IoT.

• Cost reduction: replace an expensive inference model with a smaller distilled model when production inference costs are too high.

• Latency optimization: for latency-sensitive applications, reduce inference latency and GPU consumption while maintaining accuracy.

• Domain knowledge transfer: transfer domain-specific knowledge (such as medical or legal expertise) from a large model to a smaller, more cost-effective model for targeted tasks.

When model distillation may not be suitable:

• 100% performance fidelity required: distillation always involves some performance loss. If any degradation is unacceptable, distillation is not the right approach.

• Trivially simple tasks: for straightforward tasks such as classification or text matching, directly fine-tuning a small model may be more cost-effective than distillation.

• No high-quality seed data: distillation quality depends heavily on the seed data. If the seed data doesn't represent your actual business scenarios well, the results may be poor.

Template types

ModelDistill provides two types of templates: General and Scenario-Based.

• General Distillation Template: follows the standard three-step pipeline (basic configuration → distillation data generation → student model training) with full parameter flexibility. See General distillation.

• Scenario-Based Distillation Templates: pre-configured pipelines and algorithm parameters optimized for specific scenarios such as chain-of-thought reasoning, multimodal tasks, and Agent tool calling. Select a scenario to reuse best-practice configurations and quickly produce SFT / DPO / multimodal training data.

  Note

  Some templates support only distillation data generation. Others support both data generation and student model training, with training as an optional step — you can run data generation alone.

  Template	Use case	Description
  Instruction distillation	Instruction-following capability	Transfers the teacher model's instruction-following ability to the student model. For details, see Instruction distillation.
  Chain-of-thought reasoning distillation	CoT reasoning capability	Transfers the teacher model's chain-of-thought reasoning ability to the student model. For details, see Chain-of-thought reasoning distillation.
  Agent tool-call distillation	Tool/function calling	Transfers the teacher model's tool and function calling ability to the student model. For details, see Agent tool call distillation.
  Text-to-video distillation	Text-to-video generation	Transfers the teacher model's text-to-video generation ability to the student model. For details, see Text-to-video distillation.
  Coding Agent distillation	Code Agent capability	A coding Agent modifies code in a remote sandbox and replays test verification to produce a high-quality SFT dataset. The student model learns to locate issues, modify code, and verify results in real codebases. For details, see Coding Agent Distillation.

Template selection

First, evaluate whether a scenario-specific template matches your use case. If it does, use that template. If none of them match, use the general-purpose template for custom configuration.

How do I choose a teacher model and a student model?

• Teacher model: choose a large model that has already been validated in your business scenario and performs well. In general, a larger model has more knowledge and raises the ceiling for distillation quality.

• Student model: choose a smaller model from the same model family as the teacher. For example, distill from Qwen-30B to Qwen-8B. This ensures architecture compatibility and maximizes knowledge transfer efficiency.

How long does a distillation task take?

Duration depends on dataset size, model scale, training method, and compute resources. For small datasets (thousands of entries) and mid-size models (7B–14B), tasks typically complete in tens of minutes to a few hours.

Can I resume a task that was interrupted due to timeout?

No. If a task is interrupted because it reached the maximum runtime, the task fails and cannot be resumed. Review the task logs to determine whether the issue was insufficient resources (slow execution) or a timeout setting that was too short. Then increase the maximum runtime or use higher-spec resources and resubmit the task.

The task failed with CUDA out of memory. What do I do?

This is a GPU out-of-memory error. Follow these steps to resolve it:

1. Reduce the batch size: in the Student model training configuration, find Hyperparameter and halve the per_device_train_batch_size value (for example, from 2 to 1), then retry.

2. Switch training method: if reducing the batch size doesn't help, go back to the configuration page and switch the training method from LoRA or full-parameter fine-tuning to QLoRA. QLoRA significantly reduces GPU memory usage.

3. Upgrade the GPU spec: if neither of the above works, the model and data scale require more hardware. In Compute Resource, select a GPU instance with more GPU memory.