Online Policy Distillation Training Configuration

Model Studio online policy distillation training configuration: submission methods, teacher model configuration, hyperparameters, tuning pitfalls, task management, and FAQ

OPD requires configuring a teacher model and hyperparameters when submitting a training job. This document covers three submission methods, teacher model configuration, hyperparameter values, tuning decisions, job management, and common troubleshooting.

Submit a Training Job

Three submission methods, with increasing flexibility. Configuration priority:__init__()(SDK init function) Default Value < YAML < run() runtime parameters (the latter overrides the former). Pass in teacher_model to trigger distillation.

Recommend: one step done (run)

Pass AgenticRL().run(...) to complete function registration, data upload, and job submission in one step; pass an additional teacher_model to trigger distillation.

from dashscope.finetune.agentic_rl import AgenticRL          # OPD 训练客户端
from dashscope.finetune.reinforcement import (               # 组件与数据类型
    RewardFunctionComponent, FunctionComponentModel,
    FunctionComponentRuntime, TrainingDataset,
    ValidationDataset, DataSourceType,
)

client = AgenticRL()                                        # 初始化训练客户端
result = await client.run(                                  # 异步提交：注册函数→上传数据→提交任务
    job_name="my-opd-job",
    model="qwen3.5-9b",                        # 学生模型（训练对象，被蒸馏的学生）
    teacher_model="qwen3.5-397b-a17b",         # 教师模型，OPD 唯一新增参数；传入即触发蒸馏
    training_datasets=[TrainingDataset(        # 训练集：FILE_ID 方式由 SDK 自动上传并换 file_id
        data_source_type=DataSourceType.FILE_ID,
        file_name="./data/train.jsonl")],
    validation_datasets=[ValidationDataset(    # 验证集：用于训练中评估，指标驱动调参
        data_source_type=DataSourceType.FILE_ID,
        file_name="./data/validation.jsonl")],
    functions=[
        # 不传 Rollout 组件 → Model OPD（原生 Rollout）；传 Rollout 组件 → Agentic OPD（自定义 Rollout，支持多轮/自定义交互）
        RewardFunctionComponent(               # 仅传 Reward 组件：在原生 Rollout + 教师蒸馏基础上叠加任务奖励
            name="reward-1", weight=1.0, timeout=120,   # weight=奖励权重，timeout=单次打分超时秒数
            fcmodel=FunctionComponentModel(
                zipdir=".",  # 工作区根目录（含 functions/），SDK 打包此目录上传至 FC
                classpath="functions.reward.reward.MyRewardProcessor"),  # 入口类全路径（示例类名，实际由您自定义）
            runtime=FunctionComponentRuntime(
                cpu=2, memory_size=4096, disk_size=512,            # 函数实例 CPU/内存/磁盘
                concurrency=20, capacity=10, min_capacity=10, max_capacity=30)),  # 并发与弹性容量
    ],
    # 纯蒸馏场景：改传 functions=None（而非空列表 []），不注册任何函数，reward 派生指标不上报
    hyper_parameters={                         # 训练超参，完整字段见下方 §训练超参数
        "algorithm": "gspo", "learning_rate": 1e-6, "lr_scheduler_type": "cosine",
        "n_epochs": 1, "n_rollouts": 8, "batch_size": 128,
    },
    resources={"charge_type": "mtu_postpaid", "mtu_spec_code": "MTU4", "mtu_capacity": 24},  # 计费资源：后付费/规格/容量
)

functions The Parameter determines the run mode (pure distillation / Model OPD / Agentic OPD), and teacher_model is unrelated ——teacher_model only triggers distillation. For mode selection, see Overview of applicable scenarios. Here only the SDK-layer value mapping is listed; the distinguishing criterion is whether a Rollout Component is passed (determines using native Rollout or Custom Rollout):

• Model OPD: do not pass a Rollout Component → use native Rollout, including pure distillation (functions=None) and add Reward (only pass Reward Component to overlay task reward) two tiers
• Agentic OPD (None Reward): only pass Rollout Component → Custom Rollout (supports multi-turn dialogue, custom interaction, structured tool call) + teacher distillation
• Agentic OPD (with Reward): simultaneously pass Rollout + Reward → Custom Rollout + teacher distillation + Reward
• Pass None(rather than empty list []) → pure distillation, reward derived metrics not reported, distillation loss-type metrics and teacher health-type metrics still reported normally

YAML Configure

Suitable for multiple experiments or managing multiple groups of parameters, via teacher_model Field specifies the teacher model.

## config.yaml（简化示意）
name: "my-opd-job"
model:
  name: "qwen3.5-9b"                           # 学生模型（训练对象）
teacher_model: "qwen3.5-397b-a17b"             # 教师模型（OPD 新增），传入即触发蒸馏

datasets:
  - type: "training"                           # 训练集
    data_source_type: "file_id"                # file_id 方式：SDK 自动上传并换 file_id
    file_name: "./data/train.jsonl"
  - type: "validation"                         # 验证集，训练中评估用
    data_source_type: "file_id"
    file_name: "./data/validation.jsonl"

functions:                                     # 不传 Rollout 组件即 Model OPD；纯蒸馏写 functions: null
  - type: "reward"                             # 仅传 Reward：原生 Rollout + 教师蒸馏 + 任务奖励
    name: "reward-1"
    weight: 1.0                                # 奖励权重，多 reward 时加权求和
    timeout: 120                               # 单次打分超时秒数
    fcmodel:
      classpath: "functions.reward.reward.MyRewardProcessor"  # 入口类全路径（示例类名，实际由您自定义）
    runtime:
      cpu: 2                                   # 函数实例 CPU 核数
      memory_size: 4096                        # 内存 MB
      concurrency: 20                          # 单实例并发打分数
      capacity: 10                             # 当前实例数
      min_capacity: 10                         # 最小实例数（保底）
      max_capacity: 30                         # 最大实例数（弹性上限）

training:
  type: "pg_opd"                               # OPD 训练类型；传 teacher_model 后自动置此值，无需手填
  hyper_parameters:
    algorithm: "gspo"                          # 训练算法
    learning_rate: 1e-6                         # 学习率，OPD 起点低于 SFT
    lr_scheduler_type: "cosine"                 # 学习率调度器
    n_epochs: 1                                # 训练轮次
    # ... 完整字段见下方 §训练超参数
  resources:
    charge_type: "mtu_postpaid"                # 后付费计费
    mtu_spec_code: "MTU4"                       # 算力规格
    mtu_capacity: 24                           # 容量

Load and Execute:

client = AgenticRL()                                            # 初始化训练客户端
client.init(config_path="config.yaml", name="my-opd-from-yaml") # 加载 YAML 配置，name 覆盖 YAML 中的 job_name
result = await client.run()                                     # 异步提交任务；run() 参数覆盖 init() 与 YAML 默认值

CLI Command

Pass dashscope rl run Submit,--teacher-model overrides in YAML teacher_model.

dashscope rl run -c config.yaml --teacher-model qwen3.5-397b-a17b
# -c 指定 YAML 配置文件路径
# --teacher-model 指定教师模型，覆盖 YAML 中的 teacher_model 字段；传入即启用 OPD

--teacher-modelhelp text (original): Enable OPD with this teacher model (overrides YAML); Rollout and Reward are optional. Meaning: Enable OPD and specify the teacher model (overrides YAML); Rollout and Reward are Optional.

Common CLI Commands:

import asyncio
from dashscope.finetune.agentic_rl import AgenticRL          # OPD 训练客户端
from dashscope.finetune.reinforcement import (                # 组件与数据类型
    RewardFunctionComponent, FunctionComponentModel,
    FunctionComponentRuntime, TrainingDataset,
    ValidationDataset, DataSourceType,
)

async def submit(args):
    client = AgenticRL()                                    # 初始化训练客户端
    # 纯蒸馏时传 None 而非空列表：None 使控制面（调度层）执行原生 Rollout，[] 不一定等价
    # --no-reward 是示例侧便利开关（非 SDK 参数），等价于 SDK 层 functions=None
    function_components = (
        None if args.no_reward                              # 纯蒸馏：不注册任何函数
        else [RewardFunctionComponent(                      # 加 Reward：原生 Rollout + 教师蒸馏 + 任务奖励
            name="reward-1", weight=1.0, timeout=120,       # weight=奖励权重，timeout=单次打分超时秒数
            fcmodel=FunctionComponentModel(
                zipdir=str(args.workspace_dir.resolve()),   # 工作区根目录，显式指定避免入口类定位失败
                classpath="functions.reward.reward.MyRewardProcessor"),  # 入口类全路径（示例类名，实际由您自定义）
            runtime=FunctionComponentRuntime(**build_runtime()))]        # 并发与弹性容量由 build_runtime() 构造
    )
    result = await client.run(                              # 异步提交：注册→上传→提交
        job_name=args.job_name,
        model=args.student_model,          # 学生模型（训练对象）
        teacher_model=args.teacher_model,  # 教师模型，传入即触发蒸馏
        training_datasets=[...],           # 训练集（示例省略构造）
        validation_datasets=[...],         # 验证集（示例省略构造）
        functions=function_components,     # None=纯蒸馏；[Reward...]=Model OPD；含 Rollout=Agentic OPD
        hyper_parameters={...},            # 训练超参（示例省略，见下方 §训练超参数）
    )
    return result.output.job_id            # 返回 job_id，供后续查询/取消/日志使用

Configure Teacher Model

teacher_model is the submission parameter that triggers distillation. Passing it triggers distillation; not passing it does not. After it is passed, the SDK internal field training_type is automatically set to pg_opd (strategy distillation training type identifier, corresponding to that in YAML training.type), no need to and cannot manually specify the training type.

Behavior Conclusion:

• Pass teacher_model forces training_type=pg_opd, no need to and cannot manually specify the training type
• Write in YAML training.type: pg_opd but not passed teacher_model when, the control plane reports an error error 3006(teacher model missing)
• hyper_parameters Undeclared keys will not be recognized, and will neither report errors nor take effect

Three hard constraints

Teacher model configuration must simultaneously satisfy the following three constraints; failure to meet any one will cause distillation to fail or the submission to be rejected.

Training Hyperparameters

hyper_parameters field names and value ranges are consistent, grouped into 4 categories: Algorithm (algorithm; kl_loss_coef not enabled in OPD) / Optimize (learning_rate, lr_scheduler_type, lr_warmup_steps, n_epochs) / Data throughput (batch_size, n_rollouts, ppo_mini_batch_size, max_length, max_prompt_length) / Evaluation & save (eval_steps, save_steps, save_strategy, save_total_limit).

Required parameters comprehensive table

The comprehensive table below merges field definitions, value ranges, and recommended starting points by model scale × data phase; you can launch training directly by adopting the starting point column, then iterate per §tuning decision table.

Model Scale Boundary: Student model parameter count ≤14B counts as small model, ≥30B counts as large model (e.g., Qwen3.5-9b is small, 397b-A17b is large). The "small model quick validation" column in the table applies to pipeline smoke testing and small-scale training, the "large model" column applies to formal large-scale training; for cases in between, take the union of both columns and fine-tune based on actual convergence.

Three parameters that must be adjusted together(determines which must be linked during tuning):

• Sample Count: batch_size × n_rollouts = actual samples generated per step
• mini-batch upper limit: ppo_mini_batch_size ≤ batch_size × n_rollouts
• learning_rate determines distillation convergence speed—OPD has no kl_loss_coef constrains model deviation from initial policy (prevents output style drift), learning rate is the only parameter controlling convergence speed

Optional Parameters

The following parameters are used to control distributed training resource configuration, generally using default values is sufficient. The default values in the table are the values when not passed; actual values are subject to what is passed at submission.

Teacher Model Deployment Parameters and Disabled Items

• algorithm: Default gspo(GSPO, group relative policy optimization). When reward is enabled, determines the implementation method of policy optimization; pure distillation (without registering reward) does not participate in loss calculation, generally no modification needed
• Teacher model deployment resources: can explicitly configure opd_teacher_* parameter, if not passed, takes the Default Value. Example default values are as follows:

Teacher model is deployed via vLLM (Model Studio reasoning engine); KV cache (mechanism in reasoning GPU memory that caches computed tokens) capacity determines scoring throughput. Adjust according to teacher model scale and sequence length:

• Long sequence: increase opd_teacher_max_model_len(along with opd_teacher_max_num_batched_tokens↑)
• Tight GPU memory: Lower opd_teacher_gpu_memory_utilization

Other parameters generally keep the Default.

• kl_loss_coef: OPD does not consume this parameter—distillation does not constrain policy divergence via KL divergence (a metric measuring the difference between two probability distributions), and convergence speed is controlled by learning_rate alone. The "—" marked in the required parameter table indicates OPD does not consume it. You can keep the Default or leave it empty; no need to zero it out or Delete it.

Parameter Tuning and Pitfall Avoidance

Tuning Decision Table

When metrics are abnormal, refer to this table to select the preferred adjustment item and magnitude; for complete metric meanings and troubleshooting decision tree, seeOnline Policy Distillation Observable Configuration.

Tuning Operation Recommendations

• Check first valid_token_rate: When this metric is below 1, the teacher model scoring has gaps, and the explanatory power of other metrics decreases. Investigate this first before tuning hyperparameters.
• Change only one variable at a time: Tuning both learning_rate and batch_size simultaneously makes attribution impossible.
• Train for eval_steps × 3 steps before judging the trend: Distillation curves have less noise, but trends are hard to judge over short periods. Wait at least 3 evaluation intervals before judging the direction.

Anti-patterns

The following configurations have been proven in practice to cause training failure or resource waste and should be avoided.

Task Management

This section provides commands for querying, canceling, deleting, and viewing logs of training tasks, as well as experiment management and continual training methods.

SDK / CLI Command Comparison

Task management commands (query, cancel, delete, logs) are as follows; pass an extra parameter when submitting teacher_model:

Experiment Management

• YAML Multiple-experiment mode:job_name identifies experiments, finetuned_output identifies output models; change these two each time to isolate experiments; YAML centralizes configuration management for Git versioning
• Configuration priority:__init__() < YAML < run() Runtime parameters — for controlled experiments only change the outermost layer
• Parameter sweep / CI scenarios (advanced usage, general users can skip): when the function is stable and only hyperparameters change, you can functions pass at the Component level entity_id(such as RewardFunctionComponent(..., entity_id="rw-xxx")) reuse registered components, saving repeated registration and upload. Entity ID (ro- represents Rollout, rw- represents Reward) is taken from the return value of the first registration, Field Name entity_id, can be put into CI configuration for long-term reuse:

result = await client.run(...)                    # 首次提交：注册函数组件并上传代码/数据
entity_id = result.output.entity_id              # 从首次注册返回值取 entity_id，ro- 表示 Rollout、rw- 表示 Reward
# 后续扫参/CI 只换超参时，在组件级传 entity_id="rw-xxx" 复用已注册组件，省去重复注册与上传

Continued Training and Checkpoint

Checkpoint retention and selection:

• save_strategy=steps + save_steps=100 + save_total_limit=3: Keep up to 3 piece, automatically clean the oldest when exceeded
• Which Checkpoint to select: When reward is enabled, look at validation/data/reward/mean@1 select the best point (in pure distillation without reward enabled, this metric is not reported; based on actor/distillation/loss trend stabilized point selection), not necessarily the last one
• Keep 3~5 piece to balance space and rollback needs

Continue training based on published Checkpoint:

• Continue training = use the model produced last time as model input parameter and submit again; hyperparameters can be fine-tuned (lr is usually reduced)
• Applicable: Data increment / Reward adjustment / long-chain gradual convergence
• Note: The base model version must be consistent; cross-base-model cannot directly continue training
• In a multi-phase training pipeline, each phase's output can serve as the next phase's model input, progressing level by level

Next Steps

• Review principles and scenario selection → Online Policy Distillation Training Overview
• Model OPD Development (Pure Distillation/Reward) → Model OPD Development
• Agentic OPD Development (Custom Rollout) → Agentic OPD Development
• Training Metrics Quick Reference, Effect Evaluation and Troubleshooting → Online Policy Distillation Observable Configuration

FAQ