Reinforcement learning training

RL training workflow

The RL training process consists of five steps, from environment setup to model deployment:

1. Set up the environment — Install the offline SDK, get an API key, and complete RL service authorization.

2. Prepare data and code — Download the demo package to understand the data format and project structure.

3. Submit the training job — Run the demo to deploy functions, upload data, and submit the job in one step.

4. Monitor training status — View the training status, logs, and reward trends.

5. Publish and deploy the model — Publish a checkpoint to "My Models" and deploy it for API access.

Training data (user questions + reference answers)
            │
            ▼
Rollout: The model generates candidate results based on the current policy.
            │
            ▼
Reward: Scores the result on a scale of 0 to 1 by comparing it with the reference answer. This step can be merged with the rollout.
            │
            ▼
Policy update: Reinforce high-scoring policies and suppress low-scoring ones based on the reward signal.
            │
            ▼
Iteration: Resample using the updated model to improve quality in each round.

Supported models

Training unit pricing

Environment setup

Follow these steps to configure your environment:

1. Get an API key: Obtain a DashScope API key from the Model Studio console.

2. Authorize the RL service: On the Model Fine-tuning page in the Model Studio console, complete the one-click authorization. The first time you use RL training, the console prompts you to authorize Alibaba Cloud OpenTelemetry, Function Compute (FC), and Log Service (SLS).

   

3. Set environment variables: The local SDK reads the following variables to control packaging, uploading, and deployment:

   • DASHSCOPE_API_KEY (Required): Your API key.

   • FC_PYPI_LIB (Required): Specifies the offline dashscope package to be installed when the FC container starts. The SDK automatically injects this variable as FC_SDK_PACKAGE during deployment, so you do not need to install it manually in FC. The .whl filename must exactly match this variable and be placed in the project's root directory.

   • LOG_LEVEL (Optional, default: info): Set to debug to output complete request/response information, which is useful for troubleshooting submission failures.

   export DASHSCOPE_API_KEY="sk-your-api-key"
   export FC_PYPI_LIB="dashscope-1.25.16-py3-none-any.whl"
   export LOG_LEVEL="info"
   

Data and code preparation

1. Download the demo package agentic-rl-example.zip. (This demo requires a minimum of 24 Type IV model training units to run.)

2. After downloading the demo package, install the DashScope SDK (requires Python >= 3.10). Follow the instructions in the demo package to install the required dependencies:

   pip install dashscope-1.25.16-py3-none-any.whl  # Install the local .whl package
   # Or use the PyPI source: pip install dashscope==1.25.16

The project has the following structure:

agentic-rl-example/
├── submit_job.py                         ← Main entry point for submitting the training job
├── test_functions.py                     ← Function registration and remote testing
├── config.yaml                           ← Optional YAML configuration file
├── dashscope-1.25.16-py3-none-any.whl   ← Offline SDK installation package
├── requirements.txt                      ← List of dependencies
├── data/
│   ├── calc_train.jsonl                  ← Training data (full version)
│   ├── calc_train_min.jsonl              ← Training data (demo uses the minimal version by default)
│   ├── calc_validation.jsonl             ← Validation data (full version)
│   └── calc_validation_min.jsonl         ← Validation data (demo uses the minimal version by default)
├── functions/
│   ├── __init__.py
│   ├── rollout/
│   │   ├── rollout.py                    ← Rollout function (LangGraph + MCP tool)
│   │   └── rollout_only.py              ← Rollout function (simple version, with trajectory example)
│   └── reward/
│       ├── reward.py                     ← Reward function (basic scoring)
│       ├── reward_decorator.py           ← Reward function (decorator style)
│       └── group_reward.py              ← Reward function (group scoring)
└── resources/
    ├── rollout_input.json               ← Rollout test input
    ├── reward_input.json                ← Reward test input
    ├── reward_decorator_input.json      ← Decorator reward test input
    └── group_reward_input.json          ← Group reward test input

{"messages": [{"content": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?", "role": "user"}], "rollout_extra": {"solution": "72"}}

Job submission

The demo already includes the rollout and reward functions. Run the following command to deploy the functions, upload data, and submit the training job in a single step:

python submit_job.py

import asyncio
from dashscope.finetune.reinforcement import (
    RolloutFunctionComponent, RewardFunctionComponent,
    FunctionComponentModel, FunctionComponentRuntime,
    TrainingDataset, ValidationDataset, DataSourceType,
)
from dashscope.finetune.agentic_rl import AgenticRL

async def main():
    client = AgenticRL()

    # client.run() handles everything in one step: registering functions to FC, uploading training data, and submitting the training job
    result = await client.run(
        model="qwen3.5-9b",                              # Base model
        training_datasets=[TrainingDataset(              # Training dataset
            data_source_type=DataSourceType.FILE_ID,
            file_name="./data/calc_train_min.jsonl",
        )],
        validation_datasets=[ValidationDataset(          # Validation dataset (optional)
            data_source_type=DataSourceType.FILE_ID,
            file_name="./data/calc_validation_min.jsonl",
        )],
        functions=[
            # Rollout function: Defines how the model performs policy sampling
            RolloutFunctionComponent(
                name="rollout-1",
                timeout=600,
                fcmodel=FunctionComponentModel(
                    classpath="functions.rollout.rollout.CalcXRolloutProcessor"),
                runtime=FunctionComponentRuntime(
                    cpu=2, memory_size=4096, disk_size=512,
                    concurrency=30, capacity=30,
                    env={"ENABLE_TRAJECTORY": "true"})),  # Environment variables for the FC function
            RewardFunctionComponent(
                name="reward-1",
                weight=1.0,
                timeout=120,
                fcmodel=FunctionComponentModel(
                    classpath="functions.reward.reward.DemoRewardProcessor"),
                runtime=FunctionComponentRuntime(
                    cpu=2, memory_size=4096, disk_size=512,
                    concurrency=30, capacity=30)),
        ],
        # MTU billing configuration
        resource_config={
            "charge_type": "mtu_postpaid",   # pay-as-you-go
            "mtu_spec_code": "MTU4",         # Training unit specification
            "mtu_capacity": 1,               # Number of MTUs
        },
        # Hyperparameters for the GSPO algorithm (the following are all required hyperparameters for the non-MoE qwen3.5-9b model)
        hyper_parameters={
            "algorithm": "gspo",             # Training algorithm
            "n_epochs": 5,                   # Number of training epochs
            "learning_rate": 7e-6,           # Learning rate
            "lr_scheduler_type": "cosine",   # Learning rate scheduler policy
            "lr_warmup_steps": -1,           # Learning rate warmup steps (-1 = auto)
            "batch_size": 128,               # Batch size
            "n_rollouts": 8,                 # Number of rollouts per sample
            "ppo_mini_batch_size": 16,       # PPO mini-batch size
            "kl_loss_coef": 0.001,           # KL divergence loss coefficient
            "max_length": 8192,              # Maximum sequence length (prompt + response)
            "max_prompt_length": 2048,       # Maximum prompt length
            "eval_steps": 50,                # Perform evaluation every N steps
            "save_steps": 100,               # Save a checkpoint every N steps
            "save_strategy": "steps",        # Save strategy (steps or epoch)
            "save_total_limit": 3,           # Maximum number of checkpoints to keep
        })

    if result.status_code == 200:
        print(f"Training job submitted. Job ID: {result.output.job_id}")
    else:
        print(f"Submission failed: {result}")

asyncio.run(main())

Training monitoring and analysis

After you submit the job, the job_id returned by run() is the unique identifier for querying the training status. Training typically takes from tens of minutes to several hours, depending on the dataset size and model.

In the Model Studio console, go to the Model Fine-tuning page and click a job to open its details page, which contains the following five tabs:

For a detailed description of each tab, see Observability Configuration for Reinforcement Learning.

Model publishing and deployment

When training is complete, the final checkpoint is automatically published to My Models. To publish an intermediate checkpoint, go to the Model Fine-tuning > Output page in the Model Studio console and do so manually. For details, see Model Fine-tuning Console (Model Studio documentation, online only).

After a model is published, you can deploy it from the My Models page. Once deployed, you can call the model via an API. For details, see Model Deployment (Model Studio documentation, online only).

Next steps

• Learn the full details of function development → Reinforcement Learning Development Guide

• Learn about training configuration and monitoring → Reinforcement Learning Training Configuration — Submission and Configuration

• Reference for training metrics → Observability Configuration and Metric Reference for Reinforcement Learning

• Learn about observability configuration and trajectory viewing → Observability Configuration and Metric Reference for Reinforcement Learning