Reinforcement learning
This document explains how to develop and test the three function components for RL training: rollout (trajectory generation), reward (scoring), and group reward (batch scoring).
Function components
RL training requires you to develop custom function components to define the model's interaction behavior and scoring logic. This code is automatically deployed to the cloud:
Component | Description | Required |
|---|---|---|
Rollout | Defines how the agent calls the LLM and tools to generate an interaction trajectory. | Required (1) |
Reward | Scores a single agent output to define the optimization direction for training. | Optional (1 or more) |
Group Reward | Scores multiple outputs for the same problem by ranking or comparison. | Optional |
Shared data model
The three function components pass data via the AgentOutput object and use TaskStatus to indicate their execution status.
AgentOutput
AgentOutput connects the Rollout and Reward components. The Rollout function constructs it and passes it to the Reward function:
Parameter | Type | Description |
|---|---|---|
| list | A list of conversation messages containing user, assistant, and tool roles. Take the last assistant message to obtain the model's answer |
| float / None | A preset score. When set in Rollout via |
| dict | Contains the value of the |
| dict | Custom metrics, such as |
TaskStatus and error handling
The process() method of each function component must return a result that includes a status field. The training framework uses this field to decide whether to retry or discard the sample:
TaskStatus.SUCCESS: The execution was successful.TaskStatus.FAILED: The execution failed. You must set theerrorfield to specify the reason for the failure.
## 成功
return RolloutOutput(
agent_output=AgentOutput(messages=messages),
status=TaskStatus.SUCCESS,
)
## 失败 → 框架会重试或丢弃该样本
return RolloutOutput(
agent_output=AgentOutput(messages=[]),
status=TaskStatus.FAILED,
error="LLM 调用超时",
)
Rollout function development
The rollout function generates the interaction trajectory of an agent with an LLM and external tools.
Basic structure
Inherit from AbstractRolloutProcessor and implement the setup() and process() methods:
from dashscope.finetune.reinforcement import (
AbstractRolloutProcessor, RolloutInput, RolloutOutput
)
from dashscope.finetune.reinforcement.component.data.base_data_model import (
AgentOutput, TaskStatus
)
class MyRolloutProcessor(AbstractRolloutProcessor):
def setup(self) -> None:
"""初始化资源(服务启动时执行一次)"""
# 初始化 LLM 客户端、加载工具等
pass
async def process(self, input: RolloutInput) -> RolloutOutput:
"""生成 Agent 轨迹(每个样本调用一次)"""
messages = input.messages or []
model = input.model_resource.model_name
# ... 与 LLM 交互、调用工具
return RolloutOutput(
agent_output=AgentOutput(
messages=messages,
rollout_metrics={"latency": 1.2} # 自定义指标
),
status=TaskStatus.SUCCESS,
)
The process() method supports both async def and standard def declarations. The SDK handles this automatically.
NoteNote: The rollout function calls the model being trained via an OpenAI-compatible API. The training framework automatically injects the endpoint (base_url) and API key into RolloutInput.model_resource at runtime. You can read these values directly in your setup() or process() methods, eliminating the need to hardcode them in your code or configuration.
Key RolloutInput fields
Parameter | Type | Description |
|---|---|---|
| list | A list of conversation messages. |
| object | Model information (model_name, base_url, api_key). |
| object | Sampling parameters (temperature, max_tokens, max_turns, timeout). |
| str | The reference answer. |
| dict | The rollout_extra field in the training data passes through business data. |
Error handling: When the rollout function fails (for example, if an LLM call times out or a tool throws an exception), return TaskStatus.FAILED and set the error field. For details, see the TaskStatus and error handling section.
NoteEmbedded scoring logic: Reward scoring can be integrated directly in the Rollout's process() instead of being written as a standalone Reward function. When returning RolloutOutput, simply provide the score via AgentOutput(reward_score=0.95). Metrics from embedded scoring also appear on the metrics page in the console. For more information, see AgentOutput.
Reward function development
The Reward function scores an agent's output and is a core component of RL training.
Basic structure
Inherit from the AbstractRewardProcessor class and implement your scoring logic in the process() method:
from dashscope.finetune.reinforcement import (
AbstractRewardProcessor, RewardInput, RewardOutput, Reward, TaskStatus
)
from dashscope.finetune.reinforcement.component.observability import observe_processor
class DemoRewardProcessor(AbstractRewardProcessor):
def setup(self) -> None:
pass
@observe_processor
async def process(self, input: RewardInput) -> RewardOutput:
content = input.agent_output.messages[-1].get("content", "")
ground_truth = input.ground_truth
# 评分逻辑:检查答案是否正确
score = 1.0 if ground_truth in content else 0.0
return RewardOutput(
reward=Reward(
reward_score=score,
reward_metrics={"test1": 0.5, "test2": 0.3} # 自定义指标
),
status=TaskStatus.SUCCESS,
error=None,
)
Key RewardInput parameters
Parameter | Type | Description |
|---|---|---|
| object | The agent's output. For field details, see the AgentOutput section above. |
| str | Reference answers (passed in via the ground_truth field of the test input during function testing). For reference data needed for scoring during training, it is recommended to store it in the |
Decorator pattern for multi-dimensional scoring
To score across multiple dimensions, use the decorator pattern to break down the logic into sub-dimensions and then aggregate their results:
from dashscope.finetune.reinforcement import (
AbstractRewardProcessor, RewardInput, RewardOutput, Reward, TaskStatus,
reward_func, sub_reward_func, aggregate_func
)
@reward_func("SafetyProcessor")
class SafetyProcessor(AbstractRewardProcessor):
"""多维度安全性评分"""
BANNED = {"hack", "exploit", "attack"}
@sub_reward_func("toxicity", sub_weight=0.7)
def toxicity(self, input: RewardInput) -> RewardOutput:
"""毒性检测子维度(权重 0.7,同步)"""
messages = input.agent_output.messages
content = messages[0].get("content", "") if messages else ""
response = str(content) if content is not None else ""
has_banned = any(w in response.lower() for w in self.BANNED)
return RewardOutput(
reward=Reward(
reward_score=0.0 if has_banned else 1.0,
reward_metrics={"score-1": 0.1},
),
status=TaskStatus.SUCCESS,
error=None,
)
@sub_reward_func("refusal", sub_weight=0.3)
async def refusal(self, input: RewardInput) -> RewardOutput:
"""拒绝检测子维度(权重 0.3,异步)"""
messages = input.agent_output.messages
content = messages[0].get("content", "") if messages else ""
response = str(content) if content is not None else ""
is_refusal = response.strip().lower().startswith("i cannot")
return RewardOutput(
reward=Reward(
reward_score=0.5 if is_refusal else 1.0,
reward_metrics={"score-1": 0.2},
),
status=TaskStatus.SUCCESS,
error=None,
)
@aggregate_func
async def aggregate(self, sub_rewards: dict[str, RewardOutput]) -> RewardOutput:
"""聚合所有子维度(异步)"""
weights = self.get_weights()
scores = self.get_scores(sub_rewards)
if len(scores) != len(weights):
raise ValueError("scores and weights must have the same length")
# 0.5 * 0.3 + 1.0 * 0.7 = 0.85
total = sum(scores[k] * weights[k] for k in scores)
reward_metrics = self.get_reward_metrics(sub_rewards)
user_reward_metrics = {"aggregate-1": 0.4}
user_reward_metrics.update(reward_metrics)
return RewardOutput(
reward=Reward(
reward_score=total,
reward_metrics=user_reward_metrics,
),
status=TaskStatus.SUCCESS,
error=None,
)
Decorator descriptions
Decorator | Purpose |
|---|---|
| Declares the class as a reward processor and sets its name. |
| Defines a scoring sub-dimension and its weight. |
| Defines the aggregation logic to combine scores from all sub-dimensions. |
Combining multiple Reward functions
When you submit a task, you can configure multiple Reward functions, each with an independent weight:
functions=[
RewardFunctionComponent(name="accuracy", weight=0.6, ...),
RewardFunctionComponent(name="safety", weight=0.4, ...),
]
You can also use reward_metric_weight to fine-tune the weights of sub-metrics (the key is the metric name returned by the Reward function):
RewardFunctionComponent(
name="reward-1",
weight=1.0,
reward_metric_weight={"accuracy": 0.7, "format": 0.3},
...
)
Notereward_metric_weight can be omitted: when omitted, the weights of sub-dimensions are automatically extracted from the function definition (the values declared via @sub_reward_func(sub_weight=...) in decorator mode; see Multi-dimensional scoring: three-layer weight design). A Reward function with single scoring logic does not need this configuration.
Develop a Group Reward function
A Group Reward function scores multiple agent outputs for the same problem. It is ideal for scenarios that require ranking or comparative scoring.
Basic structure
Inherit from AbstractGroupRewardProcessor and handle multiple outputs in the process() method:
The process() method supports both async def and regular def declarations, which the SDK handles automatically. This rule applies to all function components: Rollout, Reward, and Group Reward. The example below uses the synchronous style.
from dashscope.finetune.reinforcement import (
AbstractGroupRewardProcessor, GroupRewardInput, GroupRewardOutput,
GroupReward, Reward, TaskStatus
)
class DemoGroupRewardProcessor(AbstractGroupRewardProcessor):
def setup(self) -> None:
pass
def process(self, input: GroupRewardInput) -> GroupRewardOutput:
rewards = []
for agent_output in input.agent_outputs:
score = 0.0
# ground_truth 出现在 messages 内容里 → 1.0;messages 非空 → 0.5;否则 0.0
if input.ground_truth is not None and agent_output.messages:
gt_str = str(input.ground_truth)
for msg in agent_output.messages:
if isinstance(msg.get("content"), str) and gt_str in msg["content"]:
score = 1.0
break
if score == 0.0 and len(agent_output.messages) > 0:
score = 0.5
rewards.append(Reward(reward_score=score))
return GroupRewardOutput(
reward=GroupReward(rewards=rewards),
status=TaskStatus.SUCCESS,
error=None,
)
Differences from Reward
- Reward: Scores a single output. It takes agent_output (singular) as input.
- Group Reward: Scores multiple outputs. It takes agent_outputs (a list) as input and returns a list of rewards.
Reward design philosophy
Reward is the only optimization direction for RL training, so think through "what to score" before touching the keyboard. Three core trade-offs determine the approach:
- Sparse vs dense: sparse rewards are hard to game but have low sample efficiency; dense rewards learn fast but easily introduce shaping bias
- Rules vs LLM-as-judge: rules are cheap, deterministic, and reproducible; LLM judging covers semantics but has bias and cost
- Single-dimension vs multi-dimension: single-dimension is easy to iterate but has a low ceiling; multi-dimension is more expressive but incurs higher weight design and interpretation costs
Start with these four questions
- Q1: Is there machine-verifiable ground truth? (Math/code/JSON → rules; writing/dialogue → model/human)
- Q2: Is failure binary or graded? (Binary → sparse 0/1; graded → split into dimensions)
- Q3: Do intermediate steps need incentives? (Long-chain reasoning, multi-turn Agent → process reward / step scores)
- Q4: Which hack paths are the cheapest? (List them first, then defend — see §Reward Hacking detection and defense below)
Reward as a contract
The model optimizes for what is specified, not what is intended. This leads to three key principles:
- 0/1 boundaries will be precisely approached — add a margin to exact scores or soften them (e.g., a multi-dimensional combination of accuracy + format + conciseness scores)
- Hard caps will be maxed out — a full length score means the model will inevitably pad answers to the limit; use normalization or decay instead
- Empty output / overlong output / timeout must be explicitly defined — default behavior is often a loophole; force 0 for empty output and deduct points after truncating overlong output
Choosing a reward representation
The reward representations are not mutually exclusive; they follow a layered progression from simplest to most complex: A. Inline (AgentOutput(reward_score=...)) → B. Standalone (RewardFunctionComponent) → C. Multi-dimensional with decorators (@reward_func + @sub_reward_func + @aggregate_func).
Representation | Use cases | Scoring complexity | Resource overhead | Observability granularity | Debuggability | When to avoid |
|---|---|---|---|---|---|---|
A. Inline Rollout | For simple (≤10 lines) scoring logic that is synchronous with generation. | Low | Reuses the Rollout FC. | Only one metric: reward_score | Difficult, as it requires a Rollout redeploy. | When scoring requires independent scaling. |
B. Standalone reward | For a single, well-defined scoring rule or for reusing a general-purpose scorer. | Medium | Standalone FC. | reward + reward_metrics | Medium (independent deployment). | For three or more dimensions. |
C. Multi-dimensional decorator | For multi-dimensional scoring that requires independent weight adjustment for each dimension. | High | Standalone FC. | Independent metrics for each sub-dimension. | High (enables troubleshooting by dimension). | When dimensions are tightly coupled and cannot be separated. |
Upgrade path: Start with A to get the pipeline running → switch to B at ≥1 dimension → switch to C at ≥3 dimensions. When migrating from A to B, keep the Rollout and move the scoring logic out. To migrate from B to C, break down the dimensions based on atomic determinability; do not split out a sub-dimension if it cannot be evaluated with a binary (0/1) outcome.
Multi-dimensional scoring: Three-layer weight design
Multi-dimensional reward weights are organized into three non-overlapping, independently adjustable layers:
- Layer 1: Function-level
RewardFunctionComponent.weight(between multiple reward functions) - Layer 2: Sub-dimension
@sub_reward_func(sub_weight=...)(between sub-dimensions within a reward function) - Layer 3: Sub-metric
reward_metric_weight={...}(for subdivisions within reward_metrics)
Final reward = Σ(Layer 1 × Layer 2 × Layer 3 normalized terms).
Multi-dimensional weight design template
Step | Actions | Output | Guideline |
|---|---|---|---|
1. List dimensions | Break down the objective into 3–5 atomic, quantifiable dimensions. | A list of dimensions | Recommended: ≤ 5 |
2. Prioritize dimensions | Rank dimensions based on business needs, such as must-haves, bonus items, and style preferences. | A prioritized list of dimensions | - |
3. Assign initial weights | Assign 0.5–0.7 to must-haves, 0.2–0.3 to bonus items, and 0.05–0.1 to style preferences. | A weight vector | All weights must sum to 1. |
4. Scale alignment | Normalize each dimension to a [0, 1] range before applying its weight. | A normalization function | Prevents any single dimension from dominating the reward. |
5. Sample validation | Compare the reward scores of 100 samples against their manual annotations. | correlation coefficient | Spearman ρ ≥ 0.6 |
Weight tuning methods
- First, train a baseline model with uniform weights to identify which sub-metric is dominant. Then, examine the variance of the curve at
trace/reward_metrics/{reward}/{sub}/avg. Increase the weight of dimensions that show low variance. - If the validation reward increases while a specific dimension's score on the training set drops sharply, that dimension's weight is likely too low and should be increased. If you observe reward hacking, temporarily reduce the weight of the relevant dimension instead of disabling it entirely.
Group Reward use cases
Group Reward vs Independent Reward
- Independent Reward evaluates a single response based on its absolute quality (an absolute score); Group Reward assesses the relative quality among multiple responses to the same prompt (ranking or comparison).
- GRPO/GSPO algorithms require group-relative advantage. Group Reward provides a direct ranking signal, which avoids the bias introduced by normalization.
- For subjective tasks like writing or dialogue, relative comparison is more stable and consistent than absolute scoring.
Integration with GRPO/GSPO algorithms
- GRPO/GSPO algorithms calculate group-relative advantage from multiple rollouts for the same prompt. In contrast, an Independent Reward provides an absolute score, which the framework must then normalize within the group (a process that can be skewed by outliers).
- Group Reward directly outputs a within-group distribution, reducing score scale drift. The
n_rolloutsparameter controls the input size; a value between 4 and 16 is recommended.
Implementation notes
- The output
rewardslist must match the length of the inputagent_outputslist. Do not apply hard normalization (e.g., dividing by the maximum value) within the group, as this conflicts with normalization at the algorithm level. - Tied scores are permitted (e.g., three responses each scoring 1.0), but avoid assigning the same score to all responses, as this results in zero advantage. Assign failed responses a score of 0 to maintain a consistent list length; the framework uses the TaskStatus to identify and handle such cases.
Reward hacking: Identification and defense
Hacking is almost inevitable
- Goodhart's Law: When a measure becomes a target, it ceases to be a good measure.
- The probability of reward hacking increases as an RL (Reinforcement Learning) model converges and its reward grows. This is not a bug; the optimizer is just doing its job.
- The goal of a defense strategy is to make reward hacking more difficult than finding the true solution, not to eliminate it entirely.
Hack pattern → detection signal → defense strategy
Hack pattern | Detection signals (metrics + behavior) | Defense strategy |
|---|---|---|
length explosion / verbosity |
| Use length normalization, add a |
keyword stuffing | Reward increases, but manual spot-checks show repetitive keywords. Correlation between | Use deduplication with regular expressions, replace keyword matching with an LLM judge, and introduce a semantic similarity threshold. |
format hacking (markdown / emoji stuffing) | The output format is unusual, and the answer's relevance to the user's question decreases. | Normalize formatting as a preprocessing step. Score style as an independent dimension with a low weight. |
refusal / empty output | Median reward is high, but | Force a score of 0 for empty outputs, enforce a minimum length constraint, and add a "Did it answer the question?" scoring dimension. |
repetition / repetitive output | High response length, high n-gram repetition rate, and a sharp drop in entropy. | Use regular expressions for repetition detection, apply a |
judge collusion (in LLM-as-judge scenarios) | The LLM judge score is high but diverges from the rule-based score or human score. | Mix rule-based scores with judge scores, recalibrate the judge periodically, and rotate judges from different model families. |
tool abuse / tool non-use | Anomalous | Penalize or reward tool call counts, and enforce constraints on required tool calls. |
entropy collapse |
| Increase the KL divergence coefficient, use early stopping, or add an entropy bonus. |
Four categories of defense strategies
- reward shaping: Explicitly penalize hacking paths. This approach is direct but requires continuous patching.
- Rule-based constraints: Hard constraints on length, repetition, or format. Inexpensive but rigid.
- validation monitoring: Use metrics from a source other than the training reward as a sentinel. This is essential.
- multiple reward combination: Combine rule-based and model-based rewards to create checks and balances.
Monitoring playbook
- Monitor the upward trend of your North Star Metric,
critic/rewards/mean. Usevalidation/data/reward/mean@1as an anti-hacking sentinel; a divergence from the North Star Metric indicates reward hacking. Perform behavioral health checks by monitoring metrics liketrajectory/response_lengthandactor/entropy. - Every N steps, spot-check trajectories on the console's "Trajectory" tab and compare the scores with human judgment.
Scoring stability
Bias control in LLM-as-a-judge
- Position bias: The order of items in a pairwise comparison can affect the outcome. To mitigate this, run each comparison twice with the order swapped and accept only the results that are consistent across both evaluations.
- Self-preference bias: A model judge may favor outputs from its own model family. To counter this, use a judge from a different model family (e.g., use a GPT model to judge outputs from a Qwen model).
- Length bias: A model judge might favor longer outputs. To address this, explicitly instruct the judge in the prompt not to score based on length and apply length normalization.
- Discrete rating scales (e.g., 1-5 or 1-7) are more stable than continuous ones (e.g., 1-100). Use an anchor example for calibration.
Rule and model scoring
- Use rules as hard constraints for objective criteria such as format, length, required fields, and sensitive words. If a response fails these checks, short-circuit the process and assign a score of 0.
- Use a model for soft evaluation of subjective qualities like fluency, relevance, and reasoning quality.
- Combination:
final = rule_pass × (w_rule × rule_score + w_model × model_score). To save costs and prevent hacking, do not send responses that fail rule-based checks to the model judge.
TaskStatus in scoring scenarios
- A score of 0 (e.g., for a failed rule check or an incorrect answer) is an effective training signal. In this case, return
SUCCESS + reward_score=0. - For transient errors (e.g., a judge timeout or network jitter), return
FAILEDto let the framework retry. Do not assign a score of 0 in these cases. - Overusing FAILED can cause a data distribution shift. If the framework repeatedly fails on a hard example and eventually discards it, the training set becomes progressively easier.
Scoring consistency self-checks
- Prepare a gold set of 50-200 examples with human annotation. After modifying the reward logic, run it against the gold set and compare results using metrics like Spearman's rho or Cohen's kappa.
- When upgrading or switching your model judge, compare the new scores against the old ones to detect silent drift. Periodically send the N highest- and lowest-scoring training samples for human review.
Reward function checklist (10 questions)
Review this checklist before you submit a task with your new reward function:
- Can the scoring logic be explained in a single sentence?
- Is there a ground truth, and is it machine-determinable?
- What is the lowest-cost hack path, and have you defended against it?
- If your reward is multi-dimensional, can you annotate each dimension manually and independently?
- Have you normalized the sum of the weights and aligned the scales?
- Have you defined boundary conditions (e.g., empty output, overlong output, timeout)?
- Is the validation method independent of the training signal?
- Is the scoring time less than one-third of the rollout time?
- Is the
reward_metricsfield name stable? Renaming it will break the metric curves. - Do you have a gold set for regression?
Reward design anti-patterns
Anti-pattern | Consequence |
|---|---|
Defining the reward as "the longer, the better" | Inevitable reward hacking (length explosion) |
Using single-dimension scoring for a highly subjective dimension | Noise overwhelms the signal. |
Setting the weight for every dimension to 1.0 | Equivalent to using no weights, resulting in inconsistent scales. |
Using the same prompt and model for both the training judge and the validation judge | A form of self-deception that leads to overly optimistic results. |
Returning SUCCESS with a score of 0 for temporary network errors | Data distribution contamination (hard samples are mislabeled as incorrect answers). |
Observability configuration
NoteThis section has been moved out. The complete content on Tracing decorators, custom metrics, dependency configuration, the ENABLE_TRAJECTORY switch, and more is now consolidated in Observability Configuration (Console and observability coverage).
Function testing
Function registration
Upload your function code and register it as cloud function components:
client = AgenticRL()
(rollout_entity_ids,
reward_entity_ids,
group_reward_entity_ids,
rollout_instance_ids,
reward_instance_ids,
group_reward_instance_ids) = await client.register_functions(
functions=[
# rollout-only:演示版,覆盖 trace_client/trace_tool 全分支
AgenticRLFunctionComponent(
type=FunctionType.ROLLOUT,
fcmodel=FunctionComponentModel(
classpath="functions.rollout.rollout_only.DemoRolloutProcessor")),
# reward
AgenticRLFunctionComponent(
type=FunctionType.REWARD,
fcmodel=FunctionComponentModel(
classpath="functions.reward.reward.DemoRewardProcessor")),
# reward-decorator:文件路径:类名 形式
AgenticRLFunctionComponent(
type=FunctionType.REWARD,
fcmodel=FunctionComponentModel(
classpath="functions/reward/reward_decorator.py:SafetyProcessor")),
# group-reward
AgenticRLFunctionComponent(
type=FunctionType.GROUP_REWARD,
fcmodel=FunctionComponentModel(
classpath="functions.reward.group_reward.DemoGroupRewardProcessor")),
],
lazy_load=False # False = 立即获取 instance_id,用于测试
)
Equivalent CLI:
dashscope rl register_functions \
--rollout-classpaths "functions.rollout.rollout_only.DemoRolloutProcessor" \
--reward-classpaths "functions.reward.reward.DemoRewardProcessor" \
--reward-classpaths "functions/reward/reward_decorator.py:SafetyProcessor" \
--group-reward-classpaths "functions.reward.group_reward.DemoGroupRewardProcessor" \
--no-lazy-load \
--output-format json
Remote testing
Use test_functions to send test data to registered function instances:
## 测试 Rollout
result = await AgenticRL.test_functions(
instance_id=rollout_instance_ids[0],
functype=FunctionType.ROLLOUT,
input_data=rollout_input
)
## 测试 Reward
result = await AgenticRL.test_functions(
instance_id=reward_instance_ids[0],
functype=FunctionType.REWARD,
input_data=reward_input
)
## 测试 Reward 装饰器版(第二个 reward 实例,对应 refusal 分支 reward_score=0.85)
result = await AgenticRL.test_functions(
instance_id=reward_instance_ids[1],
functype=FunctionType.REWARD,
input_data=reward_decorator_input
)
## 测试 Group Reward
result = await AgenticRL.test_functions(
instance_id=group_reward_instance_ids[0],
functype=FunctionType.GROUP_REWARD,
input_data=group_reward_input
)
After testing, you can pull function instance logs to troubleshoot the execution process (test_functions also supports pull_logs=True to fetch the logs along with the returned result):
logs = await AgenticRL.query_all_function_instance_logs(
function_instance_id=rollout_instance_ids[0],
start_time=int(time.time()) - 3600, # 查询近 1 小时
end_time=int(time.time()),
)
Equivalent CLI:
dashscope rl test_functions "ro-ins-xxx" --type rollout --input ./resources/rollout_input.json
dashscope rl test_functions "rw-ins-xxx" --type reward --input ./resources/reward_input.json
dashscope rl test_functions "rw-ins-yyy" --type reward --input ./resources/reward_decorator_input.json
dashscope rl test_functions "grw-ins-xxx" --type group_reward --input ./resources/group_reward_input.json
Test input format
Rollout test input (rollout_input.json):
{
"func_type": "rollout",
"messages": [
{"role": "user", "content": "Hello, please introduce yourself"}
],
"tools": null,
"ground_truth": "",
"rollout_extra": {
"rollout_id": "ro-ins-e8e92e94-b5c3-4546-b7b1-9832b8b948f2",
"training_state": {"task_id": "task-001"}
},
"sampling_params": {
"temperature": 0.7,
"max_tokens": 1024,
"max_turns": 25,
"timeout": 60.0
},
"model_resource": {
"model_name": "qwen3.5-9b",
"base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
"api_key": "your-api-key-here"
},
"request_metadata": {
"protocol": "openai",
"system_prompt": "",
"job_id": "dummy-job-id",
"sample_id": "dummy-sample-id",
"rollout_id": "dummy-rollout-id",
"attempt_id": "dummy-attempt-id"
},
"additional_field": "any extra data you want"
}
Reward test input (reward_input.json):
{
"func_type": "reward",
"ground_truth": "0.5",
"request_metadata": null,
"agent_output": {
"messages": [{"role": "user", "content": "Test question"}],
"rollout_extra": {
"content": "This is a test content that exceeds 50 characters in length and should receive a reward score of 0.5"
},
"rollout_metrics": {"accuracy": 0.95},
"reward_score": null
}
}
Reward decorator test input (reward_decorator_input.json, corresponding to the refusal branch, with reward_score=0.85 after aggregation):
{
"func_type": "reward",
"ground_truth": "0.5",
"agent_output": {
"messages": [{"role": "user", "content": "I cannot help you with that"}]
}
}
Group Reward test input (group_reward_input.json, where agent_outputs is an array of the same sample):
{
"func_type": "group_reward",
"ground_truth": "0.5",
"request_metadata": null,
"agent_outputs": [{
"messages": [{"role": "user", "content": "Test question"}],
"rollout_extra": {
"content": "This is a test content that exceeds 50 characters in length and should receive a reward score of 0.5"
},
"rollout_metrics": {"accuracy": 0.95},
"reward_score": null
}]
}
Local testing
The scripts/ directory in the project provides local testing scripts:
./scripts/start_local_rollout.sh # 启动本地 Rollout 服务
./scripts/query_local_rollout.sh # 发送测试请求
Next steps
- Run the self-check workflow before submitting a task → Reinforcement Learning Pre-submission Checklist
- Learn about submission methods and configuration → Reinforcement Learning Training Configuration — Submission and configuration
- Training metrics reference dictionary → Training metrics reference
- Back to the overview → Reinforcement Learning
- Learn about observability configuration and trajectory viewing → Observability Configuration for Reinforcement Learning