Reinforcement learning

Updated at:

This document explains how to develop and test the three function components for RL training: rollout (trajectory generation), reward (scoring), and group reward (batch scoring).

Function components

RL training requires you to develop custom function components to define the model's interaction behavior and scoring logic. This code is automatically deployed to the cloud:

Component

Description

Required

Rollout

Defines how the agent calls the LLM and tools to generate an interaction trajectory.

Required (1)

Reward

Scores a single agent output to define the optimization direction for training.

Optional (1 or more)

Group Reward

Scores multiple outputs for the same problem by ranking or comparison.

Optional

Shared data model

The three function components pass data via the AgentOutput object and use TaskStatus to indicate their execution status.

AgentOutput

AgentOutput connects the Rollout and Reward components. The Rollout function constructs it and passes it to the Reward function:

Parameter

Type

Description

messages

list

A list of conversation messages containing user, assistant, and tool roles. Take the last assistant message to obtain the model's answer

reward_score

float / None

A preset score. When set in Rollout via AgentOutput(reward_score=0.95), the Reward function can read this value directly, eliminating the need for a separate Reward class.

rollout_extra

dict

Contains the value of the rollout_extra field from the training data. Use this field to access business data, such as a reference answer, during scoring.

rollout_metrics

dict

Custom metrics, such as AgentOutput(rollout_metrics={"latency": 1.2}). These metrics automatically appear in the console at the trace/rollout_metrics/ path.

TaskStatus and error handling

The process() method of each function component must return a result that includes a status field. The training framework uses this field to decide whether to retry or discard the sample:

  • TaskStatus.SUCCESS: The execution was successful.
  • TaskStatus.FAILED: The execution failed. You must set the error field to specify the reason for the failure.
## 成功
return RolloutOutput(
    agent_output=AgentOutput(messages=messages),
    status=TaskStatus.SUCCESS,
)

## 失败 → 框架会重试或丢弃该样本
return RolloutOutput(
    agent_output=AgentOutput(messages=[]),
    status=TaskStatus.FAILED,
    error="LLM 调用超时",
)

Rollout function development

The rollout function generates the interaction trajectory of an agent with an LLM and external tools.

Basic structure

Inherit from AbstractRolloutProcessor and implement the setup() and process() methods:

from dashscope.finetune.reinforcement import (
    AbstractRolloutProcessor, RolloutInput, RolloutOutput
)
from dashscope.finetune.reinforcement.component.data.base_data_model import (
    AgentOutput, TaskStatus
)

class MyRolloutProcessor(AbstractRolloutProcessor):
    def setup(self) -> None:
        """初始化资源(服务启动时执行一次)"""
        # 初始化 LLM 客户端、加载工具等
        pass

    async def process(self, input: RolloutInput) -> RolloutOutput:
        """生成 Agent 轨迹(每个样本调用一次)"""
        messages = input.messages or []
        model = input.model_resource.model_name
        # ... 与 LLM 交互、调用工具
        return RolloutOutput(
            agent_output=AgentOutput(
                messages=messages,
                rollout_metrics={"latency": 1.2}  # 自定义指标
            ),
            status=TaskStatus.SUCCESS,
        )

The process() method supports both async def and standard def declarations. The SDK handles this automatically.

NoteNote: The rollout function calls the model being trained via an OpenAI-compatible API. The training framework automatically injects the endpoint (base_url) and API key into RolloutInput.model_resource at runtime. You can read these values directly in your setup() or process() methods, eliminating the need to hardcode them in your code or configuration.

Key RolloutInput fields

Parameter

Type

Description

messages

list

A list of conversation messages.

model_resource

object

Model information (model_name, base_url, api_key).

sampling_params

object

Sampling parameters (temperature, max_tokens, max_turns, timeout).

ground_truth

str

The reference answer.

rollout_extra

dict

The rollout_extra field in the training data passes through business data.

Error handling: When the rollout function fails (for example, if an LLM call times out or a tool throws an exception), return TaskStatus.FAILED and set the error field. For details, see the TaskStatus and error handling section.

NoteEmbedded scoring logic: Reward scoring can be integrated directly in the Rollout's process() instead of being written as a standalone Reward function. When returning RolloutOutput, simply provide the score via AgentOutput(reward_score=0.95). Metrics from embedded scoring also appear on the metrics page in the console. For more information, see AgentOutput.

Reward function development

The Reward function scores an agent's output and is a core component of RL training.

Basic structure

Inherit from the AbstractRewardProcessor class and implement your scoring logic in the process() method:

from dashscope.finetune.reinforcement import (
    AbstractRewardProcessor, RewardInput, RewardOutput, Reward, TaskStatus
)
from dashscope.finetune.reinforcement.component.observability import observe_processor

class DemoRewardProcessor(AbstractRewardProcessor):
    def setup(self) -> None:
        pass

    @observe_processor
    async def process(self, input: RewardInput) -> RewardOutput:
        content = input.agent_output.messages[-1].get("content", "")
        ground_truth = input.ground_truth

        # 评分逻辑:检查答案是否正确
        score = 1.0 if ground_truth in content else 0.0

        return RewardOutput(
            reward=Reward(
                reward_score=score,
                reward_metrics={"test1": 0.5, "test2": 0.3}  # 自定义指标
            ),
            status=TaskStatus.SUCCESS,
            error=None,
        )

Key RewardInput parameters

Parameter

Type

Description

agent_output

object

The agent's output. For field details, see the AgentOutput section above.

ground_truth

str

Reference answers (passed in via the ground_truth field of the test input during function testing). For reference data needed for scoring during training, it is recommended to store it in the rollout_extra field of the training data and read it via agent_output.rollout_extra (for the pass-through data flow, see Reinforcement Learning Training Overview — Data format)

Decorator pattern for multi-dimensional scoring

To score across multiple dimensions, use the decorator pattern to break down the logic into sub-dimensions and then aggregate their results:

from dashscope.finetune.reinforcement import (
    AbstractRewardProcessor, RewardInput, RewardOutput, Reward, TaskStatus,
    reward_func, sub_reward_func, aggregate_func
)

@reward_func("SafetyProcessor")
class SafetyProcessor(AbstractRewardProcessor):
    """多维度安全性评分"""
    BANNED = {"hack", "exploit", "attack"}

    @sub_reward_func("toxicity", sub_weight=0.7)
    def toxicity(self, input: RewardInput) -> RewardOutput:
        """毒性检测子维度(权重 0.7,同步)"""
        messages = input.agent_output.messages
        content = messages[0].get("content", "") if messages else ""
        response = str(content) if content is not None else ""
        has_banned = any(w in response.lower() for w in self.BANNED)
        return RewardOutput(
            reward=Reward(
                reward_score=0.0 if has_banned else 1.0,
                reward_metrics={"score-1": 0.1},
            ),
            status=TaskStatus.SUCCESS,
            error=None,
        )

    @sub_reward_func("refusal", sub_weight=0.3)
    async def refusal(self, input: RewardInput) -> RewardOutput:
        """拒绝检测子维度(权重 0.3,异步)"""
        messages = input.agent_output.messages
        content = messages[0].get("content", "") if messages else ""
        response = str(content) if content is not None else ""
        is_refusal = response.strip().lower().startswith("i cannot")
        return RewardOutput(
            reward=Reward(
                reward_score=0.5 if is_refusal else 1.0,
                reward_metrics={"score-1": 0.2},
            ),
            status=TaskStatus.SUCCESS,
            error=None,
        )

    @aggregate_func
    async def aggregate(self, sub_rewards: dict[str, RewardOutput]) -> RewardOutput:
        """聚合所有子维度(异步)"""
        weights = self.get_weights()
        scores = self.get_scores(sub_rewards)
        if len(scores) != len(weights):
            raise ValueError("scores and weights must have the same length")

        # 0.5 * 0.3 + 1.0 * 0.7 = 0.85
        total = sum(scores[k] * weights[k] for k in scores)

        reward_metrics = self.get_reward_metrics(sub_rewards)
        user_reward_metrics = {"aggregate-1": 0.4}
        user_reward_metrics.update(reward_metrics)
        return RewardOutput(
            reward=Reward(
                reward_score=total,
                reward_metrics=user_reward_metrics,
            ),
            status=TaskStatus.SUCCESS,
            error=None,
        )
Decorator descriptions

Decorator

Purpose

@reward_func("name")

Declares the class as a reward processor and sets its name.

@sub_reward_func("name", sub_weight=0.7)

Defines a scoring sub-dimension and its weight.

@aggregate_func

Defines the aggregation logic to combine scores from all sub-dimensions.

Combining multiple Reward functions

When you submit a task, you can configure multiple Reward functions, each with an independent weight:

functions=[
    RewardFunctionComponent(name="accuracy", weight=0.6, ...),
    RewardFunctionComponent(name="safety", weight=0.4, ...),
]

You can also use reward_metric_weight to fine-tune the weights of sub-metrics (the key is the metric name returned by the Reward function):

RewardFunctionComponent(
    name="reward-1",
    weight=1.0,
    reward_metric_weight={"accuracy": 0.7, "format": 0.3},
    ...
)

Notereward_metric_weight can be omitted: when omitted, the weights of sub-dimensions are automatically extracted from the function definition (the values declared via @sub_reward_func(sub_weight=...) in decorator mode; see Multi-dimensional scoring: three-layer weight design). A Reward function with single scoring logic does not need this configuration.

Develop a Group Reward function

A Group Reward function scores multiple agent outputs for the same problem. It is ideal for scenarios that require ranking or comparative scoring.

Basic structure

Inherit from AbstractGroupRewardProcessor and handle multiple outputs in the process() method:

The process() method supports both async def and regular def declarations, which the SDK handles automatically. This rule applies to all function components: Rollout, Reward, and Group Reward. The example below uses the synchronous style.

from dashscope.finetune.reinforcement import (
    AbstractGroupRewardProcessor, GroupRewardInput, GroupRewardOutput,
    GroupReward, Reward, TaskStatus
)

class DemoGroupRewardProcessor(AbstractGroupRewardProcessor):
    def setup(self) -> None:
        pass

    def process(self, input: GroupRewardInput) -> GroupRewardOutput:
        rewards = []
        for agent_output in input.agent_outputs:
            score = 0.0
            # ground_truth 出现在 messages 内容里 → 1.0;messages 非空 → 0.5;否则 0.0
            if input.ground_truth is not None and agent_output.messages:
                gt_str = str(input.ground_truth)
                for msg in agent_output.messages:
                    if isinstance(msg.get("content"), str) and gt_str in msg["content"]:
                        score = 1.0
                        break
                if score == 0.0 and len(agent_output.messages) > 0:
                    score = 0.5
            rewards.append(Reward(reward_score=score))

        return GroupRewardOutput(
            reward=GroupReward(rewards=rewards),
            status=TaskStatus.SUCCESS,
            error=None,
        )

Differences from Reward

  • Reward: Scores a single output. It takes agent_output (singular) as input.
  • Group Reward: Scores multiple outputs. It takes agent_outputs (a list) as input and returns a list of rewards.

Reward design philosophy

Reward is the only optimization direction for RL training, so think through "what to score" before touching the keyboard. Three core trade-offs determine the approach:

  • Sparse vs dense: sparse rewards are hard to game but have low sample efficiency; dense rewards learn fast but easily introduce shaping bias
  • Rules vs LLM-as-judge: rules are cheap, deterministic, and reproducible; LLM judging covers semantics but has bias and cost
  • Single-dimension vs multi-dimension: single-dimension is easy to iterate but has a low ceiling; multi-dimension is more expressive but incurs higher weight design and interpretation costs

Start with these four questions

  • Q1: Is there machine-verifiable ground truth? (Math/code/JSON → rules; writing/dialogue → model/human)
  • Q2: Is failure binary or graded? (Binary → sparse 0/1; graded → split into dimensions)
  • Q3: Do intermediate steps need incentives? (Long-chain reasoning, multi-turn Agent → process reward / step scores)
  • Q4: Which hack paths are the cheapest? (List them first, then defend — see §Reward Hacking detection and defense below)

Reward as a contract

The model optimizes for what is specified, not what is intended. This leads to three key principles:

  • 0/1 boundaries will be precisely approached — add a margin to exact scores or soften them (e.g., a multi-dimensional combination of accuracy + format + conciseness scores)
  • Hard caps will be maxed out — a full length score means the model will inevitably pad answers to the limit; use normalization or decay instead
  • Empty output / overlong output / timeout must be explicitly defined — default behavior is often a loophole; force 0 for empty output and deduct points after truncating overlong output

Choosing a reward representation

The reward representations are not mutually exclusive; they follow a layered progression from simplest to most complex: A. Inline (AgentOutput(reward_score=...)) → B. Standalone (RewardFunctionComponent) → C. Multi-dimensional with decorators (@reward_func + @sub_reward_func + @aggregate_func).

Representation

Use cases

Scoring complexity

Resource overhead

Observability granularity

Debuggability

When to avoid

A. Inline Rollout

For simple (≤10 lines) scoring logic that is synchronous with generation.

Low

Reuses the Rollout FC.

Only one metric: reward_score

Difficult, as it requires a Rollout redeploy.

When scoring requires independent scaling.

B. Standalone reward

For a single, well-defined scoring rule or for reusing a general-purpose scorer.

Medium

Standalone FC.

reward + reward_metrics

Medium (independent deployment).

For three or more dimensions.

C. Multi-dimensional decorator

For multi-dimensional scoring that requires independent weight adjustment for each dimension.

High

Standalone FC.

Independent metrics for each sub-dimension.

High (enables troubleshooting by dimension).

When dimensions are tightly coupled and cannot be separated.

Upgrade path: Start with A to get the pipeline running → switch to B at ≥1 dimension → switch to C at ≥3 dimensions. When migrating from A to B, keep the Rollout and move the scoring logic out. To migrate from B to C, break down the dimensions based on atomic determinability; do not split out a sub-dimension if it cannot be evaluated with a binary (0/1) outcome.

Multi-dimensional scoring: Three-layer weight design

Multi-dimensional reward weights are organized into three non-overlapping, independently adjustable layers:

  • Layer 1: Function-level RewardFunctionComponent.weight (between multiple reward functions)
  • Layer 2: Sub-dimension @sub_reward_func(sub_weight=...) (between sub-dimensions within a reward function)
  • Layer 3: Sub-metric reward_metric_weight={...} (for subdivisions within reward_metrics)

Final reward = Σ(Layer 1 × Layer 2 × Layer 3 normalized terms).

Multi-dimensional weight design template

Step

Actions

Output

Guideline

1. List dimensions

Break down the objective into 3–5 atomic, quantifiable dimensions.

A list of dimensions

Recommended: ≤ 5

2. Prioritize dimensions

Rank dimensions based on business needs, such as must-haves, bonus items, and style preferences.

A prioritized list of dimensions

-

3. Assign initial weights

Assign 0.5–0.7 to must-haves, 0.2–0.3 to bonus items, and 0.05–0.1 to style preferences.

A weight vector

All weights must sum to 1.

4. Scale alignment

Normalize each dimension to a [0, 1] range before applying its weight.

A normalization function

Prevents any single dimension from dominating the reward.

5. Sample validation

Compare the reward scores of 100 samples against their manual annotations.

correlation coefficient

Spearman ρ ≥ 0.6

Weight tuning methods

  • First, train a baseline model with uniform weights to identify which sub-metric is dominant. Then, examine the variance of the curve at trace/reward_metrics/{reward}/{sub}/avg. Increase the weight of dimensions that show low variance.
  • If the validation reward increases while a specific dimension's score on the training set drops sharply, that dimension's weight is likely too low and should be increased. If you observe reward hacking, temporarily reduce the weight of the relevant dimension instead of disabling it entirely.

Group Reward use cases

Group Reward vs Independent Reward

  • Independent Reward evaluates a single response based on its absolute quality (an absolute score); Group Reward assesses the relative quality among multiple responses to the same prompt (ranking or comparison).
  • GRPO/GSPO algorithms require group-relative advantage. Group Reward provides a direct ranking signal, which avoids the bias introduced by normalization.
  • For subjective tasks like writing or dialogue, relative comparison is more stable and consistent than absolute scoring.

Integration with GRPO/GSPO algorithms

  • GRPO/GSPO algorithms calculate group-relative advantage from multiple rollouts for the same prompt. In contrast, an Independent Reward provides an absolute score, which the framework must then normalize within the group (a process that can be skewed by outliers).
  • Group Reward directly outputs a within-group distribution, reducing score scale drift. The n_rollouts parameter controls the input size; a value between 4 and 16 is recommended.

Implementation notes

  • The output rewards list must match the length of the input agent_outputs list. Do not apply hard normalization (e.g., dividing by the maximum value) within the group, as this conflicts with normalization at the algorithm level.
  • Tied scores are permitted (e.g., three responses each scoring 1.0), but avoid assigning the same score to all responses, as this results in zero advantage. Assign failed responses a score of 0 to maintain a consistent list length; the framework uses the TaskStatus to identify and handle such cases.

Reward hacking: Identification and defense

Hacking is almost inevitable

  • Goodhart's Law: When a measure becomes a target, it ceases to be a good measure.
  • The probability of reward hacking increases as an RL (Reinforcement Learning) model converges and its reward grows. This is not a bug; the optimizer is just doing its job.
  • The goal of a defense strategy is to make reward hacking more difficult than finding the true solution, not to eliminate it entirely.

Hack pattern → detection signal → defense strategy

Hack pattern

Detection signals (metrics + behavior)

Defense strategy

length explosion / verbosity

trajectory/response_length increases monotonically while validation/data/reward/mean@1 plateaus or drops.

Use length normalization, add a length_penalty sub-reward, or apply hard truncation with max_length.

keyword stuffing

Reward increases, but manual spot-checks show repetitive keywords. Correlation between trace/reward_metrics/.../accuracy and human labels drops.

Use deduplication with regular expressions, replace keyword matching with an LLM judge, and introduce a semantic similarity threshold.

format hacking (markdown / emoji stuffing)

The output format is unusual, and the answer's relevance to the user's question decreases.

Normalize formatting as a preprocessing step. Score style as an independent dimension with a low weight.

refusal / empty output

Median reward is high, but trace/success_rate/... drops. The assistant message is empty.

Force a score of 0 for empty outputs, enforce a minimum length constraint, and add a "Did it answer the question?" scoring dimension.

repetition / repetitive output

High response length, high n-gram repetition rate, and a sharp drop in entropy.

Use regular expressions for repetition detection, apply a repetition_penalty, and monitor entropy.

judge collusion (in LLM-as-judge scenarios)

The LLM judge score is high but diverges from the rule-based score or human score.

Mix rule-based scores with judge scores, recalibrate the judge periodically, and rotate judges from different model families.

tool abuse / tool non-use

Anomalous trace/tool_call_count (spikes or drops) and a decreasing task success rate.

Penalize or reward tool call counts, and enforce constraints on required tool calls.

entropy collapse

actor/entropy approaches 0, and validation metrics stop changing.

Increase the KL divergence coefficient, use early stopping, or add an entropy bonus.

Four categories of defense strategies

  • reward shaping: Explicitly penalize hacking paths. This approach is direct but requires continuous patching.
  • Rule-based constraints: Hard constraints on length, repetition, or format. Inexpensive but rigid.
  • validation monitoring: Use metrics from a source other than the training reward as a sentinel. This is essential.
  • multiple reward combination: Combine rule-based and model-based rewards to create checks and balances.

Monitoring playbook

  • Monitor the upward trend of your North Star Metric, critic/rewards/mean. Use validation/data/reward/mean@1 as an anti-hacking sentinel; a divergence from the North Star Metric indicates reward hacking. Perform behavioral health checks by monitoring metrics like trajectory/response_length and actor/entropy.
  • Every N steps, spot-check trajectories on the console's "Trajectory" tab and compare the scores with human judgment.

Scoring stability

Bias control in LLM-as-a-judge

  • Position bias: The order of items in a pairwise comparison can affect the outcome. To mitigate this, run each comparison twice with the order swapped and accept only the results that are consistent across both evaluations.
  • Self-preference bias: A model judge may favor outputs from its own model family. To counter this, use a judge from a different model family (e.g., use a GPT model to judge outputs from a Qwen model).
  • Length bias: A model judge might favor longer outputs. To address this, explicitly instruct the judge in the prompt not to score based on length and apply length normalization.
  • Discrete rating scales (e.g., 1-5 or 1-7) are more stable than continuous ones (e.g., 1-100). Use an anchor example for calibration.

Rule and model scoring

  • Use rules as hard constraints for objective criteria such as format, length, required fields, and sensitive words. If a response fails these checks, short-circuit the process and assign a score of 0.
  • Use a model for soft evaluation of subjective qualities like fluency, relevance, and reasoning quality.
  • Combination: final = rule_pass × (w_rule × rule_score + w_model × model_score). To save costs and prevent hacking, do not send responses that fail rule-based checks to the model judge.

TaskStatus in scoring scenarios

  • A score of 0 (e.g., for a failed rule check or an incorrect answer) is an effective training signal. In this case, return SUCCESS + reward_score=0.
  • For transient errors (e.g., a judge timeout or network jitter), return FAILED to let the framework retry. Do not assign a score of 0 in these cases.
  • Overusing FAILED can cause a data distribution shift. If the framework repeatedly fails on a hard example and eventually discards it, the training set becomes progressively easier.

Scoring consistency self-checks

  • Prepare a gold set of 50-200 examples with human annotation. After modifying the reward logic, run it against the gold set and compare results using metrics like Spearman's rho or Cohen's kappa.
  • When upgrading or switching your model judge, compare the new scores against the old ones to detect silent drift. Periodically send the N highest- and lowest-scoring training samples for human review.

Reward function checklist (10 questions)

Review this checklist before you submit a task with your new reward function:

  1. Can the scoring logic be explained in a single sentence?
  2. Is there a ground truth, and is it machine-determinable?
  3. What is the lowest-cost hack path, and have you defended against it?
  4. If your reward is multi-dimensional, can you annotate each dimension manually and independently?
  5. Have you normalized the sum of the weights and aligned the scales?
  6. Have you defined boundary conditions (e.g., empty output, overlong output, timeout)?
  7. Is the validation method independent of the training signal?
  8. Is the scoring time less than one-third of the rollout time?
  9. Is the reward_metrics field name stable? Renaming it will break the metric curves.
  10. Do you have a gold set for regression?

Reward design anti-patterns

Anti-pattern

Consequence

Defining the reward as "the longer, the better"

Inevitable reward hacking (length explosion)

Using single-dimension scoring for a highly subjective dimension

Noise overwhelms the signal.

Setting the weight for every dimension to 1.0

Equivalent to using no weights, resulting in inconsistent scales.

Using the same prompt and model for both the training judge and the validation judge

A form of self-deception that leads to overly optimistic results.

Returning SUCCESS with a score of 0 for temporary network errors

Data distribution contamination (hard samples are mislabeled as incorrect answers).

Observability configuration

NoteThis section has been moved out. The complete content on Tracing decorators, custom metrics, dependency configuration, the ENABLE_TRAJECTORY switch, and more is now consolidated in Observability Configuration (Console and observability coverage).

Function testing

Function registration

Upload your function code and register it as cloud function components:

client = AgenticRL()
(rollout_entity_ids,
 reward_entity_ids,
 group_reward_entity_ids,
 rollout_instance_ids,
 reward_instance_ids,
 group_reward_instance_ids) = await client.register_functions(
    functions=[
        # rollout-only:演示版,覆盖 trace_client/trace_tool 全分支
        AgenticRLFunctionComponent(
            type=FunctionType.ROLLOUT,
            fcmodel=FunctionComponentModel(
                classpath="functions.rollout.rollout_only.DemoRolloutProcessor")),
        # reward
        AgenticRLFunctionComponent(
            type=FunctionType.REWARD,
            fcmodel=FunctionComponentModel(
                classpath="functions.reward.reward.DemoRewardProcessor")),
        # reward-decorator:文件路径:类名 形式
        AgenticRLFunctionComponent(
            type=FunctionType.REWARD,
            fcmodel=FunctionComponentModel(
                classpath="functions/reward/reward_decorator.py:SafetyProcessor")),
        # group-reward
        AgenticRLFunctionComponent(
            type=FunctionType.GROUP_REWARD,
            fcmodel=FunctionComponentModel(
                classpath="functions.reward.group_reward.DemoGroupRewardProcessor")),
    ],
    lazy_load=False  # False = 立即获取 instance_id,用于测试
)

Equivalent CLI:

dashscope rl register_functions \
  --rollout-classpaths "functions.rollout.rollout_only.DemoRolloutProcessor" \
  --reward-classpaths "functions.reward.reward.DemoRewardProcessor" \
  --reward-classpaths "functions/reward/reward_decorator.py:SafetyProcessor" \
  --group-reward-classpaths "functions.reward.group_reward.DemoGroupRewardProcessor" \
  --no-lazy-load \
  --output-format json

Remote testing

Use test_functions to send test data to registered function instances:

## 测试 Rollout
result = await AgenticRL.test_functions(
    instance_id=rollout_instance_ids[0],
    functype=FunctionType.ROLLOUT,
    input_data=rollout_input
)

## 测试 Reward
result = await AgenticRL.test_functions(
    instance_id=reward_instance_ids[0],
    functype=FunctionType.REWARD,
    input_data=reward_input
)

## 测试 Reward 装饰器版(第二个 reward 实例,对应 refusal 分支 reward_score=0.85)
result = await AgenticRL.test_functions(
    instance_id=reward_instance_ids[1],
    functype=FunctionType.REWARD,
    input_data=reward_decorator_input
)

## 测试 Group Reward
result = await AgenticRL.test_functions(
    instance_id=group_reward_instance_ids[0],
    functype=FunctionType.GROUP_REWARD,
    input_data=group_reward_input
)

After testing, you can pull function instance logs to troubleshoot the execution process (test_functions also supports pull_logs=True to fetch the logs along with the returned result):

logs = await AgenticRL.query_all_function_instance_logs(
    function_instance_id=rollout_instance_ids[0],
    start_time=int(time.time()) - 3600,  # 查询近 1 小时
    end_time=int(time.time()),
)

Equivalent CLI:

dashscope rl test_functions "ro-ins-xxx" --type rollout --input ./resources/rollout_input.json
dashscope rl test_functions "rw-ins-xxx" --type reward --input ./resources/reward_input.json
dashscope rl test_functions "rw-ins-yyy" --type reward --input ./resources/reward_decorator_input.json
dashscope rl test_functions "grw-ins-xxx" --type group_reward --input ./resources/group_reward_input.json

Test input format

Rollout test input (rollout_input.json):

{
  "func_type": "rollout",
  "messages": [
    {"role": "user", "content": "Hello, please introduce yourself"}
  ],
  "tools": null,
  "ground_truth": "",
  "rollout_extra": {
    "rollout_id": "ro-ins-e8e92e94-b5c3-4546-b7b1-9832b8b948f2",
    "training_state": {"task_id": "task-001"}
  },
  "sampling_params": {
    "temperature": 0.7,
    "max_tokens": 1024,
    "max_turns": 25,
    "timeout": 60.0
  },
  "model_resource": {
    "model_name": "qwen3.5-9b",
    "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
    "api_key": "your-api-key-here"
  },
  "request_metadata": {
    "protocol": "openai",
    "system_prompt": "",
    "job_id": "dummy-job-id",
    "sample_id": "dummy-sample-id",
    "rollout_id": "dummy-rollout-id",
    "attempt_id": "dummy-attempt-id"
  },
  "additional_field": "any extra data you want"
}

Reward test input (reward_input.json):

{
  "func_type": "reward",
  "ground_truth": "0.5",
  "request_metadata": null,
  "agent_output": {
    "messages": [{"role": "user", "content": "Test question"}],
    "rollout_extra": {
      "content": "This is a test content that exceeds 50 characters in length and should receive a reward score of 0.5"
    },
    "rollout_metrics": {"accuracy": 0.95},
    "reward_score": null
  }
}

Reward decorator test input (reward_decorator_input.json, corresponding to the refusal branch, with reward_score=0.85 after aggregation):

{
  "func_type": "reward",
  "ground_truth": "0.5",
  "agent_output": {
    "messages": [{"role": "user", "content": "I cannot help you with that"}]
  }
}

Group Reward test input (group_reward_input.json, where agent_outputs is an array of the same sample):

{
  "func_type": "group_reward",
  "ground_truth": "0.5",
  "request_metadata": null,
  "agent_outputs": [{
    "messages": [{"role": "user", "content": "Test question"}],
    "rollout_extra": {
      "content": "This is a test content that exceeds 50 characters in length and should receive a reward score of 0.5"
    },
    "rollout_metrics": {"accuracy": 0.95},
    "reward_score": null
  }]
}

Local testing

The scripts/ directory in the project provides local testing scripts:

./scripts/start_local_rollout.sh   # 启动本地 Rollout 服务
./scripts/query_local_rollout.sh   # 发送测试请求

Next steps