Reinforcement learning pre-submission self-check

Updated at:

A layered self-check workflow before submitting an RL training job: from free local validation to low-cost pipeline smoke tests, catching data, code, configuration, and function errors before any training costs are incurred.

RL training is billed by Training units hours (for example, pay-as-you-go with 24 Type IV instances costs approximately 984 CNY/hour), and errors that only surface after the job starts are the most expensive. This page provides a pre-submission self-check workflow with escalating costs: each step costs far less than the steps that follow and catches errors that previous steps cannot detect.

Description of each step:

Step

Cost level

Errors caught

Prerequisites

Check the data format

CNY 0 (Local)

JSONL format errors, missing fields, insufficient data volume

Python 3.10+ local environment

Validate functions locally

CNY 0 (Local)

Missing dependencies, classpath errors, data model construction errors, reward logic errors

Local environment + project directory

Test functions remotely

A few CNY (FC instance cold starts and a small number of invocations)

Cloud container environment errors (dependency installation failures, packaging issues, runtime exceptions)

API Key + DashScope SDK

Run a submission smoke test

From approximately CNY 16

Submission configuration errors, rejected hyperparameters, invalid resource configuration

API Key + MTU with pay-as-you-go enabled

Run a streamlined first training

From several hundred CNY (depending on when you stop it; over CNY 1,000 for a complete run)

Issues that only real training reveals (slow sampling, errors on the first step)

Same as a formal training run

If you have already gone through the full self-check workflow and only make partial changes afterwards, there is no need to rerun the entire workflow:

  • Replace the data (a new dataset or expanded data volume) → restart from "Check data format"
  • Modify the function code → restart from "Local function validation"
  • Only adjust hyperparameters or resource configuration → restart from "Submit a smoke test" (about 16 CNY to verify that the server accepts the new configuration)

Check data format (Free)

Run the following script in the project root directory to validate the training/validation data line by line (replace the files paths with your actual datasets):

import json, sys
from pathlib import Path

errors = []
files = ["data/calc_train_min.jsonl", "data/calc_validation_min.jsonl"]
batch_size = 64  # 与提交时的 hyper_parameters.batch_size 一致

for name in files:
    lines = Path(name).read_text(encoding="utf-8").strip().splitlines()
    for i, ln in enumerate(lines, 1):
        try:
            obj = json.loads(ln)
            if "messages" not in obj:
                errors.append(f"{name}:{i} 缺 messages 字段")
            if "rollout_extra" not in obj:
                errors.append(f"{name}:{i} 缺 rollout_extra 字段")
        except json.JSONDecodeError as e:
            errors.append(f"{name}:{i} JSON 解析失败: {e}")
    if len(lines) <= batch_size:
        print(f"[WARN] {name} 仅 {len(lines)} 行,数据量须大于 batch_size")

print(f"共检查 {len(files)} 个文件,错误 {len(errors)} 项")
for e in errors[:10]:
    print("[FAIL]", e)
sys.exit(1 if errors else 0)

Pass criteria: Errors: 0 items, no WARN (or a WARN data volume that matches expectations); also confirm two points — the validation set and the training set are independent of each other (different sources, no overlap); the keys in rollout_extra match the keys read by the Reward scoring logic.

The relationship between the number of data rows and batch_size depends on the scenario:

  • Submit a smoke test: a small amount of data is enough (for example, 10 rows), and reduce batch_size at the same time — the server requires batch_size to be within the range 8 to 2048, so setting it to 8 for smoke testing is enough to submit.
  • Formal training: the training data volume must be greater than batch_size; the larger the batch_size, the more stable the training. Set the value according to the Training Configuration — required parameters table.

For the meanings of data format fields and data volume recommendations, see Reinforcement learning training overview — data format.

Local function validation (Free)

Components are importable and constructible

classpath is a package path.module.class name string (for example, functions.reward.my_reward.MyRewardProcessor); the value filled in for local self-checks, function registration, and training submission must be the same. Replace the class in the following example with your custom function class, then run it in the project root directory:

import sys
sys.path.insert(0, ".")  # 在项目根目录执行

try:
    # 1) 数据模型可构造(AgentOutput 四字段)
    from dashscope.finetune.reinforcement.component.data.base_data_model import AgentOutput
    AgentOutput(messages=[{"role": "user", "content": "hi"}],
                reward_score=0.95, rollout_extra={"answer": "42"},
                rollout_metrics={"latency": 1.2})

    # 2) 自定义函数类可导入(与注册/提交时的 classpath 为同一字符串)
    from functions.rollout.my_rollout import MyRolloutProcessor
    from functions.reward.my_reward import MyRewardProcessor

    print("导入自检 OK")
except Exception as e:
    print(f"[FAIL] {type(e).__name__}: {e}")
    sys.exit(1)

Pass criteria: Import self-check OK. A common FAIL is ModuleNotFoundError (the dependency is not listed in requirements.txt or is not installed locally) — the cloud container reports the same error, so fix it locally first.

NoteThere is a common mistake when using TaskStatus in scoring functions: a score of 0 should return SUCCESS, and only transient errors (timeouts, network jitter) should return FAILED. For more information, see Development guide — TaskStatus decisions in scoring scenarios.

Design minimal test cases

Before live-testing the scoring locally, first design 3 test inputs for your custom function whose expected score can be worked out manually — the expected score is determined by your custom scoring logic:

Test case

Construction method

Expected score

Full-score sample

Put a response in messages that is guaranteed to hit the scoring conditions (contains the correct answer, meets the format), and fill ground_truth with the corresponding ground truth

The full score of the scoring logic (for example, 1.0)

Zero-score sample

Put a response in messages that is guaranteed to miss (wrong answer, format mismatch)

0.0

Boundary sample

The forms most likely to be misjudged: empty messages, extremely long responses, partially matching answers, different casing

Follow the defined boundary behavior (if a boundary behavior is undefined, define it in the scoring logic first — undefined boundaries are a common source of errors during training)

Save the three test inputs separately (for example, my_reward_test_full.json, my_reward_test_zero.json, my_reward_test_edge.json). Reward input template:

{
  "func_type": "reward",
  "ground_truth": "42",
  "agent_output": {
    "messages": [
      {"role": "user", "content": "计算 6 × 7"},
      {"role": "assistant", "content": "### ANSWER: 42 ###"}
    ]
  }
}

For the structure of Rollout test inputs (fields such as messages, model_resource, and sampling_params), see Development guide — test input format.

At the scoring logic design level, complete two self-checks before submission: the 10 self-check questions before writing a Reward; and prepare a gold set of 50-200 manually annotated items for the scoring consistency self-check. Also keep the field names in reward_metrics stable — renaming them breaks the continuity of the metric curves in the console.

Live testing against a local service

Without deploying to the cloud, start the custom function as a local service and send the above test inputs one by one:

# 终端 1:启动本地函数服务(bash,PROCESSOR_CLASS 换为自定义类)
FUNC_TYPE=reward PROCESSOR_CLASS=functions.reward.my_reward.MyRewardProcessor \
PYTHONPATH=. python -m dashscope.finetune.reinforcement.component.server.server

# 终端 2:逐个发送三份测试输入
curl -X POST http://localhost:8000/api/v1 \
  -H "Content-Type: application/json" \
  -d @my_reward_test_full.json

Windows PowerShell equivalent command:

# 终端 1
$env:FUNC_TYPE="reward"
$env:PROCESSOR_CLASS="functions.reward.my_reward.MyRewardProcessor"
$env:PYTHONPATH="."
python -m dashscope.finetune.reinforcement.component.server.server

# 终端 2(PowerShell 下 curl 为 Invoke-WebRequest 别名,@file 语法不可用)
Invoke-RestMethod -Method Post -Uri http://localhost:8000/api/v1 `
  -ContentType "application/json" -InFile my_reward_test_full.json

Example expected return (field positions are subject to the actual return):

{"reward": {"reward_score": 1.0, "reward_metrics": {"accuracy": 1.0}},
 "status": "success", "error": null, "error_code": null}

Pass criteria: The returned status of all three inputs is success, and each reward_score in reward matches the expected score one by one. Any mismatch indicates a defect in the scoring logic; fix it locally until they are consistent before proceeding to remote testing.

NoteFor live testing of the Rollout function, replace model_resource in the test inputs with an available inference endpoint and API Key: you must use an API Key that supports inference calls (created on the API-KEY page of the Bailian console, with the sk- prefix). If the credentials used for training submission do not support inference calls, a 401 (Incorrect API key) is returned here. During training there is no need to configure the inference endpoint yourself—the endpoint and Key are injected automatically at runtime by the training framework (see RolloutInput key fields).

Remote function testing

After local tests pass, register the function to the cloud and execute it once in a real FC container—only this step can catch differences between the local and container environments (dependency installation, packaging paths, runtime behavior).

NoteWindows system limitation: The current SDK version has a compatibility defect when packaging functions on Windows—registration can succeed but test_functions fails with Precondition Failed (the function instance exits before startup). We recommend running this step and subsequent submission operations in WSL or a Linux environment; this issue has been reported to the SDK maintainers.

import asyncio, json
from dashscope.finetune.reinforcement import (
    AgenticRLFunctionComponent, FunctionType, FunctionComponentModel)
from dashscope.finetune.agentic_rl import AgenticRL

async def main():
    client = AgenticRL()
    # 注册自定义函数(classpath 与本地导入自检为同一字符串)
    ids = await client.register_functions(
        functions=[AgenticRLFunctionComponent(
            type=FunctionType.REWARD,
            fcmodel=FunctionComponentModel(
                classpath="functions.reward.my_reward.MyRewardProcessor"))],
        lazy_load=False)   # 立即拉起实例用于测试
    # register_functions 返回 6 元组,依次为:
    # (rollout_entity_ids, reward_entity_ids, group_reward_entity_ids,
    #  rollout_instance_ids, reward_instance_ids, group_reward_instance_ids)
    # 未注册的类型为空列表
    reward_entity_ids, reward_instance_ids = ids[1], ids[4]

    # 逐个发送本地验证的三份测试用例(input_data 接受 dict)
    for case in ["my_reward_test_full.json", "my_reward_test_zero.json",
                 "my_reward_test_edge.json"]:
        with open(case, "r", encoding="utf-8") as f:
            input_data = json.load(f)
        result = await AgenticRL.test_functions(
            instance_id=reward_instance_ids[0],
            functype=FunctionType.REWARD,
            input_data=input_data,
        )
        print(case, result)

asyncio.run(main())

Pass criteria: The returns of all three test cases match the local validation results item by item. If cloud results differ from local ones, check requirements.txt first (the container installs dependencies based on it); instance logs can be pulled via query_all_function_instance_logs (for usage, see Reinforcement learning development guide — remote testing)—if the function instance fails during the startup phase, logs may not have been generated yet (an empty list is returned); in that case, refer to the error message in the test_functions return result.

Note

  • Remote testing of the Rollout function calls the model_resource in the test inputs as a real inference endpoint; just fill in a valid base model and API Key. CLI equivalent command: dashscope rl test_functions "rw-ins-xxx" --type reward --input my_reward_test_full.json.
  • Each register_functions call and each run() submission registers a new function entity that accumulates in your account (the local project root also generates start.sh and temporary packaging files, which can be ignored or deleted). Once the function is stable, you can use submit_job with entity_id to reuse the registration result and avoid duplicate registration (see Training configuration — step-by-step submission).

Submission smoke test

Make a real submission with parameters (functions, data, hyperparameters, resources) exactly identical to the formal submission, and cancel it immediately once the server accepts all configurations. Billing basis: taking 24 pay-as-you-go instances as an example, about CNY 984/hour (the minimum pay-as-you-go billing granularity is 1 minute, i.e. about CNY 16/minute); keeping submission-to-cancel within 1 minute costs about CNY 16 and up, and every additional minute of delay before canceling adds about CNY 16.

import asyncio
from dashscope.finetune.agentic_rl import AgenticRL

async def main():
    client = AgenticRL()
    # run 为 async;参数为正式提交的完整参数(函数/数据/超参/资源)
    result = await client.run(
        model="your-model",
        functions=[...], training_datasets=[...], validation_datasets=[...],
        hyper_parameters={...}, resources={...})

    # 先确认提交结果再取 job_id(提交被拒时 output 为空,直接取值会报错)
    if result.status_code != 200:
        print(f"提交被拒: {result.message}")   # 服务端拒绝原因(如 batch_size 超范围)
        return

    # ---- 拿到 job_id 后立即执行以下两行(get/cancel 为同步方法,无需 await)----
    job_id = result.output.job_id
    g = AgenticRL.get(job_id=job_id)      # 回显任务详情,核对超参与资源配置
    c = AgenticRL.cancel(job_id=job_id)   # 立即取消,控制费用

asyncio.run(main())

Pass criteria:

  • run returns normally (status_code is 200) and job_id is non-empty—the full chain of function registration, data upload, and parameter validation is accepted by the server
  • In the hyper_parameters echoed by get, fields not explicitly filled in are automatically populated with default values by the server (such as enable_thinking, max_off_policy_strategy, etc.)—check that the populated values do not conflict with expectations
  • Verify two resource configurations: mtu_capacity = nnodes_rollout × n_gpus_rollout + nnodes_training × n_gpus_training (constraint description); the function Runtime's memory_size ≥ 4096 and disk_size ≥ 512 (Runtime resource configuration)

WarningExecute cancel immediately after obtaining the job_id—uncanceled jobs queue up to enter RUNNING and are billed by training unit hours. If the script throws an error before cancel (for example, misusing await on a synchronous method), the job may have been submitted but the script lost the job_id; in that case, use AgenticRL.list() to retrieve the latest job and cancel it.

Lean first training run

Before the formal full training, make a real run with minimal data + 1 epoch: confirm that the first step can produce trajectories and metrics, and that the Reward has gradients (not constantly 0 or 1).

Duration expectations (measured with demo lean data of 128 entries, batch_size=64, and 24 pay-as-you-go instances):

  • About 1 minute from submission to entering RUNNING (when resource queuing is normal)
  • Producing trajectories and metrics for the first step usually takes more than 30 minutes (the first step must complete all Rollout sampling)
  • About 1 hour 45 minutes for a complete run through to SUCCEEDED

Cost basis (billed by training unit hours; for unit prices, see Training unit price):

  • With "first step metrics appear and are normal" as the stopping point, the cost is typically on the order of hundreds of CNY; a complete run costs about 1,000+ CNY
  • Jobs canceled before entering training are not billed (measured usage is 0); after entering training, billing follows actual usage, and the usage field in the job details echoes the cumulative usage (measured reference: a job that fully ran in 1h43m had usage=4384)
  • Full training cost estimate: Number of instances × estimated duration × hourly unit price—the estimated duration can be roughly calculated as "data volume / batch_size × time per step × n_epochs", or scaled proportionally from the measured duration of the lean first run

For the criteria for judging whether the first step metrics are normal (health indicators such as critic/rewards/mean and actor/entropy), see Observability configuration — key metrics quick reference. For concrete practices (lean datasets, n_epochs=1), see Reinforcement learning training overview — getting started tips.

Common error quick reference

This table covers high-frequency errors from data checking through submission. For attributing errors after the job has entered training (first step crash, sampling timeout, garbled output, etc.), see Observability configuration — troubleshooting decision tree.

High-frequency errors and localization for each self-check layer:

Stage

Error message

Cause analysis

Resolution

Submit

ValidationError: extra inputs are not permitted (e.g. weight)

This parameter does not belong to the current component type (weight is a field of RewardFunctionComponent, not on AgenticRLFunctionComponent)

Verify the component type and field ownership; step-by-step submission does not include weight

Submit

[3005] Job submission failed: TypeError(unexpected keyword argument)

Called a non-existent method parameter (for example, run(lazy_load=...) or submit_job(training_file_ids=...))

For datasets, pass the full set of three parameters with datasets=[Dataset(type=DatasetsType.VALIDATION, data_source_type=DataSourceType.FILE_ID, file_id=...)]; for the remaining parameters, follow the examples on this page and the submission methods in the configuration guide

Submit

[400] InvalidParameter:batch_size need to be in range [8, 2048]

batch_size is outside the range allowed by the server (during a smoke test the data volume is small, so batch_size must be adjusted to 8)

Adjust batch_size to 8~2048; in smoke-test scenarios, pair it with a small amount of data (see Check data format)

Submit

[1002] I/O error: Failed to load from dict

YAML dialect error (unsupported top-level key or field type)

Use the job.yaml template structure from the configuration guide

Submit

TypeError: object FineTune can't be used in 'await' expression

Misusing await on synchronous methods (submit_job / get / cancel / list / logs)

Remove await; note that run/register_functions/upload_datasets/test_functions are async, while job management methods are synchronous

Remote testing

ModuleNotFoundError (in instance logs)

requirements.txt is missing dependencies, or the package structure cannot be resolved inside the container

Complete requirements.txt and go back to local validation to reproduce

Remote testing

Precondition Failed (returned by test_functions), or instance logs contain bash\r: No such file or directory, or a backslash module path appears in start.sh

Windows packaging compatibility issue (the CLI and SDK share the same packaging pipeline, so the issue reproduces on both sides; logs are empty when the instance exits immediately after startup)

Run remote testing and submission in a WSL or Linux environment; this issue has been reported to the DashScope SDK maintainers

Next steps