Reinforcement learning observability and metrics

@observe_processor: Trace the processor entry point

Apply this decorator to the process() method to create the top-level ENTRY span. The SDK automatically identifies the type based on the parent class. Inheriting from AbstractRolloutProcessor creates a ROLLOUT-type span, and inheriting from AbstractRewardProcessor creates a REWARD-type span. The decorator automatically records the input, output, duration, and success/failure status of each call.

For Rollout (functions/rollout/rollout.py):

from dashscope.finetune.reinforcement.component.observability import (
    observe_processor
)

class CalcXRolloutProcessor(AbstractRolloutProcessor):
    @observe_processor  # Span type = ROLLOUT
    async def process(self, input: RolloutInput) -> RolloutOutput:
        await self._async_setup()
        return await self._async_process(input)

For Reward (functions/reward/reward.py):

class DemoRewardProcessor(AbstractRewardProcessor):
    @observe_processor  # Span type = REWARD
    async def process(self, input: RewardInput) -> RewardOutput:
        score = await evaluate(content, input.ground_truth)
        return RewardOutput(
            reward=Reward(
                reward_score=score,
                reward_metrics={"test1": 0.5, "test2": 0.3}
            ),
            status=TaskStatus.SUCCESS,
        )

trace_client(): Trace an LLM client

Call this function in your initialization method to wrap an LLM client instance. All subsequent LLM requests from this client will automatically generate an LLM span that records the model name, request content, token usage, latency, and more.

Supported client types (auto-detected via duck typing):

• OpenAI clients (AsyncOpenAI / OpenAI)

• OpenAI completions resources (.chat.completions)

• LangChain classes like ChatOpenAI (via .client / .async_client)

• DashScope Generation class (pass the class itself, not an instance)

Example (functions/rollout/rollout.py):

from langchain_openai import ChatOpenAI
from dashscope.finetune.reinforcement.component.observability import (
    trace_client
)

class CalcXRolloutProcessor(AbstractRolloutProcessor):
    def _build_llm(self, input: RolloutInput) -> ChatOpenAI:
        llm = ChatOpenAI(
            model=input.model_resource.model_name,
            openai_api_key=api_key,
            openai_api_base=input.model_resource.base_url,
            ...
        )
        trace_client(llm)  # Wrap the client to automatically trace all LLM calls
        return llm

trace_tool(): Trace tool calls

After retrieving tool instances, call this function to wrap them. Each subsequent tool call will generate a TOOL span, recording the tool name, parameters, return value, and latency.

Supported input formats:

• A single LangChain BaseTool

• A list, tuple, or dictionary (iterated over automatically)

• A LangGraph ToolNode (automatically expands .tools_by_name)

• MCP tools (auto-detected, with provider set to "mcp")

Example (functions/rollout/rollout.py):

from dashscope.finetune.reinforcement.component.observability import (
    trace_tool
)

class CalcXRolloutProcessor(AbstractRolloutProcessor):
    async def _init_resources_async(self):
        client = MultiServerMCPClient({
            "calculator": {"url": "http://localhost:10086/sse"}
        })
        tools = await client.get_tools()
        trace_tool(tools)  # Must be called after get_tools()

@observe_llm: Custom LLM function

If trace_client() cannot automatically detect your LLM client, use this decorator to manually mark a function as an LLM call.

Signature requirement: The function must include the model and messages keyword arguments after a *.

from dashscope.finetune.reinforcement.component.observability import (
    observe_llm
)

@observe_llm  # Mark as an LLM span
async def call_custom_llm(*, model: str, messages: list, **kwargs):
    # Custom LLM call logic
    ...

@observe_tool: Custom tool function

If trace_tool() cannot automatically detect your tool (for example, a standard Python function), use this decorator to mark it manually. You can customize the span name using the name parameter.

from dashscope.finetune.reinforcement.component.observability import (
    observe_tool
)

@observe_tool(name="response_scorer")  # Mark as a TOOL span
def score_response(*, messages: list) -> float:
    # Custom scoring logic
    ...

Data flow and cost drivers

• Data flow: Function code (core decorators) → OpenTelemetry SDK → ARMS → Trace/Metrics tabs in the console

• Cost drivers: ARMS span storage (pay-as-you-go), minor CPU and network overhead on the function side, and a slight increase in training latency.

Development vs. large-scale training strategy

To disable tracing, set runtime.env = {"ENABLE_TRAJECTORY": "false"}.

Cost management for custom metrics

• The number and cardinality of metrics in reward_metrics and rollout_metrics affect ARMS storage costs.

• Avoid using high-cardinality fields, such as user_id or request_id, as metric keys.

• Limit key metrics to 10 or fewer, and remove temporary debugging metrics.

• Sub-dimensional metrics from @sub_reward_func are aggregated into the reward_metrics of the corresponding Reward function. This aggregation helps keep the data volume manageable.

Interpreting metrics in layers

• Essentials (8-10 metrics for daily health checks): The north star metric, three core metrics, truncation rate, and single-step latency (see the Key Metric Quick Reference Table for health criteria).

• Scenario-specific (by use case): For agent scenarios, monitor trace/num_llm_calls and trajectory/num_turns. For long-text scenarios, review the length-related metrics.

• Troubleshooting drill-down: timing sub-items / fully_async queue / critic distribution max/min

To learn when to use each of the five tabs in the console, see the tab quick reference table in the "View Results in the Console" section above.

P1: Non-convergence and P2: oscillation

• P1 primary signal: A long plateau in critic/rewards/mean; actor/ppo_kl ≈ 0; and actor/grad_norm is extremely small.

• P1 root cause: The learning rate is too small, the reward function consistently returns the same value, or there is insufficient data.

• P2 primary signal: Major oscillation in rewards/mean, a spike in grad_norm, and pg_clipfrac > 5%.

• P2 root cause: The learning rate is too high, the batch size is too small, or the advantage variance is not normalized.

P3: KL explosion and P4: sudden entropy drop (mode collapse)

• P3 primary signal: actor/ppo_kl is consistently > 0.1, pg_clipfrac increases concurrently, and the trajectory shows garbled or repetitive output.

• P3 recommended action: Increase kl_loss_coef (e.g., from 0.001 to 0.01), decrease the learning rate, and decrease ppo_mini_batch_size.

• P4 primary signal: actor/entropy drops to nearly 0 within a few dozen steps, rewards/mean shows a false increase, and sample outputs are repetitive and template-like.

• P4 root cause: A single mode is excessively rewarded, the KL constraint is too weak, or the temperature is too low.

P5: Reward hacking and P6: validation set collapse

• P5 primary signal: The training reward increases while the validation reward stagnates or decreases; response_length_non_aborted/mean increases linearly and rapidly; and a specific sub-dimension of reward_metrics shows exclusive gains.

• P5 attribution path: Analyze the abnormal metric, then sample the trajectory. Check if the assistant is padding its output, copying the prompt, or using tools to exploit loopholes in the reward function (for details, see "Reinforcement Learning Development Guide" § Identifying and preventing reward hacking).

• P6 primary signal: validation/data/reward/mean@1 shows a downward inflection point while the training reward continues to increase.

• P6 recommended action: Apply early stopping (select an earlier checkpoint from the Outputs tab), increase the KL constraint, and expand the validation set.

P7: Slow training, P8: garbled output, and P9: high truncation rate

• P7 primary signal: A sudden increase in timing/s/step.

• P7 drill-down: Investigate rollout/agent_loop_latency and update_actor. Possible root causes include an increase in response length, delays in FC auto-scaling (check the fully_async queue), or high tool latency.

• P8 primary signal: The trajectory contains garbled or non-natural language output. This is often caused by a KL explosion (P3). Address this issue using the P3 solution.

• P9 primary signal: A high value for trajectory/response_length/clip_ratio or prompt_length/clip_ratio.

• P9 recommended action: Increase max_length or shorten the prompt. Also, check if the reward function inadvertently encourages longer outputs.

Using Step, Sample, and Trajectory

Standard workflow: Spot an inflection point on the Metrics tab → Use Step to pinpoint the failing training segment (trend inflection) → Use Sample to find which samples are skewing the mean → Use Trajectory to examine the reasoning chain or tool calling details → Return to the code to modify the reward function or data.

Tool calling analysis and tracing span tree

• Tool calling analysis: Tool name, parameters, return value, and duration—essential for troubleshooting agent tool failures.

• The Tracing subtab provides a span tree for each trajectory (ENTRY:ROLLOUT → LLM → TOOL → REWARD).

• Typical use case: If the reward suddenly drops, check if a tool has failed or the MCP server is unavailable.

• Note: The server-side @observe_tool decorator does not affect the client. You must use trace_tool(tools) on the client side.

Custom metrics for attribution (rollout_metrics and reward_metrics)

• System metrics tell you that an error occurred, but custom metrics pinpoint which dimension is failing.

• Recommended naming convention: Name your metrics according to business sub-dimensions, such as accuracy, format_score, tool_success_rate, or answer_length.

• Handling multiple reward functions: Distinguish them using RewardFunctionComponent(name=...) and adjust their weights using reward_metric_weight.

• Anti-pattern: Avoid stuffing log or debugging information into metrics, as this can cause a cardinality explosion.

When to look beyond the console

• If you suspect an issue with the reward implementation, reproduce it locally using test_functions.

• If you suspect data issues, sample the JSONL file and inspect rollout_extra.

• If you suspect incorrect hyperparameters, review the submission script and check it against the "Reinforcement Learning Training Configuration — Submission and Configuration" guide.

Standard troubleshooting workflow

1. Step 1: Details tab Confirm the job status and the time of failure.

2. Step 2: Logs tab Inspect the last 100-500 lines of the log (SDK: AgenticRL.logs(job_id, lines=100) / CLI: dashscope rl logs --lines 100).

3. Step 3: Identify the error layer:

   • User function error (exceptions in rollout/reward functions) → Reproduce locally with test_functions → Fix the code → Re-register and run.

   • Framework error (OOM, insufficient resources, network issues) → Check the fully_async queue and adjust concurrency / capacity.

   • Data error (JSONL parsing failure) → Validate the format of individual records and the rollout_extra field.

Common error patterns