Evaluator-AgentLoop(AgentLoop)-阿里云帮助中心

During agent development, traditional evaluation methods (such as regex matching, keyword retrieval, and calculating Rouge/BLEU scores) can no longer effectively measure complex agents with reasoning and tool-use capabilities. The AgentLoop Evaluator is an AI-native quality measurement tool that supports multiple evaluation methods. The evaluator family includes the LLM-reasoning Agent Judge and the Code evaluator, which delegates judging logic to user-authored Python or JavaScript code for deterministic, reproducible results.

Advantages

Compared to traditional evaluation solutions, the Agent Judge has true agent attributes:

Autonomous reasoning: Understands complex business context and captures semantic-level nuances.
Capability mounting: Equipped with "hands and feet" (Skills & MCP), it can execute code to verify results or retrieve external facts.
Black-box transparency: Through chain-of-thought (CoT) traces, it fully reconstructs the judge's scoring logic.
Structured output: Produces standardized data to directly drive automated A/B testing and dataset iteration.

Built-in Evaluators

AgentLoop provides a set of deeply optimized evaluators for the core quality dimensions in agent development, helping you quickly establish a quality safeguard.

Quality and Factuality Dimension

Evaluator	Description
Agent Correctness	Evaluates whether the response is objectively factual or logically correct in business terms
Agent Hallucination Detection	Specifically identifies model-fabricated information and intercepts false content
Agent Response Evidence Support	Designed for RAG scenarios, verifies whether responses are rigorously grounded in retrieved context

Task and Action Dimension

Evaluator	Description
Agent Task Completion	Focuses on the final state, evaluating whether the user's original goal was fully resolved
Agent Tool Selection Rationality	Audits the decision process, evaluating whether tool selection is optimal and parameter passing is accurate

Efficiency and Cost Dimension

Evaluator	Description
Agent Execution Step Efficiency	Identifies redundant steps by auditing reasoning traces (Trace), helping reduce Token costs and shorten response time while maintaining quality

Create Custom Evaluator

When built-in evaluators cannot meet specific business scenarios, you can build a custom evaluator by following the steps below.

Step 1: Basic Information

Configuration Item	Description
Name and Identifier	Required. Defines the globally unique identifier for the evaluator, used for API calls and system indexing. The value cannot be modified after creation.
Evaluator Type	Required. Select Agent or Code. The value cannot be modified after creation. Agent continues with prompt and capability configuration; Code opens the Evaluation Code tab, where the judging logic is written as executable code. For the full procedure, see the "Code Evaluator" document in the same directory.
Display Name (Optional)	Optional. A user-friendly display name (e.g., "E-commerce Compliance Screener"). If left blank, the system defaults to displaying the evaluator name.
Description	Detailed explanation of the evaluator's purpose, applicable scenarios, and judging preferences. A thorough description helps team members reuse this asset.
Tags	Enter a tag name and press Enter to add. Tags allow you to categorize evaluators by dimension (e.g., "security", "RAG", "Finance Dept.") for instant retrieval.

Step 2: Configure Prompt and Capabilities (Agent Type)

For the Agent type, the prompt and capability mounting are the core configuration area of the evaluator. For the Code type, write the evaluate function on the Evaluation Code tab; see the new "Code Evaluator" document in the same directory for the full configuration flow.

Edit Prompt

An evaluation prompt is essentially the "standard operating procedure (SOP)" you define for the Agent Judge. To ensure objective and stable scoring, a professional evaluation instruction typically includes the following five core elements:

Element	Description	Notes
Role	Clearly defines the judge's professional identity and core evaluation objective.	Example: "You are a risk and compliance expert specializing in the financial industry. Your task is to conduct a risk audit of the customer service Agent's responses based on the given compliance guidelines."
Evaluation Dimensions	Describes the specific sub-items that make up the evaluation metrics, preventing the judge from scoring on "gut feeling" alone.	Example: Factual consistency: Does the response contradict the original data in the knowledge base? Tone compliance: Does it use legally prohibited absolute phrases (e.g., "guaranteed profits")?
Scoring Criteria	Defines the scoring format (e.g., 5-point scale, 3-point scale, binary Pass/Fail, deduction or bonus system) and its corresponding business meaning.	Example: Uses a 5-point scale. 5 points: Perfect response, rigorous logic, no flaws. 3 points: Mostly correct, but with minor information omissions. 1 point: Serious factual errors or compliance violations.
Scoring Checklist	The judge's execution manual, specifying how different scores combine with evaluation dimensions.	Example: Deduct 2 points for detecting a sensitive word; Deduct 1 point for incoherent logic; Award full score for a completely resolved issue delivered with empathy.
Context Variable Injection	References built-in system data through {{variable name}} syntax, placing the judge in a real-world evaluation context.	The system supports the following variables for automatic substitution: {{input}}: The user's original query. {{output}}: The tested Agent's response. {{expected_output}}: The reference ground truth. {{agent_trajectory}}: The Agent's minimal internal execution chain. {{tool_context}}: Tool calling parameters, return results, and context. {{rag_context}}: Raw fragments retrieved by RAG.

Note: You can choose to reference a built-in template with one click and fine-tune the scoring levels and criteria for your business scenario.

For example, if you want to create a "RAG Faithfulness Evaluator", your prompt structure can be as follows:

Role
You are a rigorous academic paper reviewer.

Evaluation Dimensions
Compare {{output}} with {{rag_context}} to check whether the response is completely faithful to the given reference passages.

Scoring Criteria (1–3 scale)
3 points: All claims in the response are supported by evidence in the reference passages.
2 points: Most content is correct, but contains a few "hallucinated" details not mentioned in the passages.
1 point: The response seriously deviates from the facts or directly contradicts the reference passages.

Scoring Checklist
Extract the core assertions from {{output}}.
Verify each one against {{rag_context}}.
Score based on evidence coverage, and provide specific conflicting evidence in the reason field.

Capability Mounting

Skills (Skill Extensions): Upload a SKILL.md file to grant the evaluator capabilities such as code execution and multi-step verification, enabling it to "verify by doing."

Step 3: Define Output Protocol

Output Field	Type	Description
Score	Required	Structured numeric score, drives the analytics dashboard and comparison reports
Reason	Required	Qualitative feedback explaining the scoring basis and guiding subsequent optimization
CoT (Chain of Thought)	Optional	Records the judge's reasoning process during scoring; a critical tool for diagnosing evaluator misjudgments

Module Entry Point

Access path: Log in to the AgentLoop console, find and click the Evaluation module in the left-side navigation pane.
Switch tabs: After entering the evaluation module, locate the tab navigation bar at the top of the page. Click Evaluator to enter the evaluator asset management center.
Feature layout:
- Left-side list/quick filter: Supports quick retrieval by type (built-in/custom) and tags.
- Top action bar: Supports searching evaluator names, or click Create Evaluator to start the custom build process.
- Main workspace: Displays all evaluator assets in a list view, supporting viewing details, editing configuration, deleting, or cloning.