Evaluation Tasks
An evaluation task is the process of combining specific evaluation data sources with evaluation metrics (evaluators) to execute automated assessments and produce evaluation results. By deeply analyzing task results, developers can precisely quantify the actual performance level of AI applications, gaining critical insights that inform key decisions such as business deployment, configuration tuning, and cost optimization. This document guides you through creating, configuring, and managing evaluation tasks.
Create Evaluation Task
Creating an evaluation task is the process of deeply binding "data to be evaluated" with "evaluation criteria," consisting of two core stages: data configuration and evaluator mapping.
Step 1: Data Configuration
This step defines the boundaries of the evaluation task, determining where the system retrieves data and at what frequency.
Basic Configuration
Task Name (required): Give the task a descriptive name, such as "E-commerce Customer Service Overall Test."
Task Description (optional): Provide additional explanation of the evaluation task's purpose and scope.
Data Source
The data source determines where the evaluation task retrieves the data to be evaluated. Three types are supported:
Source Type | Description | Scenarios |
Trace | Retrieves evaluation data from the call chain of the Agent application | Online traffic evaluation, continuous monitoring |
Log | Retrieves data from the log system | Offline analysis based on logs |
Dataset | Uses pre-prepared test data | Pre-launch testing, version comparison, regression evaluation |
Execution Strategy
You can flexibly select one or more execution modes based on your business requirements:
Continuous evaluation based on new data: Once enabled, the system acts as a real-time sentinel, automatically triggering evaluation as soon as new data is generated.
Evaluation based on historical data: Once enabled, the system performs batch retroactive evaluation on existing data within a selected time period. Supports Data Preview to verify sample completeness.
Sampling Configuration
Sampling Ratio: 0%–100%, set via a slider. A high sampling rate is suitable for fine-grained evaluation; a low sampling rate is suitable for cost control in high-traffic scenarios.
Maximum Sample Size: Limits the maximum number of data records per evaluation to avoid excessive resource consumption.
Note: The core principle of sampling configuration is to strike a balance between statistical reliability and cost. For comprehensive pre-launch testing, 100% sampling is recommended; for continuous online monitoring, a sampling rate of 10%–50% is recommended based on traffic volume.
Step 2: Select Evaluators
This step defines what criteria to evaluate by and how the evaluator reads the data.
Preview Data: Displays a real-time preview of the data source, helping you understand the actual data structure and confirming whether variable mappings are correct.
Evaluator List: Supports multiple selections from the "Preset Evaluators" or "Custom Evaluators" library, enabling multi-dimensional parallel evaluation of the same dataset.
After selecting an evaluator, the right panel displays:
Evaluator name and type tags
The list of input variables required by the evaluator
The source field mapping for variables (automatically matched by the system or manually specified)
Variable mapping ensures that variables in the evaluator Prompt correctly correspond to fields in the actual data. For example, the input variable maps to the user input field in the Span data, and output maps to the Agent output field.
After completing the configuration, click Save and Run to start executing the task.
Evaluation Task List
The evaluation task list page displays all created tasks, including the following information:
Column | Description |
Task Name/ID | Task identifier; click to view details |
Description | Additional information about the task |
Data Source | Data source type (Trace/Log/Dataset) |
Status | Current execution status, including Pending, Scheduling, Running, Completed, Failed, and Terminated. |
Created Time | Task creation time |
Updated Time | Task update time |
Manage Evaluation Tasks
In the task list, the following operations are supported:
Start: Start a stopped task
Stop: Pause a running task
Edit Task: Modify task configuration (description, execution strategy, evaluators, etc.)
Trigger Run: Trigger a new evaluation execution
Copy: Quickly create a new task based on an existing task configuration
Delete: Delete a task (deleting a task does not affect viewing historical data).