Evaluation task

An evaluation task is the core feature of application evaluation, used to systematically assess the output quality of an application. It supports the evaluation of agent and workflow applications and facilitates multi-dimensional assessment by combining automatic evaluators with manual labels.

In the upper-left corner of the Evaluation task page, click Return to Old Version to return to the previous version of application evaluation.

Create an evaluation task

1. Go to the Evaluation task page, click Create Evaluation Task, and configure the following basic information:

   Field	Description
   Task name	A custom task name. The name can be up to 50 characters long.
   Task description	The purpose or use of the task. The description can be up to 200 characters long.
   Select evaluation set	Select an evaluation set and its version from the list of published evaluation sets.
   Select application	Select how to associate an application. Three options are available: Do not associate an application (Default): Does not associate any application. This option is suitable for scenarios that require only manual annotation. Workflow: Associates a workflow application. The system calls the workflow to perform an evaluation using the data from the evaluation set. Agent: Associates an agent application. The system calls the agent to perform an evaluation using the data from the evaluation set.
   Evaluator	Add an evaluator for automatic scoring. After you select an evaluator, you must configure parameter mappings to map the variable parameters of the evaluator to the actual data source. For more information, see evaluator. Configuration requirements: All variables must be mapped before you can save the configuration. Ensure that the mappings are correct to prevent inaccurate scoring results. Add 3 to 5 evaluators to each evaluation task to assess application quality from different dimensions. Common combinations include: Relevance evaluator (LLM) + Format validation evaluator (Code) Correctness evaluator (LLM) + String matching evaluator (Code) Agent tool invocation evaluator (LLM) + Parameter check evaluator (Code) Each evaluation task supports a maximum of 10 evaluators.
   Label	Add labels to the evaluation task for manual annotation (optional): Click the Add Label button and select the required labels from the list. The labels are used for the manual annotation feature on the task details page. After the task is created, you can also add labels by clicking Label Configuration on the task details page.

2. After confirming all configurations, click Complete Creation to create the evaluation task.

Manage tasks

After an evaluation task is created, click Details to the right of the task to open the task details page.

On the task details page, view data details and metric statistics:

• Data details: Displays the detailed results for each piece of evaluation data. You can perform operations such as data annotation and label configuration.

  • Evaluator scoring results: The automatic scoring column. It displays the score (from 0 to 1 or a custom range) given by the specified evaluator for each data entry.

  • Label configuration: Add labels to the evaluation task.

  • Annotation mode:
    • Normal mode: The page displays content in a tiled layout with fields arranged horizontally.
    • Quick annotation: After you click this, custom labels become editable. Category labels are displayed as drop-down lists, and input types are displayed as text boxes. Changes are saved immediately after you make them.

    In both modes, click Annotate to view the complete information for a single data entry and annotate it.

    The single-item annotation page has three columns. The Evaluation set data column on the left shows the original data. The Application output column in the middle shows the agent's output. The Manual annotation area on the right contains an Agent label (Boolean) field (select true, false, or Pass) and a Text (text) input box (up to 200 characters). At the bottom of the page, use Previous and Next to navigate through the data entries.

• Metric statistics: Displays information such as the overall score and evaluation progress. The metric statistics page includes an Overall score dashboard that shows the overall evaluation score as a percentage. It also has an Evaluation progress area that shows the total, unfinished, and completed quantities from the evaluation set. The Score summary area shows the pass rate of each evaluator in a bar chart. The page also includes a Score details - Data item distribution area.

Billing information

You are charged for the tokens consumed when an evaluation task calls a large language model. To view the number of consumed tokens, go to the Evaluation task page. For pricing details, see the Model Studio console.

FAQ

1. Can I modify an evaluation task after it is created?

   After an evaluation task is created, its configuration (application, evaluation set) cannot be modified. However, you can add manual labels for annotation at any time. If you need to use a different configuration, create a new evaluation task.

2. What is the "Do not associate an application" option for?

   The "Do not associate an application" option is for scenarios that require only manual annotation. If you select this option, the system does not automatically call any application for evaluation. You must rely entirely on manual labels for data annotation and assessment.

3. How do evaluators and labels work together?

   Evaluators are used for automatic scoring and are suitable for evaluation dimensions with clear rules or standard answers. Labels are used for manual annotation and are suitable for dimensions that require subjective judgment. In the same evaluation task, you can use evaluators for automatic scoring and add labels for supplementary manual annotation to achieve comprehensive evaluation coverage.

4. How do I return to the previous version of application evaluation?

   Go to the application evaluation page and click Return to Old Version in the upper-left corner.