Manual annotation evaluation

Describes the manual annotation workflow for evaluation tasks that use manual evaluation dimensions: configuring manual dimensions on the Case Analysis tab, quick annotation mode, and comparing the reference answer with the model output in the evaluation details drawer to apply labels.

Manual annotation evaluation overview

Manual annotation evaluation applies to scenarios that require human judgment, such as creative writing and professional content review. Evaluation tasks created with Manual Evaluation - Classification dimensions (see Manual evaluation) require personnel to annotate Pass/Fail labels for model inference results one by one. Baseline evaluation tasks cannot enter the manual annotation page.

Annotation workflow

1. In the evaluation task list, find the evaluation task that uses manual evaluation dimensions, and click the task name to open the details.
2. On the Case Analysis tab, you can first click Manual Dimension Configuration to add a dimension from the created manual evaluation dimension list in the dialog, or click Create Manual Dimension to create one (up to 10 manual annotation dimensions), and switch to Quick annotation mode to improve annotation efficiency.
3. Click Details of a sample to open the evaluation details drawer: the left side displays the complete multi-turn conversation in dialogue form, where you can compare the reference answer with the model output; the label area at the bottom shows the manual annotation dimensions and labels, and you complete the annotation by selecting the corresponding label for the current sample.
4. Use the Previous and Next buttons to page between samples until all data is annotated.

Annotation progress and task status

After all data is annotated, the task status automatically changes to Completed; for result interpretation, see View evaluation results. Before annotation is finished, the task remains in the In Progress status.

Compared with automatic evaluation, manual evaluation incurs no judge model scoring costs but requires labor input and takes longer. For detailed label design instructions, see Manual evaluation.