Manual evaluation

Manual evaluation dimensions use evaluators to manually classify model outputs by label. They suit scenarios that require subjective judgment, such as creative writing and professional domain judgment.

Manual evaluation overview

Manual evaluation uses evaluators to review model inference results one by one and annotate Pass/Fail labels. It does not involve a judge model, incurs no scoring costs, but requires labor input and takes longer. It suits scenarios that machines can hardly evaluate accurately:

• Creative writing, tone and style, and other subjective quality judgments.
• Compliance review of professional domain content.

For comparison with other dimension types and selection recommendations, see Evaluation dimensions.

Create manual evaluation dimensions

Log on to the Model Studio console, choose Model Evaluation in the left-side navigation pane, switch to the Evaluation Dimensions tab, and click Create Evaluation Dimension, and in the dialog, click + Create Custom Evaluation Dimension to go to the custom configuration page. After you fill in the common fields (dimension name up to 20 characters, description up to 100 characters, and the type cannot be changed after creation), complete the configuration as follows:

1. Configure the Pass label: define the meaning of the label for pass.
2. Configure the Fail label: define the meaning of the label for fail. Pass and Fail labels must not duplicate each other.
3. Click Save.

Label design recommendations

Pass and Fail labels should be mutually exclusive and exhaustive, that is, all possible outputs can fall into one of the classes. Binary classification is sufficient for simple scenarios; for complex scenarios that require multiple labels, control the number of labels to reduce evaluation ambiguity. Label design should be concise and clear and cover all possible judgment outcomes.

Evaluation tasks using manual evaluation dimensions change to the Completed status only after all data is annotated. For annotation operations, see Manual annotation evaluation.