Rule evaluation

Rule evaluation dimensions use algorithms to calculate scores directly, without a judge model. They support string matching and text similarity methods and are the lowest-cost evaluation method.

Rule evaluation overview

Rule evaluation uses algorithms to calculate scores directly, without a judge model and with no scoring costs. It is the lowest-cost and fastest evaluation method:

• Rule evaluation - string matching: programmatically determines the exact match relationship between the model output and the reference text (equal/not equal/contains). It suits scenarios with deterministic standard answers, such as Function Calling and NL2SQL.
• Rule evaluation - text similarity: uses algorithms to calculate the similarity between the model output and the reference text (BLEU/ROUGE/cosine, etc.). It suits quantifying the similarity of open-ended answers, such as translation, summarization, and rewriting.

For comparison with other dimension types and selection recommendations, see Evaluation dimensions.

Create rule evaluation dimensions

Log on to the Model Studio console, choose Model Evaluation in the left-side navigation pane, switch to the Evaluation Dimensions tab, and click Create Evaluation Dimension to open the selection dialog: you can filter the Preset templates list by type and tag (text matching, text similarity) (28 templates in total; see the list below) and click Use Template on a template card to apply it in one click, or click + Create Custom Evaluation Dimension to go to the custom configuration page.

After you fill in the common fields (dimension name up to 20 characters, description up to 100 characters, and the type cannot be changed after creation), complete the configuration as follows:

Rule evaluation preset templates

Rule evaluation provides 28 preset templates, grouped by tag into text similarity (14) and text matching (14). Filter them by tag in the Create Evaluation Dimension dialog and click Use Template to apply one; the list is dynamically fetched from the system configuration.

Text similarity preset templates (14):

• ROUGE-1/2/3/4/5: calculates similarity based on 1- to 5-gram (n-word) overlap.
• ROUGE-L: based on the longest common subsequence (LCS), commonly used for summarization evaluation.
• BLEU: based on n-gram precision and brevity penalty, commonly used for machine translation evaluation.
• Sentence BLEU: sentence-level BLEU, optimized for single-sentence evaluation.
• GLEU: an improved version of BLEU, more stable for sentence-level evaluation.
• Cosine: calculates cosine similarity based on TF-IDF vectorization.
• Jaccard: based on the ratio of the intersection to the union of word sets.
• F1: token-level precision and recall, supporting mixed Chinese and English text.
• Levenshtein: based on the Levenshtein distance, suitable for spelling correction and approximate string matching.
• Edit distance: normalizes the edit distance into a similarity score.

Text matching preset templates (14):

• Exact match: the text under evaluation is identical to the reference text.
• Substring match: the text under evaluation contains the reference text as a substring.
• Prefix match/suffix match: the text under evaluation starts/ends with the reference text.
• Contains all substrings/contains any substring: the text under evaluation contains all/any of the specified substrings.
• Regex match: matches by regular expression pattern.
• Word overlap rate/character overlap rate: word-level/character-level overlap rate.
• Numeric accuracy: checks whether the numbers in the reply are accurate within a tolerance range.
• JSON format validation/JSON content match: validates JSON format legality or deeply compares JSON key-value content.
• Reasoning format validation/reasoning tool format validation: validates the format compliance of reasoning content and tool call JSON.

The Contains all substrings, Contains any substring, and Regex match templates use the extra custom field of the evaluation data (see Evaluation set format) as the evaluation input.

Similarity algorithm selection

Select an appropriate similarity algorithm based on the evaluation task type:

• Translation tasks: BLEU (measures n-gram exact matching).
• Summarization tasks: ROUGE-L (measures the longest common subsequence).
• Semantic relevance: Cosine Similarity (measures semantic vector distance).
• Classification tasks: Accuracy (measures the exact match rate).
• OCR and speech recognition: Fuzzy Matching.

FAQ