DocsICP FilingConsole
Documentation
Home Page Alibaba Cloud Model Studio Models Model Training Evaluation Evaluation Dimensions LLM evaluation

LLM evaluation

Updated at:
Copy as MD
Product
My Favorites

LLM evaluation dimensions use a judge model to score the outputs of the evaluated model based on a scorer prompt. They support both numeric and classification methods and provide 19 preset templates, AI optimization, and trial runs.

LLM evaluation overview

LLM evaluation uses a judge model (such as Qwen-Max) to score the outputs of the evaluated model based on the scorer prompt you configure. It is divided into two methods:

  • LLM evaluation - numeric: a judge model assigns continuous numeric scores to model outputs (such as 0~5). It suits scenarios that require fine-grained quantification, such as Q&A quality evaluation and content generation quality scoring.
  • LLM evaluation - classification: a judge model classifies model outputs by user-defined labels (Pass/Fail). It suits judgment scenarios that require semantic understanding, such as content safety detection and answer correctness checking.

Numeric dimensions output integer scores within the scoring range and determine Pass/Fail through the passing threshold; classification dimensions directly output labels. Both methods incur judge model scoring costs billed by token (currently free of charge for a limited time). For comparison with other dimension types and selection recommendations, see Evaluation dimensions.

Create LLM evaluation dimensions

Log on to the Model Studio console, choose Model Evaluation in the left-side navigation pane, switch to the Evaluation Dimensions tab, and click Create Evaluation Dimension to open the selection dialog: you can filter the Preset templates list by type and tag and click Use Template on a template card to apply it in one click, or click + Create Custom Evaluation Dimension to go to the custom configuration page.

When creating a custom evaluation dimension, first fill in the following common fields:

  • Dimension name: required. The length cannot exceed 20 characters.
  • Description: optional. The length cannot exceed 100 characters.
  • Type: required. Select LLM evaluation - numeric or LLM evaluation - classification. The scorer type cannot be changed after creation. If you select the wrong type, you must delete and recreate the dimension.

After you select a type, complete the configuration as follows:

Numeric

  1. Select a judge model (Qwen-Max is recommended for its strong reasoning capability; judge model calls are currently free of charge for a limited time).
  2. Select a scorer template: use Select preset template to apply a platform preset template (see Preset templates for numeric templates), or modify the default prompt in the editor.
  3. Write the scorer prompt: the prompt must contain at least one variable.
  4. Set the scoring range: configure the minimum and maximum values, both of which are integers. The minimum value must be ≥ 0 and the maximum value must be ≥ 1. The default is 0~5.
  5. Set the passing threshold: outputs scored at or above this value are marked as Pass. The step is 0.1. The threshold is automatically linked to the scoring range.
  6. Click Save.

Classification

  1. Select a judge model (Qwen-Max is recommended).
  2. Select a scorer template: use Select preset template to apply a platform preset template (see Preset templates for classification templates), or modify the default prompt in the editor.
  3. Write the scorer prompt: the prompt must contain at least one variable (${messages}, ${completion}, or ${output}) and be no longer than 20,000 characters.
  4. Configure the Pass label and Fail label: define the classification labels for pass and fail. Each label can contain up to 20 characters. Pass and Fail labels must not duplicate each other.
  5. Click Save.

Configure scorer prompts

A scorer prompt is a prompt that instructs the judge model on how to evaluate output quality. It defines the evaluation criteria and output format of the judge model.

Prompt variables

Scorer prompts support the following three variables, which the system automatically replaces with actual values during evaluation execution (type / in the prompt editor to quickly reference a variable):

  • ${messages}: the complete conversation. For multi-turn (ChatML) evaluation datasets, it is all messages; for single-turn (prompt&completion) evaluation datasets, it is the user input.
  • ${completion}: the reference reply, that is, the reference answer preset in the evaluation dataset (for multi-turn datasets, the last assistant message).
  • ${output}: the output of the model under test, that is, the model's reply to the current question.

Use the ${variable name} syntax in the prompt to reference variables. The prompt must contain at least one variable; otherwise, it cannot be submitted.

Preset templates

In the Create Evaluation Dimension dialog, filter preset templates by type (LLM evaluation - numeric, LLM evaluation - classification) and tag (general quality, text matching, text similarity, safety, text classification), and then click Use Template to apply one. The template list is dynamically fetched from the system configuration.

Preset templates for the numeric type (14):

  • Answer Completeness: evaluates whether the reply covers all key content requested by the user.

  • Q&A Relevance: evaluates the relevance and completeness of the model reply to the user query.

  • Hallucination: detects whether the reply contains false or hallucinated information.

  • Instruction Compliance: evaluates whether the reply strictly follows the given instructions, formats, and constraints.

  • Consistency with Reference Answer: checks the consistency between the reply and the standard answer.

  • Translation Quality: evaluates how comprehensively the translation restores the meaning, tone, terminology, and expression of the source text.

  • Content Creativity: evaluates whether the reply is novel and valuable in expression or conception while meeting the task requirements.

  • Risk Identification Completeness: evaluates whether the reply completely identifies the key risks in a plan or behavior.

  • Privacy Protection: evaluates whether the reply properly protects personal information, sensitive data, and user privacy.

  • Suggestion Actionability: evaluates whether the suggestions in the reply are specific, feasible, and able to guide next steps.

  • Answer Helpfulness: evaluates whether the reply truly helps the user achieve the current goal.

  • Logical Coherence: evaluates the argumentative structure, consistency, and reasoning flow of the reply.

  • Answer Conciseness: evaluates whether the reply avoids redundancy and completes the task with the necessary length.

  • Language Fluency: evaluates whether the reply is natural, smooth, and easy to read.

Preset templates for the classification type (5):

  • Problem Resolution Status Classification: determines whether the reply solves the user's current problem.
  • Question Answerability Classification: determines whether the current question can be directly answered based on the provided information.
  • Safety Risk Classification: identifies whether the reply is safe, has potential risks, or contains high-risk content.
  • User Intent Classification: identifies the user's primary intent in the complete conversation.
  • Sentiment Classification: marks the reply under evaluation as positive, neutral, or negative.

Custom prompt writing

Key points:

  • Use the ${messages}, ${completion}, and ${output} variables to reference evaluation data.
  • Clearly define the scoring criteria. For the classification type, define the judgment conditions for each label; for the numeric type, define the quality standard for each score.
  • Specify the output format requirements to ensure the system can correctly parse the evaluation results of the judge model.

The following is an example custom prompt for the classification type:

Evaluate the quality of the following model answer. Complete conversation: ${messages}. Model answer: ${output}. Reference answer: ${completion}. Scoring criteria: if the answer contains the core elements of the reference answer and contains no factual errors, mark it as Pass; if the answer omits key information or contains factual errors, mark it as Fail. Output only Pass or Fail.

Switching a scorer template overwrites the current prompt content. Back up manually edited content before switching.

AI optimization and trial run

  • AI optimization: click Optimize in the toolbar of the prompt editor, and the system generates an optimized version based on the current prompt. In the dialog, you can choose Use Now, Regenerate, or exit.
  • Trial run: after configuration, click Start Run in the trial run area to verify the scoring effect with sample data; the run result is displayed on the right. Trial runs and judge model calls are currently free of charge for a limited time.

FAQ

Issue: The system prompts that variables are missing when I submit a scorer prompt.

The scorer prompt must contain at least one variable (${messages}, ${completion}, or ${output}). Check the prompt content to ensure that variables are correctly referenced in the ${variable name} format.

Issue: The prompt content I manually edited disappears after I switch the scorer template.

This is expected behavior. Switching a template replaces the prompt with the preset content of the template. If you have important custom prompt content, back it up before switching templates.

Issue: The evaluation is complete, but the scores do not meet expectations—they cluster in one score range or show low discrimination.

First, check whether the scorer prompt has clear scoring criteria and correct variable references. Vague scoring criteria cause judge model scores to cluster in one score range. It is recommended to provide a clear description of the judgment conditions for each score level.

Second, confirm whether the reasoning capability of the judge model is sufficient (Qwen-Max is recommended), and regular manual spot checks and calibration are recommended. For scorer prompt optimization, see Prompt best practices.

Issue: Do evaluation dimensions of the LLM evaluation type incur fees?

Creating and managing evaluation dimension templates is free of charge. However, during evaluation task execution, the LLM evaluation types call the judge model for inference, and the resulting inference fees are charged according to the billing standard of the judge model (currently free of charge for a limited time).

Previous：Define evaluation dimensionsNext：Rule evaluation