Model evaluation
Evaluating models across multiple dimensions using datasets helps you fully understand their performance and ensures they deliver effective results in real-world applications.
Model evaluation overview
Model evaluation is a model capability assessment feature provided by the Model Studio platform. It supports two methods: custom evaluation and baseline evaluation. It scores and compares model inference results through evaluation dimensions to generate evaluation reports and leaderboards. You can use preset models or fine-tuned models as evaluation objects to quantify model performance and support model selection decisions. Currently, only text generation model evaluation is supported. Custom evaluation datasets support both single-turn (Prompt/Completion) and multi-turn dialogue (ChatML) formats, and evaluation results support multi-run sampling statistics such as Pass@K and export by bucketing rules. For more feature descriptions, see Model evaluation product overview.
Use cases
- Model selection comparison: Evaluate multiple candidate models with the same evaluation datasets and dimensions, and horizontally compare overall scores and pass rates on the leaderboard to drive model selection decisions with data.
- Fine-tuning effectiveness validation: Run evaluations before and after model fine-tuning respectively, compare the score changes, and quantify the capability improvements brought by fine-tuning. For fine-tuning operations, see Model fine-tuning.
- Capability quantification: Generate evaluation reports containing overall scores, pass rates, and score distributions to provide objective data support for team collaboration and management reporting.
- Continuous quality monitoring: Establish a regular evaluation mechanism, track model performance trends under the same evaluation conditions, and promptly detect model capability degradation.
- Difficult case mining and data backflow: Run multiple inferences per sample to compute Pass@K, use result bucketing to filter difficult samples, and backflow them as platform datasets for targeted fine-tuning.
Custom evaluation and baseline evaluation
Model Studio model evaluation supports two methods: custom evaluation based on business outcomes (evaluating with your datasets and custom dimensions) and baseline evaluation based on model general capabilities (quickly evaluating model basic capabilities with public standard datasets).
Custom evaluation uses your datasets and custom dimensions for evaluation and supports the following three scoring methods:
- Model evaluation (AI automatic evaluation): A judge model (such as Qwen-Max) scores the responses of the evaluated model. Numeric dimensions output integer scores within the scoring range, and classification dimensions output Pass/Fail labels. It applies to scenarios that require semantic understanding, such as QA quality and content safety.
- Rule evaluation (automated metrics): Scores are calculated directly with algorithms, without a judge model. String matching determines whether the output exactly matches the reference answer, and text similarity is calculated with algorithms such as ROUGE, BLEU, and Cosine. It applies to scenarios with deterministic standards, such as translation, summarization, and Function Calling.
- Manual evaluation (manual annotation): Personnel review model inference results one by one and annotate Pass/Fail labels. It applies to scenarios that machines can hardly evaluate accurately, such as creative writing and professional domain judgment.
Baseline evaluation provides 13 public standard benchmarks across five major categories: general knowledge, mathematics, code, context reasoning, and domain-specific. It quickly evaluates model basic capabilities and is available only in the Beijing region. For details, see Create a baseline evaluation task. The two evaluation methods are compared as follows:
Comparison item | Custom evaluation | Baseline evaluation |
|---|---|---|
Data source | User datasets or inference result sets | Public standard datasets (platform preset) |
Evaluation dimensions | Custom created | System preset benchmark library (13 across 5 categories, some support sub-dimension selection) |
Scoring method | AI evaluation / rule evaluation / manual annotation | Automatic system scoring |
Result download | Supported | Supported |
Availability | All regions | Beijing region only |
Evaluation workflow and core terms
The end-to-end evaluation workflow consists of four phases: prepare datasets (upload EvaluationSet-type data in the data management module) → create evaluation dimensions (define scoring criteria and methods) → create evaluation tasks (select models and associate datasets and dimensions) → view results (analyze score reports, optionally join a leaderboard for comparison).
Before use, distinguish two core concepts: the Scorer Prompt is configured on the evaluation dimension and instructs the judge model how to score the responses of the evaluated model. The System Prompt is configured on the evaluation task and sets the role or behavior guidelines for the evaluated model, and can usually be left blank. The two differ in target and cost attribution.
Quick start
This section guides you through your first model evaluation via the shortest path. Each step provides the console navigation path and the minimum configuration of the key form fields. After completing the following four steps, you will obtain an evaluation report containing the overall score and pass rate.
The top bar of the console page displays the current workspace (switchable), region (China (Beijing) by default), and the Back to earlier version entry. The evaluation methods available vary by region (for example, baseline evaluation is available only in the Beijing region).
Prerequisites- You have activated an Alibaba Cloud Model Studio account and can access the model evaluation feature through the Model Studio console.
- You have prepared an evaluation dataset: upload EvaluationSet-type data in the data management module. The sample format can be Text generation (prompt&completion) (single-turn, with Prompt and Completion columns) or Text generation (ChatML-messages) (multi-turn dialogue); either format works for the quick start.
- Create an evaluation dimension: In the left navigation pane of the console, choose Model Evaluation, switch to the Evaluation Dimensions tab, and click Create Evaluation Dimension; in the dialog, choose + Create Custom Evaluation Dimension. Minimum configuration: select Model Evaluation - Numeric as the dimension type, select Qwen-Max as the judge model, keep the scoring range at 0-5, set the pass threshold to 3, and keep the default prompt. For detailed configuration, see Create evaluation dimensions.
- Create an evaluation task: Switch to the Evaluation Tasks tab, click Create Evaluation Task, and select custom evaluation as the evaluation method. Minimum configuration: select the evaluated model, select evaluation dataset as the data source and specify the uploaded dataset, associate the dimension created in the previous step, keep the concurrency and inference times per sample at their defaults, and keep leaderboard participation off. After confirming, click Start Evaluation. For detailed configuration, see Create a custom evaluation task.
- Wait for the evaluation to complete: After submission, the task enters the Pending status and then automatically changes to In Progress. The evaluation duration depends on the data volume and model inference speed—50 data entries usually complete within minutes. After completion, you can view or annotate the evaluation results, or build a leaderboard of evaluation task performance to compare the effects of different models under the same dimension.
- View evaluation results: After the task is completed, click the task name to open the details page. View the task basic information and overall metrics on the Task Overview tab, and view the input, output, and dimension scores of each sample on the Case Analysis tab. For detailed interpretation, see View evaluation results.
Manage evaluation tasks
On the model evaluation page of the Model Studio console, you can perform daily management of evaluation tasks, evaluation dimensions, and leaderboards.
The reversibility and impact of each management operation are shown in the following table. Confirm the impact before performing an irreversible operation.
Operation | Reversibility | Impact |
|---|---|---|
Terminate task | Irreversible | Evaluated data retained; cannot be resumed |
Delete task | Irreversible | Task and results permanently deleted |
Rename | Reversible | Only affects the display name |
Remove task from leaderboard | Reversible | Can be added again |
Delete leaderboard | Irreversible | Does not affect associated evaluation tasks |
Delete evaluation dimension | Irreversible | Empty bound dimensions in the leaderboard will block new tasks |
Evaluation task management
- Terminate task: Available only in the In Progress status; the operation is irreversible. After termination, the evaluated data is retained but cannot be resumed, and the score column of the task in the leaderboard displays a dash.
- Delete task: Available only in the three terminal statuses: Completed, Terminated, and Failed; the operation is irreversible. The task and its evaluation results are permanently deleted. Tasks associated with a leaderboard must be removed from the leaderboard first.
- Rename: Only affects the display name, not the evaluation results.
- Download results: Download the evaluation result data file. Download is not supported in the Pending status.
- Filter and search: Supports searching by task name, filtering by status, and sorting by creation time.
Evaluation dimension management
Deleting an evaluation dimension is irreversible. After a dimension bound to a leaderboard is deleted, the bound dimensions of the leaderboard become empty, which blocks creating new evaluation tasks under that leaderboard. Confirm whether a leaderboard is using the dimension before deleting it.
Billing
Model evaluation costs consist of two parts: the inference costs of the evaluated model and the scoring costs of the judge model. For the unit prices of specific models, see the pricing information in Preset model list.
The cost structure of each dimension type is as follows:
Dimension type | Evaluated model inference cost | Judge model scoring cost |
|---|---|---|
Model Evaluation (Numeric/Classification) | Billed by token | Billed by token |
Rule Evaluation (String Matching/Text Similarity) | Billed by token | None |
Manual Evaluation (Classification) | Billed by token | None |
When using an inference result set | None (no model inference called) | Determined by the dimension type |
Evaluated model inference costs
When an evaluation dataset is used as the data source, the system calls the evaluated model for inference, billed by token consumption. Formula: input tokens / 1M x input unit price + output tokens / 1M x output unit price. When the inference times per sample is set to a value greater than 1, each sample is inferred the specified number of times, and the inference costs increase with the number of inferences, calculated based on actual consumption.
Evaluating deployed fine-tuned models incurs no additional charges—the inference costs are already included in the deployment computing costs.
When an inference result set is used as the data source, no model inference is performed, and no evaluated model inference costs are incurred.
Baseline evaluation is billed the same way as custom evaluation—by inference token consumption.
Judge model scoring costs
Only Model Evaluation dimensions (Numeric and Classification) involve judge model scoring costs, billed by token consumption. Rule Evaluation (string matching and text similarity) and manual evaluation do not use a judge model and incur no scoring costs.
Because the length of model inference output is uncertain, the system cannot estimate the total cost before the evaluation—costs are calculated based on actual consumption.
Cost optimization strategies
- Validate at small scale first: Use 50-100 data entries to verify configuration correctness and scoring quality. After confirming that everything is correct, scale up the data volume to avoid wasted consumption caused by configuration errors.
- Save inference result sets for reuse: After the first evaluation, download the results containing the model Output, and use the inference result set method for subsequent evaluations to avoid costs from repeated inference.
- Prefer rule evaluation: Rule evaluation incurs no judge model costs and is the lowest-cost evaluation method. Use it first for scenarios with deterministic standards, such as translation, summarization, and Function Calling.
FAQ
Question: When configuring an evaluation dimension, I easily confuse the scoring range and the pass threshold.
The scoring range defines the scoring interval of the judge model (integers, such as 0-5) and determines the score granularity. The pass threshold is the dividing line for Pass/Fail determination (decimals are supported, with a step size of 0.1, such as 3.0) and is used to calculate the pass rate. The two are configured independently: the former determines "how many points the model can score", and the latter determines "how many points count as a pass".
Question: I want to automate the evaluation workflow via API or SDK.
Model evaluation currently supports console operations only and does not provide a public API/SDK. For programmatic evaluation capabilities, you can use the PAI Judge Model API as an alternative.
Question: The evaluation shows that the model performs insufficiently in specific scenarios.
You can fine-tune the model and re-evaluate to verify the effect (see Model fine-tuning). If the model lacks knowledge in a specific domain, you can introduce a knowledge base to enhance model capabilities (see the knowledge base documentation).