Model evaluation (ModelEval)

Quick start

Run your first evaluation in 5 minutes: use the CMMLU dataset to measure the Chinese language understanding and reasoning capabilities of Qwen3-4B.

In the PAI console, choose Model Application > Model Evaluation (ModelEval) in the left navigation pane.
On the Model Evaluation page, click Create Task.
Configure the following parameters:
- Basic Configuration: The Task name and Result Output Path are auto-generated. Accept the defaults or modify them as needed.
  
  Note
  If the workspace doesn't have a default OSS storage path, manually select an output path.
- Evaluation Mode Configuration: select Single Model Online Evaluation.
- Evaluation Object Configurations: For Evaluation Object, select Public Model. In the Public Model drop-down list, search for and select Qwen3-4B.
- Evaluation Method Configuration: Select Public Dataset Evaluation. In the dataset list, select CMMLU.
- Resource Configuration: Select Single node - Standard.
  - Resource Type: Select General Computing.
  - Source: Select Public Resources.
  - Job Resource: Select a GPU instance type, for example, ecs.gn7i-c8g1.2xlarge (24 GB).
  - If the selected instance type is out of stock, choose another GPU instance.
Click OK at the bottom of the page. The task detail page opens.

When the task status changes to Succeeded, open the Evaluation Report tab to see how Qwen3-4B scores on the CMMLU dataset.

Configure evaluations

Configure evaluation mode

ModelEval supports three evaluation modes. Choose the mode that fits your use case.

Evaluation Mode	Description
Single Model Online Evaluation	Scores the inference output of a single model.
Dual-Model Arena	Runs online inference on both models using the same questions and compares their responses to identify the better-performing model.
Dual-Model Offline Competition	Compares two models using a pre-built question-and-answer dataset to identify the better-performing model.

Configure evaluation target

ModelEval supports four evaluation targets. Choose based on where your model or service is deployed.

Evaluation Target	Description	Use case
Public Model	Models from the PAI Model Gallery	Benchmark mainstream open-source LLMs
Custom Model	Custom models registered in AI Asset Management > Model Important Make sure the model is compatible with the vLLM framework.	Evaluate fine-tuned or customized models
PAI-EAS Service	A deployed PAI-EAS (Elastic Algorithm Service) online inference service	Evaluate model services running in production
Custom Services	Any model service with an OpenAI-compatible API	Evaluate third-party or self-hosted model services

Configure evaluation method

Use custom datasets, public datasets, or a combination of both.

Custom dataset evaluation

Use your own dataset to get evaluation results that reflect actual business scenarios.

Dataset format: JSONL (UTF-8 encoded, one JSON object per line).
Upload: Upload the dataset file to OSS (Object Storage Service) and enter its OSS path on the configuration page.

Evaluation Method	General NLP Metric Evaluation	Multi-Metric Evaluation with LLM-as-a-Judge
Purpose	For questions with ground-truth answers, this metric measures text similarity between the model's output and the reference answer. Use for translation, summarization, and knowledge-base Q&A tasks.	When questions don't have a single correct answer—such as open-ended dialogue or content generation—a powerful judge model scores the quality of the model's responses.
Dataset format	Each JSON object must include a `question` field and an `answer` field (ground truth). `{"question": "What is the capital of France?", "answer": "Paris"}`	Each JSON object must include a `question` field. The `answer` field is optional. `{"question": "Describe the history of artificial intelligence."}`
Metrics	ROUGE (ROUGE-1, ROUGE-2, ROUGE-L): Recall-based metric that measures how much of the ground truth the model's output covers. BLEU (BLEU-1, BLEU-2, BLEU-3, BLEU-4): Precision-based metric that measures how closely the model's output matches the ground truth.	The system sends the `question` and the model's output to the judge model, which scores the response across dimensions such as relevance, accuracy, and fluency.

Public dataset evaluation

Use industry-standard datasets to benchmark your model against established reference points.

Purpose: Model selection comparisons, pre-release benchmark testing, and general capability assessment.
Configuration: Select Public Dataset Evaluation, then select one or more datasets from the list.
Supported datasets:
- LiveCodeBench: Code processing capability.
- Math500: Mathematical reasoning (500 high-difficulty math competition problems).
- AIME25: Mathematical reasoning based on the 2025 AIME problems.
- AIME24: Mathematical reasoning based on the 2024 AIME problems.
- CMMLU: Chinese multi-subject language understanding.
- MMLU: English multi-subject language understanding.
- C-Eval: Comprehensive Chinese language capability evaluation.
- GSM8K: Mathematical reasoning.
- HellaSwag: Commonsense reasoning.
- TruthfulQA: Truthfulness evaluation.

Manage tasks

On the Model Evaluation page, manage evaluation tasks.

View Report: For tasks with status Succeeded, click this button to view the detailed evaluation report.
Compare: Select 2 to 5 completed tasks and click Compare to compare their scores side by side.
Stop: Stop a In operation task. This action is irreversible: the task can't be resumed and consumed compute resources aren't refunded.
Delete: Removes the task record. This action can't be undone.

Billing

ModelEval charges for the following:

Computing resources

Resource Type

Billing method

Billed item

Billing rule

Public resources

Pay-as-you-go

Actual running duration

Billing amount = (unit price / 60) × service duration (minutes)

For instance prices, see the pricing information on the console.

Resource quota

Subscription (prepaid)

Number and duration of node instances purchased

Purchase dedicated resources upfront. Charges are based on the number of node instances and the subscription duration. For details, see Billing of AI computing resources.

Judge model

Additional charges apply if you use judge model evaluation. For pricing details, see PAI Token Service pricing.