Create a custom evaluation task
Describes how to create a custom evaluation task: select the evaluation model, configure the data source (single-turn prompt&completion or multi-turn ChatML evaluation sets), associate evaluation dimensions, and set the concurrency and inference times per sample.
Create a custom evaluation task overview
Custom evaluation evaluates models based on business outcomes, using your datasets and custom dimensions. Before creating an evaluation task, confirm that you have created at least one evaluation dimension (see Evaluation dimensions) and uploaded the evaluation dataset in the data management module (see Training sets and evaluation sets).
On the model evaluation page of the Model Studio console, select the Evaluation Tasks tab, click Create Evaluation Task, and select custom evaluation.
Select an evaluation model
Select the target model to evaluate from the model list in the console. Preset models and fine-tuned models are both supported. For the specific supported models, see Preset model list. The model selection dialog supports switching between official models and your models and filtering by capability (Function Calling, structured output, etc.); models displayed in gray are currently not supported for evaluation.
Configure the data source
The data source supports two methods:
- Evaluation dataset: Supports the Text generation (prompt&completion) single-turn format (Prompt and Completion) and the Text generation (ChatML-messages) multi-turn dialogue format (the reference answer is the last assistant message). The system calls the evaluated model for inference to generate the Output before scoring, which incurs inference costs. Multi-turn dialogue data is evaluated with the complete conversation, and the last user message is used as the current question by default. It must be of the EvaluationSet type with a published version and uploaded in the data management module in advance.
- Inference result set: Already contains the model Output. The system directly scores it without calling model inference, incurring no inference costs. It applies to scenarios where you already have model outputs and want to reduce costs.
Comparison item | Evaluation dataset | Inference result set |
|---|---|---|
Data content | Contains Prompt and Completion | Already contains the model Output |
Inference call | The system calls the evaluated model for inference | No model inference called |
Inference cost | Incurred | Not incurred |
Applicable scenarios | First evaluation, standard workflow | Existing model outputs, cost reduction |
Data requirements | EvaluationSet type with a published version | Upload files by template |
Data volume recommendations: 50-100 entries for small-scale validation, 200-500 entries for formal evaluation, and over 500 entries for comprehensive assessment.
Associate dimensions and task configuration
Multi-select the dimensions to associate from the created evaluation dimensions, which can mix custom dimensions and preset templates and must be created on the evaluation dimension page in advance.
Inference parameters are dynamically loaded based on the selected model. The configurable items include System Prompt (up to 5,000 characters, setting the role of the evaluated model, which can be left blank in most scenarios), thinking mode (enable_thinking), top-p sampling (top_p), temperature, stop words (stop), web search (enable_search), and thinking chain length (thinking_budget).
The concurrency defaults to 8 and can be set from 1 to 20. It affects the execution speed of the evaluation task but does not change the evaluation results.
The inference times per sample defaults to 1. When set to a value greater than 1, each sample is inferred multiple times and aggregate metrics such as Pass@K are computed based on the multi-run sampling results, which helps determine sample difficulty and model performance stability (inference and scoring costs increase with the number of inferences).
The leaderboard participation switch is off by default. After you enable it, you must select a leaderboard, and the evaluation results will join it for ranking. If the dimensions bound to the leaderboard are empty, the system will block submission and display an error message.
When the evaluation dimensions are cleared, the system automatically clears the leaderboard selection. The task name can be up to 50 characters and is auto-filled based on the date and time by default. After configuration, click Start Evaluation to submit the task. Costs are calculated based on actual consumption (see Billing), and you can view the calculation details before submission.
After the task is submitted, it enters the execution workflow. For status transitions and how to view results, see View evaluation results.
FAQ
Question: I selected the wrong evaluated model when creating an evaluation task and want to change the model.
The target model cannot be changed after the task is submitted. Solution: delete the task and recreate it. Consumed token costs cannot be refunded.
Question: The status shows Failed after I submit an evaluation task.
Check the following in order: the dataset must be of the EvaluationSet type with a published version (training sets cannot be used for evaluation), the model must be selectable (gray indicates currently unsupported), and the data format must be correct (the prompt&completion format must contain the Prompt and Completion columns, and the ChatML format must be valid jsonl containing messages arrays).