Use the performance evaluation module to comprehensively evaluate the RAG (Retrieval-Augmented Generation) development pipeline on AI Search Open Platform. This evaluation covers the entire process, from a user's initial question to retrieval by the RAG system and answer generation by the LLM (Large Language Model).
Prerequisites
You have activated the AI Search Open Platform service. For more information, see Activate the service.
Notes
Billing for performance evaluation is based on the computing resources consumed. For more information, see Billing methods and billable items.
Procedure
-
Log on to the AI Search Open Platform console.
-
Select the China (Shanghai) region, switch to AI Search Open Platform, and then select the target workspace.
Note-
AI Search Open Platform is currently available only in the China (Shanghai) and Germany (Frankfurt) regions.
-
Users in the China (Hangzhou), China (Shenzhen), China (Beijing), China (Zhangjiakou), and China (Qingdao) regions can access the service in a different region by using a VPC address.
-
Workspaces isolate and manage data. After you activate AI Search Open Platform for the first time, the system automatically creates a Default workspace. You can also create a workspace.
-
-
In the navigation pane, choose Effect Evaluation, and then click Effect Evaluation.
-
On the Create Evaluation Task page, enter a task name and upload an evaluation dataset using the format specified in the Sample data.
Important-
An evaluation dataset can contain a maximum of 200 valid entries. If you exceed this limit, the system returns an error.
-
Follow the sample template exactly when uploading the evaluation dataset. The reference answer is optional, but you cannot mix questions with and without reference answers in the same dataset.
The Task Name must be 1 to 30 characters long, start with a letter, and contain only letters, digits, and underscores (_). The evaluation dataset must be an Excel file. Task billing is based on the computing resources consumed during the performance evaluation.
The following table describes the fields in the evaluation template and the key evaluation metrics.
Parameter
Description
question
Your question.
standard_answer
The reference answer. This field is optional and affects the metrics returned in the evaluation report.
-
If a reference answer is provided, the following evaluation metrics are used:
-
faithfulness: Measures the factual consistency of the model-generated answer against the retrieved documents. A value of 1 indicates the answer is faithful (no hallucination), while 0 indicates a hallucination.
-
context precision: Measures how accurately the retrieved documents support the reference answer. A value of 1 indicates accurate. A value of 0 indicates inaccurate.
-
context recall: Measures how completely the retrieved documents cover the information required by the reference answer. A value of 1 indicates complete. A value of 0 indicates incomplete.
-
satisfaction: A composite score indicating the overall quality of the model-generated answer compared to the reference answer.
-
If the model-generated answer is free from hallucination, accurate, and complete, the satisfaction score is 1.
-
If the model-generated answer is free from hallucination but is inaccurate or incomplete, the satisfaction score is 0.5.
-
If the model-generated answer contains a hallucination, the satisfaction score is 0.
-
-
comprehensive score: A composite score calculated from faithfulness, context precision, context recall, and satisfaction.
-
-
If no reference answer is provided, the following evaluation metrics are used:
-
context relevance: How relevant the retrieved documents are to the question. A value of 1 indicates relevant. A value of 0 indicates irrelevant.
-
credibility: A score indicating whether the model-generated answer is trustworthy and based on the provided retrieval results.
-
A score of 1 is assigned if the answer is free from hallucination and based on relevant retrieved documents. This includes cases where the model correctly responds "Unable to answer" when no relevant documents are retrieved.
-
A score of 0.5 is assigned if the answer is free from hallucination but based on irrelevant documents, or if the model incorrectly responds "Unable to answer" when relevant documents were retrieved.
-
A score of 0 is assigned if the answer contains a hallucination.
-
-
faithfulness: Measures the factual consistency of the model-generated answer against the retrieved documents. A value of 1 indicates the answer is faithful (no hallucination), while 0 indicates a hallucination.
-
comprehensive score: A composite score calculated from context relevance, faithfulness, and credibility.
-
recall_docs
Retrieved documents.
model_answer
The model-generated answer.
-
-
After configuring the parameters, click OK to create the evaluation task.
The status of an evaluation task can be:
-
Evaluating or Failed: You can delete the evaluation task.
-
Successful: You can download the evaluation report in Excel format. The report contains two sheets:
-
Sheet1 - Evaluation Task: An overview of the evaluation task. Average metric values are calculated from all successfully evaluated questions.
Sheet2 - Task Details: Detailed evaluation data for each question.
The evaluation task list displays all task information in a table, including Task Name, Task Status (such as Evaluating or Successful), Creation Time, Completion Time, Evaluation Data (the name of the uploaded data file), and the Actions column. After a task is complete, its status changes to Successful. In the Actions column, click Download Report to download the evaluation report.
-
-