Evaluation sets

Conversation analysis

• The supported file formats are .xls and .xlsx.

• Field descriptions:

  Field	Description
  Prompt	The user's input, which is usually a question or an instruction.
  Completion	The reference answer for the prompt. It is used to evaluate the model's actual response.
  SessionId	Identifies a complete conversation session. Multiple rows with the same SessionId are treated as a continuous, multi-turn conversation.

• Example:

  Prompt(user)	Completion(assistant)	SessionId	Description
  In which city were the Asian Games held?	Hangzhou.	49aa5bc0-76de-471d-9f50-c8f7710bbc9d	Multi-turn conversation - Turn 1
  In what year?	2023	49aa5bc0-76de-471d-9f50-c8f7710bbc9d	Multi-turn conversation - Turn 2

Knowledge-based Q&A

• Supported file format: .jsonl.

• Field descriptions:

  Field	Description
  query	The user's input question that the agent accepts.
  queryType	A tag for question classification, such as factual, analytical, comparative, or tutorial. You can also define custom types.
  referenceAnswer	The ground truth. It is used to evaluate the accuracy, completeness, and relevance of the response.
  fineKeywords	Fine-grained keywords. These are specific information points that the answer should contain, in a nested list format.
  coarseKeywords	Coarse-grained keywords. These are core topic words, usually 1 to 3 words.

  Note

  • coarseKeywords (coarse-grained keywords)

    • Meaning and function: Represents the core topic words of the query, such as one to three words. It determines whether the retrieved content deviates from the main topic.

    • Example: ["The Bund", "Pudong"]

  • fineKeywords (fine-grained keywords)

    • Meaning and function: These key information points are required to form the ground truth and are used to evaluate the information completeness of the response.

    • Format and example: Uses a nested array [[ ]] format. Each sub-array [ ] represents an independent information point.

      [
        ["The historic buildings on the Bund have a romantic, European charm"],
        ["Across the river, Pudong is developing rapidly with many new buildings, creating a stark contrast with the old Bund and showcasing Shanghai's vitality"],
        ["The night view of the Bund is like a fairy-tale world, a mesmerizing city of lights"],
        ["The 88-story Jin Mao Tower in Pudong is crystal clear and spectacularly echoes the twinkling lights of the Oriental Pearl Tower"]
      ]

• Example:

  {"query":"What is the difference between the night view of the Bund and the new buildings in Pudong?","fineKeywords":[["The historic buildings on the Bund have a romantic, European charm"],["Across the river, Pudong is developing rapidly with many new buildings, creating a stark contrast with the old Bund and showcasing Shanghai's vitality"],["The night view of the Bund is like a fairy-tale world, a mesmerizing city of lights"],["The 88-story Jin Mao Tower in Pudong is crystal clear and spectacularly echoes the twinkling lights of the Oriental Pearl Tower"]],"referenceAnswer":"The night view of the Bund is full of European charm, with historic buildings appearing romantic and enchanting under the lights. In contrast, the new buildings in Pudong showcase Shanghai's rapid modernization. Skyscrapers like the Jin Mao Tower and the Oriental Pearl Tower are crystal clear and brilliantly lit, creating a stark contrast with the Bund and reflecting the city's unique blend of old and new.","coarseKeywords":["The Bund","Pudong"],"queryType":"Comparative"}
  {"query":"What are the main exhibition halls of the Shanghai Science and Technology Museum?","fineKeywords":[["It consists of five main exhibition halls: Heaven and Earth Hall, Life Hall, Wisdom Hall, Creation Hall, and Future Hall, along with a temporary exhibition hall"]],"referenceAnswer":"The Shanghai Science and Technology Museum consists of five main exhibition halls: Heaven and Earth Hall, Life Hall, Wisdom Hall, Creation Hall, and Future Hall, along with a temporary exhibition hall.","coarseKeywords":["Shanghai Science and Technology Museum"],"queryType":"Factual"}
  {"query":"What is the difference in the shopping experience between Wangfujing and Sanlitun in Beijing?","fineKeywords":[["Go to Wangfujing to browse its various large department stores"],["If you want more, head to the large, trendy Sanlitun district in the afternoon for a shopping spree"],["For lunch in Wangfujing, you can eat in the major department stores, which have plenty of popular restaurants, or visit Wangfujing Snack Street for authentic local treats like tripe, bean jelly, and squid skewers to delight your palate"],["In the evening, you can eat, shop, drink, and have fun in Sanlitun"]],"referenceAnswer":"Wangfujing and Sanlitun offer different shopping experiences. Wangfujing is dominated by large department stores that offer a wide range of brands for mass consumption. Sanlitun, on the other hand, is a trendy district that focuses on fashion and uniqueness, featuring many international brands and designer boutiques. For dining, Wangfujing Snack Street provides a rich variety of authentic Beijing snacks, such as tripe and bean jelly. In contrast, Sanlitun boasts more upscale restaurants and stylish bars, offering a diverse nightlife. Overall, Wangfujing is more traditional and mainstream, while Sanlitun represents the energy and avant-garde style of a modern city.","coarseKeywords":["Wangfujing","Sanlitun"],"queryType":"Comparative"}

Automatic generation

Automatically generate an evaluation set using a large language model and a specified knowledge base. This method supports only knowledge-based Q&A evaluation tasks. For more information, see Automatic evaluation.

Manual upload

Manual upload supports both conversation analysis and knowledge-based Q&A evaluation sets.

1. Prepare the evaluation set file. The format requirements are as follows:

   Evaluation set type	File format	Description
   Conversation analysis	.xls, .xlsx	You can upload up to 10 files at a time. Each file cannot exceed 20 MB.
   Knowledge-based Q&A	.jsonl	You can upload up to 10 files at a time. Each file cannot exceed 20 MB.

2. Go to the Evaluation sets page and click Create Evaluation Set in the upper-right corner.

3. Enter a name for the evaluation set, select the evaluation set type, upload the file, and click Confirm.