Training set and evaluation set

Supported datasets

Data Management manages all large model datasets in your workspace. These datasets are categorized into two types: Training Set (for model fine-tuning) and Evaluation Set (for model evaluation).

# A typical line of expanded JSON training data:
{"messages": [
  {"role": "system", "content": "System input 1"}, 
  {"role": "user", "content": "User input 1"}, 
  {"role": "assistant", "content": "Desired model output 1"}, 
  {"role": "user", "content": "User input 2"}, 
  {"role": "assistant", "content": "Desired model output 2"}
  ...
]}

 {"role": "assistant", "content": "Desired model output 1", "loss_weight": 1.0}, 
 {"role": "assistant", "content": "Desired model output 2", "loss_weight": 0.5}

# A single line of training data (JSON format) has the following typical structure when expanded:
{"messages": [
  {"role": "system", "content": "System input 1"}, 
  {"role": "user", "content": "User input 1"}, 
  {"role": "assistant", "content": "Model output 1"}, --Intermediate assistant outputs must not include <think> tags.
   ...
  {"role": "user", "content": "User input 2"}, 
  {"role": "assistant", "content": "<think>\nDesired thinking content 2\n</think>\n\nDesired output 2"} --Include thinking content only in the final assistant output. 
]}

{"role": "assistant", "content": "Desired model output 2"}  --Instructs the model not to enable thinking.

 {"role": "assistant", "content": "<think>\nDesired thinking content 2\n</think>\n\nDesired output 2", "loss_weight": 1.0}

# A typical line of expanded JSON training data:
{"messages": [
  {"role": "system", "content": [{"text": "System input"}]},
  {"role": "user", "content": [{"text": "User input 1"}, {"image": "image_filename_1.jpg", "resized_width": 200, "resized_height": 200}]},
  {"role": "assistant", "content": [{"text": "Desired model output 1"}]},
  {"role": "user", "content": [{"text": "User input 2"}, {"video": "video_filename_1.mp4", "fps": 3.0, "resized_width": 200, "resized_height": 200, "video_start": 0.0, "video_end": 3.0}]},
  {"role": "assistant", "content": [{"text": "Desired model output 2"}]},
  {"role": "user", "content": [{"text": "User input 2"}, {"video": ["0.jpg", "1.jpg", "2.jpg", "3.jpg"], "sample_fps": 5.0, "resized_width": 200, "resized_height": 200}]},
  {"role": "assistant", "content": [{"text": "Desired model output 2"}]},
  ...
]}

# A typical line of expanded JSON training data:
{"messages": [
  {"role": "system", "content": "System input"},
  {"role": "user", "content": "User input 1"},
  {"role": "assistant", "content": "Model output 1"},
  {"role": "user", "content": "User input 2"},
  {"role": "assistant", "content": "Model output 2"},
  {"role": "user", "content": "User input 3"}
 ],
 "chosen":
   {"role": "assistant", "content": "The chosen model output for Input 3"},
 "rejected":
   {"role": "assistant", "content": "The rejected model output for Input 3"}}

{"role": "assistant", "content": "<think>Desired model thinking content</think>Desired model output"}

 "chosen":
   {"role": "assistant", "content": "The chosen model output for Input 3", "loss_weight": 1.0},

{"text":"Text content"}

Dataset creation tips

Create a dataset

This section explains how to create a dataset on the Alibaba Cloud Model Studio console.

Alibaba Cloud Model Studio currently does not limit the number of datasets you can create or the volume of data you can import.

1. Navigate to the Datasets page and click Add Dataset.

2. Enter a Dataset Name and an optional Dataset Description, and then select a Dataset Type.

   Training set

   1. For Data Format, select a Training Scenario and Training Method based on your training requirements. The available training methods depend on the selected training scenario.

   2. For Storage Location, select Platform OSS Storage or Mount Cloud Storage.

      When you select Platform OSS Storage, you use the free storage space provided by Alibaba Cloud Model Studio. Currently, there is no limit on data import volume.

   3. For Import Method, select Upload Local File, Import from OSS, or Log Backflow.

      To ensure data security, the platform encrypts your imported dataset using OSS server-side encryption (SSE-OSS), which employs the AES-256 algorithm with OSS-managed keys.

      When you select Import from OSS, you must first add the tag bailian-datahub-access=read to the target bucket. Then, on the console, select the corresponding bucket and file. You can only select files that match the required data format.

   4. For Upload File, click the icon to select and upload files. Your uploaded training data must match the structure of the provided data example. Otherwise, the import will fail.

      You can use the data template provided by Alibaba Cloud Model Studio. Replace the sample data with your training data and then upload the file.

      You cannot create empty training sets in Alibaba Cloud Model Studio.

      For the differences between system, user, and assistant roles, see Overview.

      For SFT - Text Generation and DPO - Text Generation training sets, you can upload multiple files at once. Alibaba Cloud Model Studio merges and imports them into a single training set.

   5. For Publish Settings, select Publish Now or Save as Draft. Selecting Save as Draft creates the dataset with a Draft status. Selecting Publish Now creates and publishes the dataset.

      CPT - Text Generation and I2V training sets do not support the Draft status and can only be published immediately.

      Publish status	Description
      Draft	A dataset in the Draft status can be edited online and can be used for tasks like data cleansing or data augmentation. However, it cannot be used for model fine-tuning or model evaluation. You must publish the dataset to use it for these tasks.
      Publish	A dataset in the Published status cannot be edited online. It can be used for data cleansing or data augmentation, model fine-tuning (for training sets), and model evaluation (for evaluation sets).

   Evaluation set

   1. For Dataset Type, select Text Generation. Other types are not currently supported.

   2. For Storage Location, select Platform OSS Storage.

      This option uses the free storage space provided by Model Studio. Currently, there is no limit on data import volume.

   3. For Import Method, select Upload Local File. Importing from OSS is not currently supported for evaluation sets.

   4. For Upload File, click the icon to select and upload files. Your uploaded evaluation data must match the structure of the provided data example. Otherwise, the import will fail.

      You can use the data template provided by Alibaba Cloud Model Studio. Replace the sample data with your evaluation data and then upload the file.

      You cannot create empty evaluation sets in Model Studio.

      Model Studio merges and imports the data from these files into a single evaluation set.

3. Click Confirm. The new dataset (version V1) appears in the Datasets list, and the import process begins automatically. Click the icon to view the latest Import Status.

   Import status	Description
   Importing	This process may take a long time during peak hours. No further action is required.
   Import succeeded	The dataset was imported successfully.
   Import failed.	The dataset creation failed.

Manage datasets

Billing

Data management and dataset storage are free.

API

No APIs are currently available for the data management feature of Alibaba Cloud Model Studio.

Next steps

• Once a Training Set is published, it can be used for model tuning.

• Once an Evaluation Set is published, it can be used for model evaluation.