Use Data Exchange datasets for data annotation
You can use Data Exchange to create datasets or save datasets that you subscribe to from the Alibaba Cloud Marketplace as personal datasets. This topic describes how to use the data application feature of Data Exchange to scan and parse dataset files. This process generates the manifest files required for image, video, audio, or text annotation in PAI-iTAG.
Select a source dataset for data annotation
On the Owned Datasets page in Data Exchange, select the dataset to annotate. Then, click Data Application in the Actions column.
Select the dataset version to annotate. The latest version is selected by default. You can switch to another version if needed.
For the data usage scenario option, select "Data Annotation".
Select or create a PAI workspace
Data annotation relies on the Platform for AI - iTAG (PAI-iTAG) service. First, you must activate the PAI service. PAI-iTAG is an intelligent data annotation platform that supports annotation for various data types, such as images, text, videos, and audio, and also supports multi-modal annotation. A free trial of PAI-iTAG is available.
Select a PAI workspace in the same region as the OSS Bucket that is associated with the data version. If you have not created a workspace in the current region, you can use "Quickly Create Workspace" to create one.
If you log on as a RAM user or using a role, you must be a member of the PAI workspace to select it. For more information, see Manage Workspace Members.
Generate an image, video, or audio annotation manifest file and sync it to PAI-iTAG
In the data application feature, set the data type to sync to "Image/Video/Audio". Select the folder that contains the image, video, or audio files you want to annotate from the current dataset version. You can use a wildcard character to filter the files in the folder by format.
Click "Load Data Display" to preview the list of files to be annotated. The preview is displayed in the manifest data format that PAI-iTAG requires. The manifest format for image, audio, and video annotation includes only the file paths of the files to be annotated, not the file content. For more information, see Appendix: Data annotation format.
Set the folder and filename for the manifest annotation file. By default, the file is saved to the dataset_input folder in the OSS Bucket where the dataset version is stored.
Click Submit. The data application feature saves the manifest file to the specified location and creates a PAI-iTAG dataset. After the dataset is created, you are redirected to the PAI-iTAG dataset management page.
PAI-iTAG currently supports annotation for the following formats: image (jpg, cur, webp, png, svg, gif, bmp, jpeg, ico, web), video (mp4), and audio (mp3). Other formats may not be annotated correctly.
Generate a text annotation manifest file and sync it to PAI-iTAG
In the data application feature, set the data type to sync to "Text". Then, select the CSV or Excel file you want to annotate from the current dataset version.
Click "Load Data Display" to preview the parsed file content. The preview is displayed in the manifest data format that PAI-iTAG requires. The manifest format for text annotation includes the parsed file content. For more information, see Appendix: Data annotation format.
Set the folder and filename for the manifest annotation file. By default, the file is saved to the dataset_input folder in the OSS Bucket where the dataset version is stored.
Click Submit. The data application feature saves the manifest file to the specified location and creates a PAI-iTAG dataset. After the dataset is created, you are redirected to the PAI-iTAG dataset management page.
This feature currently supports parsing CSV and Excel files and converting them to the manifest annotation format. For other formats, you must parse the files and generate the manifest format yourself.
Annotate the dataset in PAI-iTAG
In PAI-iTAG, create a data annotation task. Associate the PAI-iTAG dataset that you created in the previous step. Select an annotation configuration type. Configure options such as the task assignment method, annotators, annotation flow, and intelligent annotation services. Then, start the data annotation task. For more information, see Create an annotation task.
Sync PAI annotation results to a Data Exchange dataset
For completed annotation tasks, go to the PAI-iTAG Task Hub. Click "Get Annotation Results" to export the task as a labeled dataset. Set the OSS output location to export the annotation results to OSS in the manifest format. You can view the resulting labeled dataset on the Labeled Datasets tab of the Data Management page. For more information, see Export annotation result data.
On the Labeled Datasets tab, select the dataset to sync to Data Exchange. In the Actions column, click "Sync to Data Exchange". This creates a Data Exchange dataset that points to the annotation result manifest file.
On the Owned Datasets page in Data Exchange, view the annotation result dataset that you synced in the previous step. Click Continue to Create. Edit the version name, description, and file type. Save your changes to finish creating the dataset. You can then use the dataset for other data applications or sell it on the Alibaba Cloud Marketplace.
In the default version of the labeled dataset that is synced to Data Exchange, the annotation result manifest file from OSS is already associated. This association cannot be modified.