Manage and use multimodal data

3.1.1 Activate PAI and grant permissions

1. Use your root account to activate PAI and create a workspace. Log on to the PAI console, select a Region in the upper-left corner, and then authorize and activate the product.

2. Authorize your account. You can skip this step if you use a root account. If you use a RAM user, you must have the workspace administrator role. For more information about account authorization, see the Member role configuration section in Create and manage workspaces.

3.1.2 Get the Model Studio API key

To activate Model Studio and create an API key, see Obtain an API key.

3.1.3 Create a vector database

Create a vector database instance

Multimodal dataset management currently supports the following Alibaba Cloud vector databases:

• Elasticsearch (Vector Search Edition, version 8.17.0 or later)

• OpenSearch (Vector Search Edition)

• Milvus (version 2.4 or later)

• Hologres (version 4.0.9 or later)

• Lindorm (Vector Engine Edition)

To create an instance for each vector database, see the documentation for the respective cloud product.

Configure network and whitelist settings

• Public network access

  If your vector database instance has a public endpoint, add the following IP addresses to the instance's public network access whitelist. This allows the multimodal data management service to access the instance over the public network. To configure an Elasticsearch whitelist, see Manage IP address whitelists for public or private network access.

  Region	IP addresses
  China (Hangzhou)	47.110.230.142, 47.98.189.92
  China (Shanghai)	47.117.86.159, 106.14.192.90
  China (Shenzhen)	47.106.88.217, 39.108.12.110
  China (Ulanqab)	8.130.24.177, 8.130.82.15
  China (Beijing)	39.107.234.20, 182.92.58.94

• Private network access

  To apply for this option, submit a ticket.

Create a vector index table (Optional)

The system automatically creates an index table. You can skip this step unless you need to create a custom one.

In some vector databases, a vector index table is also called a collection or an index.

The schema for the index table must match the following definition:

{
    "id":"text",                    // Primary key ID. Required for OpenSearch; exists by default in other databases and does not need to be defined.
    "index_set_id": "keyword",      // Index set ID. Must support indexing.
    "file_meta_id": "text",         // File metadata ID.   
    "dataset_id": "text",           // Dataset ID.
    "dataset_version": "text",      // Dataset version.
    "uri": "text",                  // URI of the OSS file.
    "file_vector": {                // Vector field.
        "type": "float",            // Vector type: float.
        "dims": 1536,               // Vector dimensions (customizable).
        "similarity": "DotProduct"  // Vector distance algorithm: cosine or dot product.
    }
}

This section provides a Python example for creating a semantic index table in Elasticsearch. For other vector databases, see their respective product documentation.

from elasticsearch import Elasticsearch

# 1. Connect to the Alibaba Cloud Elasticsearch instance.
# Note:
# (1) Python 3.9 or later is required: python3 -V
# (2) The Elasticsearch client must be version 8.x: pip show elasticsearch
# (3) If using a VPC endpoint, the caller must be in a VPC that can communicate with the Elasticsearch instance's VPC. Otherwise, use a public endpoint and add the caller's public IP address to the Elasticsearch whitelist.
# The default username is elastic.
es_client = Elasticsearch(
    hosts=["http://es-cn-l4p***5z.elasticsearch.aliyuncs.com:9200"],
    basic_auth=("{userName}", "{password}"),
)

# 2. Define the index name and structure. The HNSW index algorithm is used by default.
index_name = "dataset_embed_test"
index_mapping = {
    "settings": {
        "number_of_shards": 1,          # Number of shards
        "number_of_replicas": 1         # Number of replicas
    },
    "mappings": {
        "properties": {
            "index_set_id": {
                "type": "keyword"
            },
            "uri": {
                "type": "text"
            },
            "file_meta_id": {
                "type": "text"
            },
            "dataset_id": {
                "type": "text"
            },
            "dataset_version": {
                "type": "text"  
            },
            "file_vector": {
                "type": "dense_vector",  # Define file_vector as a dense vector type
                "dims": 1536,            # Vector dimensions: 1536
                "similarity": "dot_product"  # Similarity calculation method: dot product
            }
        }
    }
}

# 3. Create the index.
if not es_client.indices.exists(index=index_name):
    es_client.indices.create(index=index_name, body=index_mapping)
    print(f"Index {index_name} created successfully!")
else:
    print(f"Index {index_name} already exists. It will not be created again.")

# 4. View the schema of the created index table (Optional).
# indexes = es_client.indices.get(index=index_name)
# print(indexes)

3.3.1 Create a smart tagging connection

1. Go to your PAI workspace. In the left-side navigation pane, choose AI Asset Management > Connection > Model Service > Create Connection.

2. Select Alibaba Cloud Model Studio Service and configure the Model Studio API key.

3. After the connection is created, the Alibaba Cloud Model Studio Service appears in the list.

3.3.2 Create a semantic indexing connection

1. If you plan to use the Model Studio Semantic Indexing model service, you can skip this step. In the left-side navigation pane, click Model Gallery, find and deploy a GME multimodal retrieval model to obtain an EAS service. The deployment takes about 5 minutes. The deployment is complete when the status changes to Running.

   Important

   When you no longer need the index model, stop and delete the service to avoid further charges.

   You can select the GME-2B retrieval model (2B parameters) or the GME-7B retrieval model and click the corresponding Deploy button to start the deployment.

2. Go to your PAI workspace. In the left-side navigation pane, choose AI Asset Management > Connection > Model Service > Create Connection.

3. Configure the model connection information based on whether you chose the Model Studio Semantic Indexing model or a custom-deployed EAS semantic indexing model.

   Model Studio

   • Connection Type: Select General Multimodal Embedding Model Service.

   • Service Provider: Select Third-party Model Service.

   • Model Name: tongyi-embedding-vision-plus

   • base_url:https://dashscope.aliyuncs.com/api/v1/services/embeddings/multimodal-embedding/multimodal-embedding

   • API key: Obtain an API key and enter it.

   Custom EAS

   • Connection Type: Select General Multimodal Embedding Model Service.

   • Service Provider: Select PAI-EAS Model Service.

   • EAS Service: Select the GME multimodal retrieval model that you just deployed. If the service provider is not in your account, select Third-party Model Service.

   In the Select EAS service dialog box, select the target model service whose status is Running.

4. After the connection is created, the model connection service appears in the list.

3.3.3 Create a vector database connection

1. In the left-side navigation pane, choose AI Asset Management > Connection > Database > Create Connection.

2. The multimodal retrieval service supports vector databases such as Milvus, Lindorm, OpenSearch, Elasticsearch, and Hologres. This section uses Elasticsearch as an example to describe how to create a database connection. Select Elasticsearch and configure parameters such as uri, username, and password. For more information, see Create a database connection.

   The connection formats for each vector database are as follows:

   Milvus

   uri: http://xxx.milvus.aliyuncs.com:19530 
   database: {your_database} 
   token: root:{password}

   OpenSearch

   uri: http://xxxx.ha.aliyuncs.com
   username: {username} 
   password: {password}

   Hologres

   host: xxxx.hologres.aliyuncs.com
   database: {your_database} 
   port: {port}
   access_key_id={password}

   Elasticsearch

   uri: http://xxxx.elasticsearch.aliyuncs.com:9200
   username: {username} 
   password: {password}

   Lindorm

   uri: xxxx.lindorm.aliyuncs.com:{port}
   username: {username} 
   password: root:{password}

3. After the connection is created, you can see the vector database connection in the list.

3.4.1 Create a smart tag definition

In the left-side navigation pane, choose AI Asset Management > Datasets > Intelligent Tag Definition > Create Intelligent Tag Definition to open the tag configuration page. The following is a configuration example:

• Guide Prompt: You are a seasoned driver with extensive experience on both highways and urban roads.

• Tag Definition:

  Sample tag definition for autonomous driving

  {
      "Reflective strips": "Usually yellow, or alternating yellow and black, attached to corners and other permanent protruding obstacles to alert drivers to avoid them. They are strip-shaped, not traffic cones, parking locks, or water-filled barriers.",
      "Parking locks": "Also known as parking space locks, they can be raised to prevent a parking space from being occupied. If a parking lock is present, you must specify whether it is in the raised or lowered state. It is in the raised state if it has a raised frame, otherwise it is in the lowered state.",
      "Lit construction vehicles": "The target is a vehicle with two arrow-shaped lights on the left and right that are lit. Otherwise, it is not considered a lit construction vehicle.",
      "Overturned vehicles": "A vehicle that has overturned on the ground.",
      "Fallen water-filled barriers": "A water-filled barrier is a plastic shell obstacle used to divide road surfaces or form a blockage, typically in the form of a red plastic wall. It is commonly used in road traffic facilities and is often seen on highways, urban roads, and at overpass intersections. It is significantly larger than a traffic cone and has a sheet-like structure. Water-filled barriers are normally upright. If one is lying on the ground, it must be clearly indicated.",
      "Fallen traffic cones": "Also known as conical traffic markers or pylons, commonly called road cones or safety cones, they are cone-shaped temporary road signs. Obstacles that are rod-shaped or sheet-like are not traffic cones because they are not conical. A traffic cone may be knocked over by a car. If a traffic cone is present in the image and you need to determine if it has fallen, observe whether the bottom of the cone (the base of the cone) is in contact with the ground. If it is, it has not fallen. Otherwise, it has.",
      "Charging spaces": "A parking space against a wall with a visible charging gun, charging pile equipment, or marked as a new energy vehicle space is a charging space. This tag only applies to spaces within a parking lot (indoor or outdoor). Note that parking locks are not related to charging.",
      "Speed bumps": "Usually yellow and black, or just yellow, these are narrow raised strips across the road, perpendicular to the road edge, used to slow down vehicles. They are not found within parking spaces.",
      "Deceleration lane lines": "Dashed lines in a fishbone pattern on both sides of the lane, inside the solid lines. Both sides must have them to be considered deceleration lane lines.",
      "Ramps": "This tag should only be used when a clear, large curve on a highway is visible. Ramps are usually on the right side of the main highway and are used to enter or exit toll stations.",
      "Ground shadows": "There are clear shadows on the ground.",
      "Cloudy": "This tag should only be used if the sky is visible and contains clear clouds.",
      "Glaring car": "The lights of a car ahead are causing glare (the light changes from a single point to a line of light), usually occurring at night or on rainy days.",
      "Left turn, right turn, U-turn arrows": "Milky white arrow markings painted on the road surface (a few are yellow), not the green and white arrows on highway signs indicating a right curve. When determining the presence of these arrows, only clear arrow markings in the middle of the road lane are the target. Those on the roadside are not. If there are arrows on the ground, the direction is determined as follows: a right-turn arrow rotates clockwise from the base to the tip; a left-turn arrow rotates counter-clockwise from the base to the tip; a U-shaped arrow is a U-turn arrow.",
      "Crosswalks": "This tag applies to crosswalks on road surfaces (including in parking lots) and at intersections. They must be white lines distributed at repeated intervals parallel to the road edge for pedestrian crossing. They are not found on highways, highway ramps, or in tunnels.",
      "Overexposure": "During the day, direct sunlight causes the lens to be overexposed (can only happen during the day).",
      "Motor vehicles": "There are other motor vehicles in the field of view.",
      "Merging in or out": "A place on a highway where multiple lanes merge into one, or one lane divides into multiple lanes.",
      "Intersections": "An intersection where there are no lane lines within the intersection area (refers to the absence of lines within the intersection itself; lines outside the intersection do not matter).",
      "No parking signs": "A sign, either hanging or standing on the ground, with the words 'No Parking' or a symbol of a 'P' in a circle with a diagonal line through it.",
      "Lane lines": "Lane lines on the road, with special attention to blurry lane lines.",
      "Fallen rocks or tires on the road": "Obstacles on the road that affect traffic.",
      "Tunnels": "Note the distinction between entering and exiting a tunnel.",
      "Wet ground on a rainy day": "The ground is slippery due to rain.",
      "Non-motorized vehicles": "Includes non-motorized objects such as bicycles, electric bikes, wheelchairs, unicycles, and shopping carts, which may be parked on the roadside, in parking spaces, or moving on the road."
    }

3.4.2 Create an offline smart tagging task

1. Click Custom Dataset, click a dataset name, and then click the Dataset jobs tab.

2. On the jobs page, click Create job > Smart tag, and configure the task parameters.

   • Dataset Version: Select the version to label, such as v1.

   • Labeling Model Connection: Select the Model Studio model connection that you created.

   • Smart Labeling Model: Supported models include Qwen-VL-Max and Qwen-VL-Plus.

   • Max Concurrency: This value depends on the specifications of the EAS model service. For a single GPU, the recommended maximum concurrency is 5.

   • Intelligent Tag Definition: Select the Smart Tag Definition that you just created.

   • Labeling Mode: Supports Increment and Full modes.

3. After the Smart Tagging task is created, it appears in the task list. You can click the links in the Actions column to view logs or stop the task.

   Note

   When you start a Smart Tagging task for the first time, the system builds the metadata. This process may take a long time.

3.9.1 Export as a file

1. Click Export as file. On the configuration page, set the export content and the destination OSS directory, and then click OK.

   On the export configuration page, for Export Content, you can select Text, Image, Audio, Video, and Human Labels. For Export Type, select Export File, and set the Export Path to the destination OSS directory.

2. To view the export progress, choose AI Asset Management > Job > Dataset jobs in the left-side navigation pane.

3. Use the exported results. After the export is complete, you can mount the exported result file and the original dataset to the appropriate training environment, such as a DLC or DSW instance. You can then write code to read the exported result file index and load the target files from the original dataset for model training or analysis.

3.9.2 Export to a logical dataset version

You can import the retrieval results of a Premium dataset into a version of another logical dataset. You can then use the data in that logical dataset version with the dataset SDK.

1. Click Export to logical dataset version, select the target logical dataset, and then click Confirm.

   In the export dialog box, select the Target Dataset and Base Dataset Version, and set the Import Scope (all files or selected files). For Import Mode, you can select Merge and Overwrite and select the tag types to import (Smart tags or Human labels).

   If no logical dataset is available for selection, create one as described in the following section:

   Create a logical dataset

   Create a logical dataset. In the left-side navigation pane, choose AI Asset Management > Dataset > Create Dataset, and then configure the following key parameters. Configure other parameters as needed.

   • Dataset Type: Select Logical.

   • Metadata OSS path: Select an OSS path for the export.

   • Import method: Select Import later.

   Click OK to create the dataset.

2. Use the logical dataset. After the import task is complete, the target logical dataset contains the exported metadata. You can use the SDK to load and use the data. For information about how to use the SDK, see the dataset details page.

   from pai_datasets.load import load_dataset
   
   dataset = load_dataset(dataset_id="xxx",
       dataset_version="v1",
       region="cn-hangzhou",
       cache_dir=CACHE_DIR,
       keep_in_memory=True)

   To install the SDK, run the following command:

   pip install https://pai-sdk.oss-cn-shanghai.aliyuncs.com/dataset/pai_dataset_sdk-1.0.0-py3-none-any.whl