Synchronize OpenLake-DLF data to Alibaba Cloud Elasticsearch

1. Navigate to the synchronization task creation page

1. Log on to the Elasticsearch console. In the top menu bar, switch to the target region.

2. Find the target instance and click the Cluster ID to go to the Basic Information page.

3. In the left-side navigation pane, click Data Processing and Sync.

4. In the Multimodal Data Processing Service section, select Workspace Name of Model Service, and then click Initialize Model. After initialization is complete, click Get Started.

   Note

   Workspace Name of Model Service: An existing space in the AI Search Open Platform. The space named default is the default space provided by the AI Search Open Platform. Other spaces (if any) are created by you.

   Initialize Model: Initializes the models in the selected space to make them available. If the space already shows the Activated status, you can skip Initialize Model and directly click Get Started.

5. On the synchronization task list page, click Create.

2. Configure basic information

This step configures the task name and establishes a connection with the DLF data source. Follow the on-screen instructions to configure Basic Information Configuration.

After you complete the configuration, click Next.

3. Configure data processing information

1. In the Original Field section, select the fields that you want to synchronize to ES.

2. In the Data Processing Configurations section, select fields and configure processing rules. For example, you can specify vector models for text and image fields. The system automatically generates vector data and synchronizes the data to ES.

4. Configure field mappings

This step establishes the mapping between source fields and ES target index fields.

ES information configuration: Enter the Username, Password, and Target Index for connecting to ES.

The system automatically loads all indexes under the current ES instance. If the target index does not appear in the Target Index drop-down list, manually create the index. The following example creates an index:

PUT /your_index_name
{
  "settings": {
    "index": {
      "number_of_shards": 1,         // Number of primary shards
      "number_of_replicas": 1,       // Number of replicas for each primary shard
      "refresh_interval": "30s",     // Data refresh interval (delay before written data becomes searchable)
      "analysis": {                  // Analyzer configuration (example)
        "analyzer": {
          "default": {
            "type": "standard"
          }
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "id": {
        "type": "keyword"
      },
      "name": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "url": {
        "type": "keyword"
      },
      "source": {
        "type": "keyword"
      },
      "embedding": {
        "type": "dense_vector",
        "dims": 768
      }
    }
  }
}

The source field types must be compatible with the ES target field types. Otherwise, an error (error code 3005) is returned when you submit the task. The following describes common compatibility relationships:

• Source scalar types (STRING, INT, FLOAT, etc.): Can be directly mapped to ES scalar fields such as keyword, text, long, and float.

• Synchronizing source scalar fields to ES dense_vector fields: You must first configure a vectorization processing rule for the field in the Configure data processing information step (the AI model generates a vector array). Direct mapping by name is not supported.

• Source ARRAY type: When mapped to an ES dense_vector field, the array dimensions must match the dims defined in the ES index.

  On the Field Mapping Configuration tab, complete the following configurations:

Configure field mapping relationships

The system automatically matches fields based on the same-name principle. You can also manually adjust the mappings.

• Fields to Sync: The original fields from the data source that need to be synchronized.

• Target Index Field: The target fields in the ES index that receive the data.

• Primary Key: Select a field as the unique identifier (_id) of documents in ES.

  Only a single primary key is supported. If your source table uses a composite primary key, create a new column in the source to concatenate the composite primary key into a unique string ID, and use that column as the synchronization primary key.

After you confirm that all fields to be synchronized are correctly mapped, click Next.

5. Configure and start the synchronization task

This step confirms the synchronization policy configuration and starts the task. On the Data Synchronization Configuration tab, confirm that the configuration is correct and click Complete to start the task.