Synchronize OpenLake-DLF data to Alibaba Cloud Elasticsearch

Synchronization mechanism

The synchronization task uses a full synchronization plus incremental synchronization mechanism. After a task is started, it automatically reads data in Paimon format from the DLF source table, applies data processing rules to vectorize text and images, converts the data into ES index format, and accelerates data retrieval in the data lake through ES.

Billing

• No fees are charged for data synchronization tasks. After data is synchronized to ES, storage fees for node storage space are incurred. You can view billing details in the Expenses and Costs console.

• AI model invocation fees: The AI models invoked during data processing (such as text vectorization and image vectorization) are provided by the AI Search Open Platform. Fees are charged based on the actual model invocation volume. No fees are charged if no models are invoked.

Create and configure a synchronization task

The following steps guide you through the complete configuration process from creating a task to starting synchronization.

1. Navigate to the synchronization task creation page

   This step guides you to the feature entry point.

   1. Log on to the Elasticsearch console. In the top navigation bar, switch to the target region.

   2. Find the target instance and click the Cluster ID to go to the Basic Information page.

   3. In the left-side navigation pane, click Data Processing and Sync.

   4. In the Multimodal Data Processing Service section, select Workspace Name of Model Service, and then click Initialize Model. After initialization is complete, click Get Started.

      Workspace Name of Model Service: An existing space in the AI Search Open Platform. The default space is the default space of the AI Search Open Platform, and other spaces (if any) are created by the user.

      Initialize Model: Initializes the models in the selected space to make them available. If the space already shows the Activated status, you can skip Initialize Model and directly click Get Started.

   5. On the synchronization task list page, click Create.

2. Configure basic information

   This step configures the task name and establishes a connection with the DLF data source. Follow the on-screen instructions to configure Basic Information Configuration.

   Parameter	Description and recommendation
   Task ID/Name	The name of the synchronization task. Use a name that is easy to identify.
   API Key	You must create an API key in the AI Search Open Platform in advance.
   Data Source	Select Data Lake Formation (DLF).
   Table Type	The DLF data source only supports data tables in Paimon format. This option is selected by default and does not require manual selection.
   Data Catalog	Set based on your actual requirements.
   Database
   Data Table
   RAM Role	Authorize ES to use AI Search Open Platform Default Role. When you perform this operation, a service-linked role is automatically created to enable the corresponding feature. Role name: AliyunServiceRoleForSearchPlat Role permission policy: AliyunServiceRolePolicyForSearchPlat Description: Allows the AI Search Open Platform service to access your resources.

   After you complete the configuration, click Next.

3. Configure data processing information

   • In the Original Field section, select the fields that you want to synchronize to ES.

   • In the Data Processing Configurations section, select fields and configure processing rules. For example, you can specify vector models for text and image fields. The system automatically generates vector data and synchronizes the data to ES.

4. Configure field mappings

   This step establishes the mapping between source fields and ES target index fields.

   ES information configuration: Enter the Username, Password, and Target Index for connecting to ES.

   Important

   The system automatically loads all indexes under the current ES instance. If the target index does not appear in the Target Index drop-down list, manually create the index by referring to the following example.

   PUT /your_index_name
   {
     "settings": {
       "index": {
         "number_of_shards": 1,         // Number of primary shards
         "number_of_replicas": 1,       // Number of replicas for each primary shard
         "refresh_interval": "30s",     // Data refresh interval (delay before written data becomes searchable)
         "analysis": {                  // Analyzer configuration (example)
           "analyzer": {
             "default": {
               "type": "standard"
             }
           }
         }
       }
     },
     "mappings": {
       "properties": {
         "id": {
           "type": "keyword"
         },
         "name": {
           "type": "text",
           "fields": {
             "keyword": {
               "type": "keyword",
               "ignore_above": 256
             }
           }
         },
         "url": {
           "type": "keyword"
         },
         "source": {
           "type": "keyword"
         },
         "embedding": {
           "type": "dense_vector",
           "dims": 768
         }
       }
     }
   }

   Important

   The source field types must be compatible with the ES target field types. Otherwise, an error (error code 3005) is returned when you submit the task. The following describes common compatibility relationships:

   • Source scalar types (STRING, INT, FLOAT, etc.): Can be directly mapped to ES scalar fields such as keyword, text, long, and float.

   • Synchronizing source scalar fields to ES dense_vector fields: You must first configure a vectorization processing rule for the field in the Configure data processing information step (the AI model generates a vector array). Direct mapping by name is not supported.

   • Source ARRAY type: When mapped to an ES dense_vector field, the array dimensions must match the dims defined in the ES index.

   On the Field Mapping Configuration tab, complete the following configurations:

   1. Configure field mapping relationships

      The system automatically matches fields based on the same-name principle. You can also manually adjust the mappings.

      • Fields to Sync: The original fields from the data source that need to be synchronized.

      • Target Index Field: The target fields in the ES index that receive the data.

      • Primary Key: Select a field as the unique identifier (_id) of documents in ES.

      Currently, only a single primary key is supported. If your source table uses a composite primary key, we recommend that you create a new column in the source to concatenate the composite primary key into a unique string ID and use that column as the synchronization primary key.

   2. After you confirm that all fields to be synchronized are correctly mapped, click Next.

5. Configure and start the synchronization task

   This step confirms the synchronization policy configuration and starts the task. On the Data Synchronization Configuration tab, confirm that the configuration is correct and click Complete to start the task.

   Note

   After you click Complete, go back to the synchronization task list page to confirm that the task is successfully created. If the new task does not appear in the list, the backend verification may have failed. Common causes include:

   • Incompatible field types (see the type compatibility description in the Configure field mappings step).

   • Processing rules are not configured for dense_vector and other vector fields in the Configure data processing information step.

   • The source data catalog, database, or data table names are incorrect or do not exist.

   You can check the error codes and details returned by the API in the Network tab of browser developer tools, or submit a ticket for assistance.

Manage and monitor synchronization tasks

After a task is created, you can manage and monitor it on the synchronization task list page.

• View task status: Task statuses include Scheduled, Running, and Run Failed.

• Management operations:

  • Create Copy: Quickly copies the configuration of an existing task. This is useful for creating similar tasks in batches.

  • Delete: Deletes a task that is no longer needed. Deleted tasks cannot be recovered. Proceed with caution.