DocsICP FilingConsole
官方文档
首页

AI-assisted processing for batch synchronization tasks

更新时间：
复制 MD 格式
产品详情
我的收藏

This feature integrates large AI model processing directly into the DataWorks Data Integration pipeline, upgrading data synchronization from simple transport to intelligent processing. You can call AI models in real time to analyze, process, and enhance data as it moves from source to destination, unlocking the hidden value of unstructured data.

Features

  • Target users: This feature targets enterprise users who need advanced analysis and processing during data synchronization, especially those using AI to improve data quality and extract value from data.

  • Seamless integration: AI processing is a built-in Data Integration step that connects seamlessly with reading from a source and writing to a destination.

  • Support for various NLP tasks: You can perform multiple natural language processing (NLP) tasks on text data during synchronization, such as sentiment analysis, summary generation, keyword extraction, and text translation.

Use cases

Industry

Typical application

Customer service / E-commerce

Analyze the sentiment of user comments and customer service tickets in real time. Automatically extract core issues and key feedback points.

Compliance / Legal / Scientific research

During synchronization, automatically generate summaries and extract key information from policy documents, legal contracts, and research papers.

Manufacturing / Supply chain / Healthcare

Intelligently analyze device logs, supply chain feedback, or doctor-patient communication records to enable risk alerts and service quality optimization.

Cross-language collaboration

Automatically translate social media comments, news articles, or business documents from around the world into a single language during synchronization to facilitate centralized analysis.

Prerequisites

  • You have created a workspace that uses Data Studio (new version).

  • You have prepared the large model service required for AI-assisted processing. Preparation steps depend on your selected model provider:

  • For batch synchronization tasks, you can configure data source details manually or use existing data sources.

  • Ensure that the workspace is bound to a resource group and that the resource group can connect to the data sources.

Billing

For Data Integration tasks that use AI-assisted processing, in addition to the standard Data Integration fees, you also incur costs for calling the large model. The billing details are as follows:

Example

This example uses Hologres to show how to use the AI-assisted processing feature when performing a batch synchronization from one Hologres table to another. The goal is to translate the data in the feedback_info column of the source table into English and synchronize it to the destination table.

Prepare source table data

CREATE TABLE customer_feedback (
    id BIGINT PRIMARY KEY,
    device STRING,
    feedback_info STRING,
    pt INT
)
PARTITIONED BY (pt)
DISTRIBUTED BY HASH(id)
WITH (table_type='Duplication');
INSERT INTO customer_feedback (id, device, feedback_info, pt)
VALUES
(8, 'Huawei MateBook D14', 'Affordable, suitable for students, performance is adequate', 2020),
(1, 'iphone', 'This product is okay, I have used it for 1 year', 2013),
(10, 'Bose QuietComfort 35 II', 'A classic among noise-canceling headphones, maximum comfort', 2021);

Step 1: Create a batch synchronization task

  1. Go to the Workspaces page in the DataWorks console. In the top navigation bar, select a desired region. Find the desired workspace and choose Shortcuts > Data Studio in the Actions column.

  2. In the left-side navigation pane, click image to go to the data development page. To the right of Project Directory, click image and choose Create Node > Data Integration > Batch Synchronization. The Create Node dialog box appears.

  3. Set the node Path, Data source destination, and Name. Then, click OK to create the batch synchronization node.

    This topic uses a Hologres-to-Hologres synchronization task as an example to demonstrate the AI-assisted processing feature in batch synchronization tasks.

Step 2: Configure the synchronization task

After you create the batch synchronization node, the task editor opens. Configure the following settings on this page:

1. Data source

Configure the source and destination for the data synchronization task.

  • Type: You select the source and destination data source types in the Create a batch synchronization task step. These types cannot be changed. To use different types, you must create a new batch synchronization task.

  • Data Source: Select the Hologres data source that is bound to the workspace, or select Add Data Source from the drop-down list.

2. Runtime resource

  1. Select the Resource Group for the synchronization task. If you use a serverless resource group, you can also specify the Resource Usage (CU) for the task.

  2. After selecting a Resource Group, Data Integration automatically tests the connectivity between the resource group and the source and destination data sources. You can also click Connectivity Check to perform the test manually.

3. Source

Configure the source table details for synchronization, such as the Schema, Tables, Partition, and Data Filtering conditions. You can click Data Preview to view the specific data to be synchronized.

In this example, select public for Schema and select customer_feedback for Table.

4. Data processing

  1. In the Data processing section, you can Enable the data processing feature. This feature requires additional compute resources and increases the resource overhead of the task.

  2. Click Add Node and select AI-assisted processing.

  3. Configure the AI-assisted processing settings.

    The key parameters are described below.

    Parameter

    Description

    Model Provider

    Supported providers are DataWorks Model Service, Model Studio, and PAI-Marketplace.

    Model Endpoint

    If you select PAI-Marketplace, enter the model invocation endpoint. To obtain the endpoint, see Test service invocation.

    Model Name

    The model that performs the intelligent data processing. Select a model as needed.

    API Key

    The API key to access the model. Obtain it from the model provider.

    Processing Description

    Use natural language to describe the operation on the source field. Reference the field name by using the #{column_name} format. For example, in this case, enter Please translate '#{feedback_info}' into English.

    Output Field

    Enter the name of the field where the result will be stored. If the field does not exist, a new field is created.
    Note

    As configured in this example, the feedback_info field from the source table is translated into English and stored in the feedback_processed field.

  4. Click Output Preview in the upper-right corner of the AI-assisted processing section to preview the final output data.

  5. (Optional) You can configure multiple data processing steps that run sequentially.

    Below the data processing list, click the + button to add a new data processing step.

5. Destination

  1. Configure the destination table information, such as Schema, Table Name, and Partition.

    • You can click Generate Target Table Schema to quickly create the destination table.

    • If a table to receive the data already exists at the destination, you can select it.

  2. Configure the Write Mode and Write Conflict Policy.

    Select public for Schema and customer_feedback_after_translate for Table. Set write mode to SQL(INSERT INTO), write conflict strategy to Ignore, and Clear Hologres table before synchronization to Do not clear destination table (false). Set maximum connections to 9.

  3. Configure whether to clear existing data in the Hologres table before synchronization.

  4. (Optional) Configure the Maximum Connections.

    The Maximum Connections setting is effective only when the write mode is SQL(INSERT INTO). When you start the task, ensure that the Hologres instance has enough idle connections. A single task can use up to nine connections.

6. Destination field mapping

After you configure the data source, data processing, and destination, this section displays the field mapping. By default, Data Integration maps fields by name and position. You can adjust the mappings as needed. The field mapping interface displays the source and destination fields side-by-side, with lines indicating the mapping relationships. To manually adjust a mapping (for example, to map feedback_processed to translate_feedback), click the corresponding fields to connect them.

Note

In this example, in addition to the same-name mappings for the existing source table fields (id, device, feedback_info, and pt), you must also manually map the feedback_processed field, which stores the translated result, to the translate_feedback field in the destination table.

Step 3: Debug the task

  1. In the right-side pane of the task editor, click Run Configuration. Configure the Resource Group and related Script Parameters for debugging this node.

  2. In the toolbar at the top of the node, click Save and then click Running. Wait for the task to finish, and check that the run was successful. You can go to the destination database to verify that the table data is as expected.

Step 4: Configure scheduling

If the batch synchronization node needs to run on a schedule, you must configure the Scheduling Policy in the Scheduling Settings pane on the right. For more information, see Configure node scheduling properties.

Step 5: Publish the node

Click the Publish icon in the node toolbar to start the publishing process. This process publishes the task to the production environment. The task runs on a schedule only after it is published.

Next steps: Task O&M

After you publish the node, you can click Backfill Data or Perform O&M in the publishing dialog box.

  • Backfill Data: This option supports data backfill for the current node only. For more complex data backfill operations, go to Operation Center. For more information, see Manage data backfill instances.

  • Perform O&M: After you publish a task, Operation Center automatically manages it. In Operation Center, you can view the task's run status or manually trigger it. For more information, see Operation Center.

Previous：VectorizationNext：Single-table real-time synchronization