Create a high-speed synchronization pipeline task

Prerequisites

You have configured the required data sources. Before you create a high-speed synchronization pipeline task, configure the source and destination data sources. This allows you to select them when configuring the input and output components. High-speed synchronization pipeline tasks support the following data sources:

• Create a MySQL data source

• Create an Oracle data source

• Create a PostgreSQL data source

• Create a Microsoft SQL Server data source

• Create a MaxCompute data source

• Create a Hive data source

• Create an HDFS data source

• Create a Doris data source

• Create a SelectDB data source

Procedure

Step 1: Create a high-speed synchronization pipeline

1. In the top navigation bar of the Dataphin homepage, choose R&D > Data Integration.

2. In the top navigation bar of the data integration page, select a Project. If you are using Dev-Prod mode, you also need to select an Environment.

3. In the offline integration list, click the New icon and select High-speed Synchronization. The Create High-speed Synchronization Pipeline dialog box opens.

4. In the Create High-speed Synchronization Pipeline dialog box, configure the pipeline parameters.

   Parameter	Description
   Pipeline name	Enter a name for the pipeline. The name can be up to 64 characters long and cannot contain the following special characters: vertical bar (|), forward slash (/), backslash (\), colon (:), question mark (?), angle brackets (<>), asterisk (*), or single quotation mark ('').
   Scheduling type	Select the scheduling type for the pipeline. Options include: Periodic node: A task that runs periodically. Manual node: A task that has no dependencies and requires manual triggering.
   Description	Enter a brief description of the high-speed synchronization pipeline, up to 1,000 characters.
   Select directory	Select the directory to store the task. If a directory does not exist, you can create a new folder by following these steps: Above the compute task list on the left, click the new folder icon to open the New Folder dialog box. In the New Folder dialog box, enter a folder Name and Select a directory location as needed. Click OK.

5. Click OK to create the high-speed synchronization pipeline.

Step 2: Develop the high-speed synchronization pipeline script

The component library for high-speed synchronization pipeline tasks supports only Input and Output components. The configuration of these components is the same as that for the input and output components of offline pipeline tasks, and you can also switch to script mode.

The input and output components for high-speed synchronization pipeline tasks are categorized into two types based on whether the data source supports Apache Arrow:

• Apache Arrow acceleration: The data source provides a native columnar read/write path, allowing data transfer in Arrow format without row-to-column conversion.

• Apache Arrow adaptation: The data source does not provide a native columnar path. During read and write operations, data must be converted between row format and Arrow columnar format.

The following input components are supported:

The following output components are supported:

Step 3: Configure scheduling properties

Click Scheduling Configuration on the canvas toolbar to configure the basic information, scheduling configuration, scheduling dependencies, scheduling parameters, runtime configuration, and resource configuration for the high-speed synchronization pipeline task. For more information, see the following topics:

• Configure basic information for offline integration pipelines

• Configure scheduling properties for offline pipelines

• Configure scheduling dependencies for offline pipelines

• Runtime configuration for offline pipeline tasks

• Configure resources for offline integration pipeline tasks

Step 4: Save and commit the high-speed synchronization pipeline

1. Click the Save icon above the canvas to save the high-speed synchronization pipeline task.

2. Click the Commit icon above the canvas. In the Commit Message dialog box, enter a commit message and click OK and Commit.

Next steps

• If your development mode is Dev-Prod mode, you need to publish a task before it can be scheduled in the production environment.

• If your development environment is in Basic mode, you can schedule the task in the production environment after you commit it. You can go to the O&M Center to view the committed task.