Configure a batch data integration node

Scenarios

Common scenarios for a batch data integration node include data migration and data transmission.

Features

• Supports any database type for source and destination tables.

• Supports field mapping, which lets you modify the names and data types of fields in the source and destination tables.

• Supports concurrent data migration.

• Supports using WHERE clauses to filter data before migration.

• Supports pre- and post-SQL statements to process data before and after data import.

Limits

• Processing a large table that does not have a primary key can cause a memory overflow or an out-of-memory (OOM) error.

• If a recurring schedule node runs successfully but then fails 10 or more consecutive times, the batch integration job fails immediately and no more Spark tasks are submitted. In this case, you must manually run the task node until it succeeds.

Procedure

1. Log on to the DMS console V5.0.

2. In the top navigation bar, choose Data+AI > Data+AI > Task Orchestration.

   Note

   If you use the DMS console in simple mode, move the pointer over the icon in the upper-left corner and choose All Features > Data+AI > Data Development > Task Orchestration.

3. Click the name of the destination task flow to go to its details page.

   Note

   For more information about how to add a new task flow, see Add a task flow.

4. Optional: Click the icon to expand the basic information panel for the task flow.

   Note

   Click the icon to collapse the panel.

5. In the Task Type list on the left of the canvas, drag the Batch Data Integration node to the blank area.

6. Double-click the offline data integration node.

7. Optional: On the configuration page for the batch data integration node, click Variable Settings on the right side of the page to configure variables for the node. You can click the icon in the upper-right corner of the Variable Settings area to view tips for configuring variables.

   • Click the Node Variable tab to configure node variables. For more information, see Configure time variables.

   • Click the Task Flow Constants tab to configure task flow constants. A task flow constant is a static value that can be used in all nodes. The format in SQL is ${name}.

   • Click the Task Flow Variable tab to configure task flow variables. For more information, see Configure time variables.

   • Click the Input Variables tab to view upstream variables, runtime status variables, and system variables.

8. On the configuration page of the Batch Data Integration node, configure the node.

   Category	Configuration item	Description
   Data Source Configuration	Source database	Search for and select the source database.
   	Source table	Select the source table.
   	Destination database	Search for and select the destination database.
   	Destination table	Select the destination table.
   Source table configuration	Data filtering	Enter a WHERE filter statement. Variables are supported.
   	Recommended sharding field	The default value is Auto.
   	Maximum concurrent tasks	Select the maximum number of concurrent tasks. The default value is 4. Note DMS limits the number of concurrent Spark tasks for a single tenant. If the number of tasks exceeds 4, the tasks are queued for execution.
   Destination table configuration	Pre-import behavior	Enter the SQL statements to execute before data import. Choose whether to clear the destination table.
   	Post-import behavior	Enter the SQL statements to execute after data import.
   Field mapping	-	Select the mappings for all fields in the destination table as needed. Map by name: The system automatically maps fields that have the same name. Note Manually add mappings for unmapped fields in the destination table. Map by row: Maps fields that are in the same row. Manual mapping: Optional: Click Cancel Mapping to remove a field mapping. Hover over a source field. Click the hollow circle to the right of the field Type and drag a line to connect it to a destination field. Optional: After mapping all destination table fields, click Auto-layout to reorder the destination table fields.

9. Click Try Run in the upper-left corner of the page. In the Prompt dialog box, click Confirm to test run the batch data integration node.

   The runtime is proportional to the data size. A larger data size results in a longer runtime.

   • If the last line of the execution log shows status SUCCEEDED, the test run is successful.

     After the test run is successful, you can query the migrated data in the SQL window of the destination database.

   • If the last line of the execution log shows status FAILED, the test run failed.

     If the test run fails, check the execution log for the cause of the failure, modify the configuration, and then try again.

   Note

   In standard mode for data warehouse development, a test run of a batch data integration node performs a dry-run to validate the configuration without affecting data.