Configure PolarDB-X (formerly DRDS) Output Widget

Prerequisites

• A PolarDB-X data source has been created. For more information, see Create PolarDB-X Data Source.

• To configure the PolarDB-X output widget properties, your account must have write-through permissions for the data source. If you lack these permissions, request them first. For more information, see or Request, Renew, and Return Data Source Permissions.

Procedure

1. On the Dataphin home page, navigate to Development > Data Integration via the top menu bar.

2. On the integration page, select Project from the top menu bar (choose Environment in Dev-Prod mode).

3. In the left-side navigation pane, click Batch Pipeline. From the Batch Pipeline list, select the offline pipeline you want to develop.

4. Click Component Library in the upper-right corner to open the Component Library panel.

5. In the Component Library panel, select Output in the left-side navigation pane. Locate the Polardb-x (formerly DRDS) widget in the list on the right and drag it onto the canvas.

6. Connect the target input, transform, or flow widget to the PolarDB-X (formerly DRDS) output widget by clicking and dragging the icon.

7. Click the icon on the PolarDB-X (formerly DRDS) output widget card to open the Polardb-x (formerly DRDS) Output Configuration dialog box.

   

8. In the Polardb-x (formerly DRDS) Output Configuration dialog box, configure the following parameters.

   Parameter		Description
   Basic Settings	Step Name	The name of the PolarDB-X output widget. Dataphin automatically generates a step name, which you can modify as needed. The name must meet the following requirements: Can only contain Chinese characters, letters, underscores (_), and numbers. Cannot exceed 64 characters.
   	Datasource	The drop-down list displays all PolarDB-X data sources, including those with and without write-through permission. Click the icon to copy the data source name. For data sources without write-through permission, click Request next to the data source to request permission. For more information, see Request Data Source Permissions. If no PolarDB-X data source exists, click Create Data Source to create one. For more information, see Create PolarDB-X Data Source.
   	Table	Select the target table for output data. You can search by keyword or enter the exact table name and click Precise Search. After you select a table, the system automatically checks the table status. Click the icon to copy the selected table name.
   	Loading Policy	Select the policy for writing data to the target table. Overwrite Data: Replaces historical data in the target table with data from the current source table. Append Data: Appends data to the target table without modifying historical data.
   	Batch Write Data Volume (optional)	The amount of data written per batch. You can also set Batch Write Count. The system writes data when either limit is reached first. The default is 32M.
   	Batch Write Count (optional)	The default is 2048 entries. During data synchronization, a batch write strategy is used. The relevant parameters are Batch Write Count and Batch Write Data Volume. When the accumulated data reaches either limit (batch write data volume or count), the system treats the batch as full and writes it to the target. We recommend setting the batch write data volume to 32 MB. Adjust the batch insert count based on the actual single-record size. A larger value is generally preferred to take advantage of batch writing. For example, if a single record is about 1 KB, you can set the batch insert byte size to 16 MB and the batch insert count to a value greater than 16 MB / 1 KB = 16,384 entries, such as 20,000 entries. With this configuration, the system triggers a batch write each time the accumulated data reaches 16 MB.
   	Preparation Statement (optional)	A SQL script to run on the database before data import. For example, to maintain service availability: before the current step writes data, create target table Target_A and write data to it. After the write completes, rename the in-service table Service_B to Temp_C, rename Target_A to Service_B, and then delete Temp_C.
   	End Statement (optional)	A SQL script to run on the database after data import.
   Field Mapping	Input Field	Input fields are populated based on the output of the upstream widget.
   	Output Field	The output fields are displayed. You can perform the following operations: Field Management: Click Field Management to select output fields. Click the icon to move the Selected Input Fields to Unselected Input Fields. Click the icon to move the Unselected Input Fields to Selected Input Fields. Batch Add: Click Batch Add to support batch configuration in JSON, TEXT format, or DDL format. JSON format example: // Example: [{ "name": "user_id", "type": "String" }, { "name": "user_name", "type": "String" }] Note name specifies the field name, and type specifies the data type after import. For example, "name":"user_id","type":"String" imports the field user_id with the String data type. TEXT format example: // Example: user_id,String user_name,String The row delimiter separates field entries. The default is a line feed (\n). Semicolons (;) and periods (.) are also supported. The column delimiter separates the field name and type. The default is a comma (,). DDL format example: CREATE TABLE tablename ( id INT PRIMARY KEY, name VARCHAR(50), age INT ); Create New Output Field: Click +create New Output Field, enter the Column name and select a Type. Click the icon to save the row.
   	Quick Mapping	Map fields between upstream input and target table fields. Quick Mapping includes Row Mapping and Name Mapping. Name Mapping: Maps fields that share the same name. Row Mapping: Maps fields by row position when source and target field names differ.

9. Click Confirm to finalize the PolarDB-X output widget configuration.