Configure TDSQL for MySQL input

Prerequisites

• You have created a TDSQL for MySQL data source. For more information, see .

• The account used for configuration must have sync read permissions for the data source. If not, you must apply for them. For more information, see Apply for data source permissions.

Steps

1. In the top navigation bar of the Dataphin homepage, choose Development > Data Integration.

2. In the top navigation bar of the Data Integration page, select a Project. In Dev-Prod mode, you must also select an environment.

3. In the left-side navigation pane, click offline integration. From the offline integration list, click the offline pipeline you want to edit.

4. In the upper-right corner of the page, click Component Library to open the Component Library panel.

5. In the left-side navigation pane of the Component Library panel, choose Input. Find the TDSQL for MySQL component in the list and drag it onto the canvas.

6. On the TDSQL for MySQL input component card, click the icon to open the TDSQL for MySQL input configuration dialog box.

7. In the TDSQL for MySQL input configuration dialog box, configure the parameters.

   Parameter	Description
   Step name	The name of the component. Dataphin automatically generates a step name that you can modify. The name must follow these rules: Can contain only Chinese characters, letters, underscores (_), and digits. Can be up to 64 characters in length.
   Source tables	The number of source tables. Valid values: single table and multiple tables. single table: This option applies to scenarios where you want to synchronize data from one table to a single target table. multiple tables: Use this option to synchronize data from multiple tables to a single target table. Data from multiple tables is merged into one table by using a UNION operation. For more information about the UNION operator, see INTERSECT, UNION, EXCEPT, and MINUS.
   Data source	From the drop-down list, select a data source. The list displays all TDSQL for MySQL data sources in Dataphin, regardless of whether you have sync read permissions. If you set Source tables to single table, you can select only one data source. Click the icon to copy the data source name. If you set Source tables to multiple tables, you can select multiple data sources that use the same time zone. If you lack sync read permissions for a data source, click Apply next to it to request them. For more information, see Apply for data source permissions. If you do not have a TDSQL for MySQL data source, click New data source in the drop-down list to create one. For more information, see .
   Database (Optional)	Select the database that contains the table. If left empty, the system uses the database specified during data source registration. If you set Source tables to multiple tables, you can select multiple databases. Click the icon to view all selected databases in the Database list dialog box.
   Table matching mode	You can select general rule or database regular expression. Note This parameter is available only when Source tables is set to multiple tables.
   Table	Select the source table or tables. If Source tables is set to single table, you can search for a table by keyword, or enter the exact table name and click Exact Match. After you select a table, the system automatically checks its status. Click the icon to copy the name of the selected table. If Source tables is set to multiple tables, enter an expression to add tables based on the selected table matching mode. If you set the table matching mode to general rule, you can enter a table expression in the input box to filter for tables with the same structure. The system supports enumeration formats, regular expression-like formats, and a combination of both. For example, table_[001-100];table_102;table_abc*. The * wildcard character is valid only at the beginning or end of an expression. If Table matching mode is set to database regular expression: In the input box, enter a regular expression that is supported by the database. When the job runs, it dynamically matches and synchronizes the latest set of tables based on this regular expression. After you enter the expression, click Exact Match to view a list of matched tables in the Confirm matching details dialog box.
   Sharding key (Optional)	The system shards data based on the configured sharding key. This parameter, combined with the degree of parallelism setting, enables concurrent data reading. You can select a column from the source table as the sharding key. For optimal data transfer performance, we recommend using a primary key or an indexed column as the sharding key. Important If you select a column of a date or time data type, the system performs a brute-force split based on the detected minimum and maximum values, the total time range, and the degree of parallelism. This method does not guarantee an even data distribution.
   Filter (Optional)	Specify a filter condition to extract a data subset. Options include: Fixed value: Extracts data that matches a specific value. Example: ds=20211111. Variable parameter: Extracts data based on a variable. Example: ds=${bizdate}.
   Output fields	This section lists all fields from the selected tables that match the filter conditions. You can perform these operations: Field management: If you do not need to output certain fields to downstream components, you can remove them. To remove a single field: In the field's row, click the icon in the Actions column. To remove fields in bulk: Click Field management. In the Field management dialog box, select the fields you want to remove, click the left arrow icon to move them to the unselected input fields list, and then click OK. Add in batches: Click Add in batches to add fields in JSON, TEXT, or DDL format. Note After you add fields in batches and click OK, this action overwrites the existing field configuration. Add fields in JSON format. Example: // Example: [{ "index": 1, "name": "id", "type": "int(10)", "mapType": "Long", "comment": "comment1" }, { "index": 2, "name": "user_name", "type": "varchar(255)", "mapType": "String", "comment": "comment2" }] Note index specifies the column number of the object, name specifies the field name after import, and type specifies the field type after import. For example, "index":3,"name":"user_id","type":"String" imports the fourth column from the file, names the field user_id, and sets its data type to String. Add fields in TEXT format. Example: // Example: 1,id,int(10),Long,comment1 2,user_name,varchar(255),Long,comment2 The row delimiter separates the information for each field. The default is a newline character (\n). You can also use a semicolon (;) or a period (.). The column delimiter, which defaults to a comma (,), separates the field name and type. The field type is optional and defaults to String if omitted. Add fields in DDL format. Example: CREATE TABLE tablename ( user_id serial, username VARCHAR(50), password VARCHAR(50), email VARCHAR (255), created_on TIMESTAMP, ); To create an output field: Click +New output field, then enter the Field, Type, Description, and select a Mapping type. After you configure the row, click the icon to save the settings.

8. Click Confirm to save the component's configuration.