Configure the TDSQL for MySQL Input Component

Prerequisites

• You have created a TDSQL for MySQL data source. For more information, see .

• The account used to configure the TDSQL for MySQL input component must have read-through permission on the data source. If the account does not have this permission, request it. For more information, see Request Data Source Permissions.

Procedure

1. On the Dataphin homepage, in the top menu bar, click Develop, and then click Data Integration.

2. On the Integration page, in the top menu bar, select a Project. In Dev-Prod mode, also select an environment.

3. In the left navigation pane, click Batch Pipeline. In the Batch Pipeline list, click the offline pipeline that you want to develop. The configuration page for the offline pipeline opens.

4. In the upper-right corner of the page, click Component Library to open the Component Library panel.

5. In the left navigation pane of the Component Library panel, click Input. In the list of input components on the right, locate the TDSQL for MySQL component and drag it onto the canvas.

6. Click the icon in the TDSQL for MySQL input component card to open the TDSQL for MySQL Input Configuration dialog box.

7. In the TDSQL for MySQL Input Configuration dialog box, configure the parameters.

   Parameter	Description
   Step Name	The name of the TDSQL for MySQL input component. Dataphin generates a step name automatically. You can change it based on your business scenario. Naming rules: Use only Chinese characters, letters, underscores (_), and digits. Do not exceed 64 characters.
   Source Table Count	Select the number of source tables. Options are Single Table and Multiple Tables: Single Table: Use this option when syncing business data from one source table to one destination table. Multiple Tables: Use this option when syncing business data from multiple source tables to one destination table. When writing data from multiple tables into one destination table, the system uses the UNION algorithm. For more information about UNION, see INTERSECT, UNION, and EXCEPT.
   Datasource	The data source drop-down list displays all TDSQL for MySQL data sources in the current Dataphin instance, including those for which you have read-through permission and those for which you do not. If the source table quantity is set to single table, you can select only one data source. You can click the icon to copy the name of the current data source. If the source table quantity is set to multiple tables, you can select multiple data sources that share the same time zone. Multiple Tables For data sources where you lack read-through permission, click Request next to the data source name to request read-through permission. For more information, see Request Data Source Permissions. If you do not have a TDSQL for MySQL data source, click Create Data Source in the drop-down list. For more information, see .
   Database (Optional)	Select the database that contains the source table. If you leave this field blank, the system uses the database specified when registering the data source. If you select Multiple Tables, you can select multiple databases. Click the icon to open the Database List dialog box and view all selected databases.
   Table Matching Method	Select Generic Rule or Database Regex. Note This parameter is available only when you select Multiple Tables for Source Table Count.
   Table	Select the source table: If you select Single Table for Source Table Count, enter a keyword to search for the table name, or enter the exact table name and click Exact Match. After you select a table, the system automatically checks the table status. Click the icon to copy the name of the selected table. If you select Multiple Tables for Source Table Count, enter an expression based on the table matching method: Generic Rule: Enter a table expression in the input box to filter tables with the same structure. The system supports enumeration, regex-like syntax, and mixed formats. Example: table_[001-100];table_102;. Database Regex: Enter a regex supported by the current database. The system matches tables in the target database using this regex. At runtime, the system dynamically matches new tables based on the regex. After entering the expression, click Exact Match to open the Confirm Match Details dialog box and view the list of matched tables.
   Shard Key (Optional)	The system partitions data based on the configured shard key column. Use this with concurrency settings to enable concurrent reads. You can use any column from the source table as the shard key. For best performance, use a primary key or indexed column. Important If you select a date-time type, the system performs brute-force partitioning based on the full time range and concurrency setting. This method does not guarantee even distribution.
   Input Filter (Optional)	Set conditions to filter the data to extract. Configuration details: Static Value: Extract matching data. Example: ds=20211111. Variable Parameter: Extract a subset of data. Example: ds=${bizdate}.
   Output Fields	The Output Fields section lists all fields from the selected table and filtered results. You can perform the following actions: Manage Fields: Remove fields that you do not need to pass to downstream components: Remove One Field: To remove a few fields, click the icon in the Actions column. Remove Multiple Fields: To remove many fields, click Field Management. In the Field Management dialog box, select multiple fields. Click the left arrow icon to move them from Selected to Not Selected. Click OK to complete the batch removal. Batch Add: Click Batch Add to add fields in JSON, TEXT, or DDL format. Note After you click OK, the batch-added fields overwrite existing field configurations. JSON format example: // Example: [{ "index": 1, "name": "id", "type": "int(10)", "mapType": "Long", "comment": "comment1" }, { "index": 2, "name": "user_name", "type": "varchar(255)", "mapType": "String", "comment": "comment2" }] Note The index specifies the column number of the specified object, name specifies the field name after import, and type specifies the field type after import. For example, "index":3,"name":"user_id","type":"String" specifies that you import the fourth column in the file, with the field name user_id and the field type String. TEXT format example: // Example: 1,id,int(10),Long,comment1 2,user_name,varchar(255),Long,comment2 The row delimiter separates field entries. Default: line feed (\n). Supported delimiters: \n, semicolon (;), and period (.). The column delimiter separates field names and types. Default: comma (,). Supported delimiters: ','. The field type is optional and defaults to ','. DDL format example: CREATE TABLE tablename ( user_id serial, username VARCHAR(50), password VARCHAR(50), email VARCHAR (255), created_on TIMESTAMP, ); Add Output Field: Click + Add Output Field. Enter values for Column, Type, and Comment. Select a Mapping Type. Click the icon to save the row.

8. Click OK to finish configuring the TDSQL for MySQL input component.