Configure the Amazon RDS for MySQL input component

Prerequisites

• An Amazon RDS for MySQL data source must exist. For more information, see Create an Amazon RDS for MySQL data source.

• The account that you use to configure the component must have the sync read permission on the data source. If you do not have this permission, apply for it. For more information, see Apply for data source permissions.

Procedure

1. In the top navigation bar of the Dataphin homepage, choose Develop > Data Integration.

2. On the integration page, select a Project from the top navigation bar. If you are in Dev-Prod mode, you must also select an environment.

3. In the left-side navigation pane, click Batch Integration. In the Batch Integration list, click the batch pipeline you want to configure to open its configuration page.

4. In the upper-right corner of the page, click Component Library to open the component library panel.

5. In the left-side navigation pane of the component library panel, select Input. In the list of input components on the right, find the Amazon RDS for MySQL component and drag it to the canvas.

6. On the Amazon RDS for MySQL input component card, click the icon to open the Amazon RDS for MySQL Input Configuration dialog box.

7. In the Amazon RDS for MySQL Input Configuration dialog box, configure the parameters.

   Parameter	Description
   Step Name	The name for the component. Dataphin generates a default name, which you can change to suit your business needs. The name must meet the following rules: Can contain only Chinese characters, letters, underscores (_), and digits. Cannot exceed 64 characters in length.
   Number of source tables	Select whether to read data from a single table or multiple tables. Valid values are single table and multiple tables. single table: Synchronizes data from one source table to one destination table. multiple tables: Synchronizes data from multiple source tables to a single destination table. The component combines data from the source tables by using a UNION operation. For more information about UNION, see INTERSECT, UNION, and EXCEPT.
   Data source	In the data source drop-down list, select an Amazon RDS for MySQL data source. The list displays all Amazon RDS for MySQL data sources in the current Dataphin project, including those with and without the required sync read permission. If you set Number of source tables to single table, you can select only one data source. Click the icon to copy the name of the current data source. If you set Number of source tables to multiple tables, you can select multiple data sources. If you do not have the sync read permission for a data source, you can click Apply next to the data source to request the permission. For more information, see Apply for data source permissions. If you do not have an Amazon RDS for MySQL data source, you can click New Data Source in the drop-down list to create one. For more information, see Create an Amazon RDS for MySQL data source.
   Database (Optional)	Select the database that contains the source table. If you leave this parameter empty, Dataphin uses the database specified during data source registration. If you set Number of source tables to multiple tables, you can select multiple databases. Click the icon to view all selected databases in the Database List dialog box.
   Table matching mode	Select either general rule or database regular expression. Note This parameter appears only when Number of source tables is set to multiple tables.
   Table	Select the source table or tables. If Number of source tables is set to single table, you can search for a table by entering a keyword, or enter the exact table name and click Exact Search. After you select a table, the system automatically checks its status. Click the icon to copy the name of the selected table. If Number of source tables is set to multiple tables, enter an expression to add tables based on the selected Table matching mode. If you select general rule: Enter an expression in the input box to filter for tables that have the same structure. The system supports enumeration, regular expression-like patterns, and a mix of both. For example, table_[001-100];table_102;table_abc*. The asterisk (*) wildcard is effective only at the beginning or end of a pattern. If you select database regular expression: Enter a regular expression supported by the source database. The system uses this expression to match tables. When the task runs, it dynamically matches new tables based on the regular expression and then synchronizes them. After you enter the expression, click Exact Search to view the list of matched tables in the Confirm Match Details dialog box.
   Split key (Optional)	The system partitions data based on the specified split key column. This feature, when used with the concurrency setting, enables parallel data reads. For optimal data transfer performance, we recommend using the primary key or a column with an index as the split key. Important If you select a column of a date or time data type, the system splits data based on the time range and concurrency level. This method does not guarantee an even distribution of data among partitions.
   Data filtering (Optional)	Specifies a filter condition to apply when extracting data. Fixed value: Use a static value to filter data. For example, ds=20211111. Variable: Use a variable to filter data. For example, ds=${bizdate}.
   Output fields	This section displays all fields from the selected tables that match the filter conditions. You can perform the following operations: Manage fields: If you do not need to output certain fields to downstream components, you can remove them. Delete a single field: To delete a single field, click the icon in the Actions column for each field you want to remove. Delete fields in bulk: To delete a large number of fields, click Field Management. In the Field Management dialog box, select the fields that you want to remove, click the left arrow icon to move them to the unselected list, and then click OK. Batch Add: Click Batch Add to configure fields in bulk by using JSON, TEXT, or DDL format. Note After you add fields in bulk and click OK, the existing field configuration is overwritten. To configure fields in bulk by using JSON format, see the following example: // Example: [{ "index": 1, "name": "id", "type": "int(10)", "mapType": "Long", "comment": "comment1" }, { "index": 2, "name": "user_name", "type": "varchar(255)", "mapType": "String", "comment": "comment2" }] Note index specifies the 1-based column number of the source data. name specifies the field name after import. type specifies the data type after import. For example, "index":3,"name":"user_id","type":"String" imports the data from the third column, names the field user_id, and sets its data type to String. To configure fields in bulk by using TEXT format, see the following example: // Example: 1,id,int(10),Long,comment1 2,user_name,varchar(255),Long,comment2 The row delimiter separates the information for each field. The default is a newline character (\n). You can also use semicolons (;) or periods (.). The column separator separates field names from field types. The default is a comma (,).The supported separator is','. The field type is optional, and the default is','. To configure fields in bulk by using DDL format, see the following example: CREATE TABLE tablename ( user_id serial, username VARCHAR(50), password VARCHAR(50), email VARCHAR (255), created_on TIMESTAMP, ); Create an output field: Click + New Output Field. Enter the Field, Type, and Comment, and select a Mapping Type. After you configure the row, click the icon to save it.

8. Click OK to save the component's configuration.