DocsICP FilingConsole
Documentation
Home Page Dataphin Intelligent Data construction and governance Dataphin-sharing mode (fully managed version) User Guide Data ingestion Data integration Offline Integration Widget library development Input component Configure the TDSQL for PostgreSQL input component

Configure the TDSQL for PostgreSQL input component

Updated at:
Copy as MD
Product
My Favorites

To synchronize data from a TDSQL for PostgreSQL data source to a target, first configure the input component to read from the source, and then configure the target data source. This topic describes how to configure the TDSQL for PostgreSQL input component.

Prerequisites

Procedure

  1. In the top navigation bar of the Dataphin homepage, choose R&D > Data Integration.

  2. In the top navigation bar of the data integration page, select a Project. If you are using the Dev-Prod mode, you also need to select an environment.

  3. In the left navigation bar, click Offline integration, and in the Offline integration list, click the offline pipeline that you want to develop to open its configuration page.

  4. Click Component Library in the upper-right corner of the page to open the Component Library panel.

  5. In the Component Library panel, click Input in the left-side navigation pane. Find the TDSQL for PostgreSQL component in the list of input components and drag it to the canvas.

  6. On the TDSQL for PostgreSQL input component, click the image icon to open the TDSQL for PostgreSQL Input Configuration dialog box.

  7. In the TDSQL for PostgreSQL Input Configuration dialog box, configure the parameters.

    Parameter

    Description

    Step name

    Dataphin generates a default step name that you can modify based on your business requirements. The name must meet the following rules:

    • Can contain only Chinese characters, letters, underscores (_), and digits.

    • Cannot exceed 64 characters.

    Data source

    The drop-down list displays all TDSQL for PostgreSQL data sources in Dataphin, including those for which you have the read permission for synchronization and those for which you do not. You can click the image icon to copy the current data source name.

    Schema

    Cross-schema table selection is supported. Select the schema that contains the source table. If you do not specify a schema, the one configured in the data source is used by default.

    Number of source tables

    Select the number of source tables. Options include single table and multiple tables:

    • single table: Use this option to synchronize business data from one table to one target table.

    • multiple tables: Use this option to synchronize business data from multiple tables to the same target table. When data from multiple tables is written to the same target table, the system uses a UNION operation to combine the data.

      For more information about the UNION operator, see INTERSECT, UNION, and EXCEPT.

    Table matching mode

    You can select General rule or Database regex.

    Note

    This parameter is available only when multiple tables is selected for Number of source tables.

    Table

    Select the source tables:

    • If you selected single table for Number of source tables, you can enter a keyword to search for the table, or enter the exact table name and click Exact Match. After you select a table, the system automatically checks its status. You can click the image icon to copy the name of the selected table.

    • If you selected multiple tables for Number of source tables, enter an expression to add tables based on the selected matching mode.

      • If you selected General rule as the Table matching mode, enter an expression in the input box to filter for tables with the same structure. The system supports enumerations, regular-like expressions, and a mix of both. For example: table_[001-100];table_102;.

      • If you selected Database regex as the Table matching mode, enter a regular expression supported by the current database. The system matches tables in the source database based on this regular expression. When the task runs, it dynamically matches and synchronizes new tables that fit the expression.

      After entering the expression, you can click Exact Match to view a list of matched tables in the Confirm Matching Details dialog box.

    Split key (Optional)

    The system shards data based on the configured split key. You can use this parameter with concurrency settings to enable parallel reads. You can specify a column from the source table as the split key. We recommend using a primary key or an indexed column as the split key to ensure optimal performance.

    Important

    If you select a date or time data type, the system performs brute-force splitting based on the detected minimum and maximum values, the total time range, and the concurrency level. This process does not guarantee an even data distribution.

    Batch size (Optional)

    Specifies the number of records to read per batch. Reading data in batches instead of one record at a time reduces database interactions, improving I/O efficiency and lowering network latency.

    Input filter (Optional)

    Specify a filter condition for the source data by using an expression similar to a WHERE clause, such as ds=${bizdate}. The Input filter is suitable for the following scenarios:

    • Filtering a fixed subset of data.

    • Filtering with parameters.

    Output fields

    This section displays all fields from the selected tables that match the filter criteria. You can perform the following operations:

    • Field management: If you do not need to output certain fields to downstream components, you can delete them.

      • Delete a single field: To delete a small number of fields, click the sgaga icon in the Actions column for the fields you want to delete.

      • Delete fields in bulk: To delete a large number of fields, click Field management. In the Field management dialog box, select multiple fields, click the image left arrow icon to move them to the unselected input fields list, and then click OK to complete the bulk deletion.

        image..png

    • Bulk add: Click bulk add to configure fields in bulk by using the JSON, TEXT, or DDL format.

      Note

      After you add fields in bulk, clicking OK overwrites the existing field configuration.

      • Example of bulk configuration in JSON format:

        // Example:
  [{
     "index": 0,
     "name": "id",
     "type": "int(10)",
     "mapType": "Long",
     "comment": "comment1"
   },
   {
     "index": 1,
     "name": "user_name",
     "type": "varchar(255)",
     "mapType": "String",
     "comment": "comment2"
 }]
        Note

        The index parameter specifies the source column's index, name specifies the destination field name, and type specifies the destination field type.

        For example, "index":3,"name":"user_id","type":"String" imports the fourth column from the source, names the field user_id, and sets its type to String.

      • Example of bulk configuration in TEXT format:

        // Example:
0,id,int(10),Long,comment1
1,user_name,varchar(255),Long,comment2

        • The row delimiter separates the information for each field. The default is a newline character (\n). Semicolons (;) and periods (.) are also supported.

        • The column delimiter is used to separate field names from field types, and the default is a half-width comma (,). The ',' delimiter is supported. The field type is optional, and the default is ','.

      • Example of bulk configuration in DDL format:

        CREATE TABLE tablename (
	user_id serial,
	username VARCHAR(50),
	password VARCHAR(50),
	email VARCHAR (255),
	created_on TIMESTAMP,
);

    • Add output field: Click + New output field. In the new row, specify the Field, Type, and Comment, and select a Mapping Type. After you finish configuring the row, click the image icon to save.

  8. Click Confirm to save the component's configuration.

Previous：Configure the dataset input componentNext：Configure the Milvus input component