DocsICP FilingConsole
Documentation
Home Page Dataphin Dataphin(Non-shared) User Guide Data Integration Offline integration Widget library development Input widget Configure the OushuDB source component

Configure the OushuDB source component

Updated at:
Copy as MD
Product
My Favorites

The OushuDB source component reads data from an OushuDB data source. To synchronize data from OushuDB to another data source, configure the OushuDB source component first, then configure the destination.

Prerequisites

  • An OushuDB data source has been created. For more information, see .

  • The account used to configure the OushuDB source component must have read permission on the data source. If you do not have the permission, apply for it. For more information, see Apply for, renew, and revoke data source permissions.

Procedure

  1. In the top navigation bar of the Dataphin homepage, choose Develop > Data Integration.

  2. In the top navigation bar of the integration page, select a Project. If you are in Dev-Prod mode, you also need to select an environment.

  3. In the left-side navigation pane, click Batch Integration. In the Batch Integration list, click the desired batch pipeline to open its configuration page.

  4. In the upper-right corner of the page, click Component Library to open the Component Library pane.

  5. In the left-side navigation pane of the Component Library, select Source. In the source component list on the right, find the OushuDB component and drag it to the canvas.

  6. On the OushuDB source component card, click the image icon to open the OushuDB Source Configuration dialog box.

  7. In the OushuDB Source Configuration dialog box, configure the following parameters.

    Parameter

    Description

    step name

    Dataphin automatically generates a step name. You can modify it as needed. The name must adhere to the following rules:

    • Can contain only Chinese characters, letters, underscores (_), and digits.

    • Cannot exceed 64 characters in length.

    data source

    Select an OushuDB data source from the drop-down list. The list displays all available OushuDB data sources, regardless of whether you have read permission. Click the image icon to copy the data source name.

    • For a data source where you lack read permission, click Apply to request it. For more information, see Apply for, renew, and revoke data source permissions.

    • If you do not have an OushuDB data source, click New Data Source to create one. For more information, see .

    Time Zone

    Time-based data is processed in the current time zone. This defaults to the time zone of the selected data source and cannot be changed.

    Note

    For tasks created before V5.1.2, you can select either Data Source Default Configuration or Pipeline Configuration Time Zone. The default option is Pipeline Configuration Time Zone.

    • Data Source Default Configuration: The default time zone of the selected data source.

    • Pipeline Configuration Time Zone: The time zone configured for the current integration task in Properties > Pipeline Configuration.

    schema (Optional)

    Select the schema containing the source table. If left blank, the system defaults to the schema specified in the data source configuration.

    source table mode

    Select a source table mode: single table or multiple tables.

    • single table: Synchronizes data from one source table to one destination table.

    • multiple tables: Synchronizes data from multiple source tables to a single destination table. Data from all source tables is combined using a UNION operation.

      For more information about UNION, see .

    Table Matching Mode

    Select General Rule or database regular expression.

    Note

    This parameter appears only when you set source table mode to multiple tables.

    Table

    Specify the source table or tables.

    • If source table mode is set to single table, you can search for a table by entering a keyword, or find an exact match by entering the full table name and clicking Precise Search. After you select a table, the system automatically checks its status. Click the image icon to copy the name of the selected table.

    • If source table mode is set to multiple tables, enter an expression to add tables based on the selected table matching mode.

      • If Table Matching Mode is set to General Rule, enter a table expression in the input box to filter for tables with the same structure. The system supports enumerations, regular expression-like patterns, and a mix of both. For example: table_[001-100];table_102;.

      • If Table Matching Mode is set to database regular expression, enter a regular expression supported by the current database. The system matches tables in the source database based on this expression. When the task runs, it dynamically identifies and synchronizes all tables that match the expression.

      After you enter the expression, click Precise Search to view a list of matched tables in the Confirm Match Details dialog box.

    split key (Optional)

    The column used to partition the source data for parallel reading. For best performance, use a primary key or an indexed column.

    Important

    If you select a column with a date/time data type, the system identifies the minimum and maximum values and performs a brute-force split based on the total time range and concurrency level. This method does not guarantee even data distribution.

    fetch size (Optional)

    The number of rows to fetch from the database in a single batch. Batched fetching reduces network round trips, improves I/O efficiency, and lowers latency.

    filter condition (Optional)

    A filter condition (similar to a SQL WHERE clause) applied to the source data, such as ds=${bizdate}. The filter condition is suitable for the following scenarios:

    • Filtering a fixed subset of data.

    • Filtering data by using parameters.

    output fields

    Lists all fields from the selected table(s) that match the filter conditions. You can perform the following operations:

    • Field Management: Remove fields that you do not need to output to downstream components.

      • To remove a single field: Click the sgaga icon in the Actions column to remove the field.

      • To remove multiple fields in bulk: Click Field Management. In the Field Management dialog box, select the fields you want to remove, click the image left arrow icon to move them from the selected list to the unselected list, and then click OK.

        image..png

    • Bulk Add: Click bulk add to configure fields in bulk by using the JSON, TEXT, or DDL format.

      Note

      After you click OK, the new configuration overwrites the existing field configuration.

      • Example of bulk configuration in JSON format:

        // Example:
  [{
     "index": 0,
     "name": "id",
     "type": "int(10)",
     "mapType": "Long",
     "comment": "comment1"
   },
   {
     "index": 1,
     "name": "user_name",
     "type": "varchar(255)",
     "mapType": "String",
     "comment": "comment2"
 }]
        Note

        index specifies the column number, name specifies the output field name, and type specifies the output field type.

        For example, "index":3,"name":"user_id","type":"String" means that the fourth column from the source is output with the field name user_id and the data type String.

      • Example of bulk configuration in TEXT format:

        // Example:
0,id,int(10),Long,comment1
1,user_name,varchar(255),Long,comment2

        • The row delimiter separates the information for each field. The default is a newline character (\n). You can also use a semicolon (;) or a period (.).

        • A column delimiter is used to separate field names from field types, and the default is a half-width comma (,), and the',' character is supported. The field type is optional, and the default is','.

      • Example of bulk configuration in DDL format:

        CREATE TABLE tablename (
	user_id serial,
	username VARCHAR(50),
	password VARCHAR(50),
	email VARCHAR (255),
	created_on TIMESTAMP,
);

    • Create Output Field: Click + Create Output Field, and then enter the Field name, Type, and Comment, and select the Mapped Type. After you finish configuring the row, click the image icon to save it.

  8. Click OK to save the configuration for the OushuDB source component.

Previous：Configure the TDSQL for PostgreSQL input componentNext：Configure the Milvus input component