Configure TDengine input widget

更新时间:
复制 MD 格式

The TDengine input widget reads data from a TDengine data source. To synchronize TDengine data to other data sources, configure the TDengine input widget to specify the source, and then set up the target data source.

Prerequisites

Before you begin, make sure the following tasks are completed:

Procedure

  1. On the Dataphin home page, select Development > Data Integration from the top menu bar.

  2. On the integration page, select Project from the top menu bar (Dev-Prod mode requires selecting an environment).

  3. In the left-side navigation pane, click Batch Pipeline. In the Batch Pipeline list, click the offline pipeline you want to develop to open its configuration page.

  4. Click Component Library in the upper right corner of the page to open the Component Library panel.

  5. In the Component Library panel's left-side navigation pane, select Input. Find the TDengine component in the input widget list on the right and drag it to the canvas.

  6. Click the image icon in the TDengine input widget card to open the TDengine Input Configuration dialog box.

  7. In the TDengine Input Configuration dialog box, configure the parameters.

    Parameter

    Description

    Step Name

    The name of the TDengine input widget. Dataphin automatically generates the step name, but you can modify it as needed. The name must meet the following requirements:

    • Can only contain Chinese characters, letters, underscores (_), and numbers.

    • Cannot exceed 64 characters.

    Datasource

    Lists all TDengine-type data sources in the current Dataphin, including those you have read-through permission for and those you do not. Click the image icon to copy the current data source name.

    For data sources without read-through permission, click Request next to the data source to request read-through permission. For more information, see request, renew, and return data source permissions.

    If you do not have a TDengine-type data source, click Create Data Source to create one. For more information, see create TDengine data source.

    Database (optional)

    Select the database where the table is located. If left empty, the database specified during data source registration is used.

    Table

    Enter table name keywords to search, or enter the exact table name and click Exact Search. After you select a table, the system automatically detects the table status. Click the image icon to copy the name of the currently selected table.

    Shard Key (optional)

    Shards data based on the configured shard key field, enabling concurrent reading when used with the concurrent degree setting. You can use a column in the source table as the shard key. We recommend using the primary key or an indexed column as the shard key for optimal transmission performance.

    Important

    For a date-time type shard key, the system performs brute-force sharding based on the total time range and concurrent degree by identifying the maximum and minimum values. No guarantee of average distribution.

    Batch Read Count (optional)

    The number of records read per batch. Configure a batch read count (such as 1024 records) instead of reading records one by one. This reduces interactions with the data source, improves I/O efficiency, and lowers network latency.

    Input Filter (optional)

    Enter a condition expression supported by the TDengine database.

    • Only enter the content after the where keyword. Do not include the where keyword itself.

    • Supports system global variables, such as the data timestamp ${bizdate}.

    Output Fields

    Displays all fields matched by the selected table and filter conditions. Drag the image icon to adjust the field order. To exclude certain fields from downstream widgets, delete the corresponding fields.

    • Single Field Deletion Scenario: To delete a small number of fields, click the sgaga icon in the operation column.

    • Batch Field Deletion Scenario: To delete many fields at once, click Field Management, select multiple fields in the Field Management dialog box, click the image left shift icon to move selected fields to unselected, and click Confirm.

      image..png

  8. Click Confirm to complete the TDengine input widget configuration.