DocsICP FilingConsole
Documentation

Configure DataHub input

Updated at:
Copy as MD
Product
My Favorites

You can use a Real-time Synchronization task to capture data from DataHub in real time. This topic shows you how to configure the DataHub input component for the task in a simple drag-and-drop editor.

How it works

When you start a Real-time Synchronization task, it runs continuously. The task waits for new data in the source DataHub, reads it, and then writes it to the destination. The DataHub input component provides the following features:

  • Reads data in real time.

  • Performs concurrent reads based on the number of DataHub shards.

Procedure

  1. Log on to the DataWorks console. In the target region, click Data Development and O&M > Data Development in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.

  2. Hover over the 新建 icon and choose Create Node > Data Integration > Real-time Synchronization.

    Alternatively, expand a scheduled workflow, right-click it, and choose Create Node > Data Integration > Real-time Synchronization.

  3. In the Create Node dialog box, set Synchronization Method to Single Table (Topic) to Single Table (Topic) ETL, enter a Name, and select a Path.

  4. Click Confirm.

  5. On the configuration page of the Real-time Synchronization node, click Input and drag it to the canvas.

  6. Click the DataHub node. In the Node Configuration dialog box, configure the parameters.

    Parameter

    Description

    Data Source

    Select a configured DataHub data source. Only DataHub data sources are supported.

    If no data source is configured, click New data source on the right to go to the Workspace Configurations > Data Source page and create a data source. For more information, see Configure a DataHub data source.

    Topic

    Select the topic to synchronize from the current data source. You can click Data Preview on the right to confirm your selection.

    Use Subscription Feature

    If you enable this feature, a Subscription ID is automatically created to subscribe to data from DataHub, which improves stability and performance. To prevent failures, do not delete a Subscription ID that is in use on DataHub.

    Output Field

    Select the fields to synchronize.

  7. Click the 保存 icon in the toolbar to save the configuration.