Configure DataHub Writer-阿里云帮助中心

DataHub is a platform for processing streaming data. It enables you to publish, subscribe to, and distribute streaming data, simplifying the creation of analyses and applications that use streaming data.

Prerequisites

Before you configure a DataHub writer, you must configure a corresponding reader or transformation node. For more information, see Data sources supported for real-time synchronization.

Background information

The writer uses the Java SDK version specified in the following dependency.

<dependency>
    <groupId>com.aliyun.datahub</groupId>
    <artifactId>aliyun-sdk-datahub</artifactId>
    <version>2.5.1</version>
</dependency>

Procedure

Log on to the DataWorks console. In the target region, click Data Development and O&M > Data Development in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.
Hover over the icon and choose Create Node > Data Integration > Real-time Synchronization.

Alternatively, expand a scheduled workflow, right-click it, and choose Create Node > Data Integration > Real-time Synchronization.
In the Create Node dialog box, set Synchronization Method to Single Table (Topic) to Single Table (Topic) ETL, enter a Name, and select a Path.
Click Confirm.
On the edit page of the real-time synchronization node, click and drag Output to the edit panel, and connect the node to a configured input or transformation node.

Click the DataHub node, and configure the parameters in the Node Configuration dialog box.

Parameter	Description
Data Source	Select a configured DataHub data source. If no data source is available, click New data source to go to the Workspace Settings > Data Source page and create a data source. For more information, see Configure a DataHub data source.
Topic	Select the destination topic from the current data source. You can click Data Preview on the right to verify your selection.
Write Mode	Specifies the mode for writing data to DataHub. The following modes are available: Tuple: Writes structured data. The destination DataHub topic must have a schema. Blob: Writes unstructured data as binary chunks. The destination DataHub topic must be of the Blob type.
Batch Size (Records)	Specifies the number of records to synchronize in each batch.
Field Mapping	Map fields from the source to the destination. The node uses these mappings to synchronize data.

Click the icon in the toolbar to save the configuration.