Configure a Kafka writer

Procedure

1. Log on to the DataWorks console. In the target region, click Data Development and O&M > Data Development in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.

2. Hover over the icon and choose Create Node > Data Integration > Real-time Synchronization.

   Alternatively, expand a scheduled workflow, right-click it, and choose Create Node > Data Integration > Real-time Synchronization.

3. In the Create Node dialog box, set Synchronization Method to Single Table (Topic) to Single Table (Topic) ETL, enter a Name, and select a Path.

4. Click Confirm.

5. On the configuration tab of the real-time synchronization node, drag Output to the canvas and connect it to the configured reader or transformation node.

6. Click the Kafka node and configure its parameters in the Node Configuration panel that appears.

   Parameter	Description
   Data Source	Select a configured Kafka data source. Only Kafka data sources are supported. If no data source is configured, click New data source to create one on the Workspace Configurations > Data Source page. For more information, see Add a Kafka data source.
   Topic	The name of the Kafka topic. A topic is a category or feed name for messages that are processed by Kafka. Each message published to a Kafka cluster belongs to a category, which is called a Topic. A Topic is a grouping of a set of messages. Note A Kafka writer node supports writing data to only one topic.
   Key Column	The source columns whose values are concatenated to form the Kafka record key. If you select multiple columns, the column values are concatenated with a comma. If you do not select any columns, the key is empty.
   Value Column	The source columns whose values are concatenated to form the Kafka record value. If you leave this parameter empty, all columns are concatenated as the value by default. The concatenation method depends on the selected write mode. For more information, see the parameter description in Kafka writer.
   Key Type	The type of the Kafka key determines the key.serializer configuration for KafkaProducer initialization. The available values include STRING, BYTEARRAY, DOUBLE, FLOAT, INTEGER, LONG, and SHORT.
   Value Type	The data type of the Kafka value determines the setting for the value.serializer parameter when you initialize a KafkaProducer. Valid values include STRING, BYTEARRAY, DOUBLE, FLOAT, INTEGER, LONG, and SHORT.
   Amount of data written at one time	The size of a single write request, in bytes. A value greater than 16,000 is recommended.
   Write Mode	The format for concatenating source columns into the Kafka record value. Valid values: text and JSON. When set to text, all columns are concatenated with the column separator. In JSON format, all columns are concatenated into a JSON string. For example, if a source record has the values a, b, and c for the columns col1, col2, and col3, the value of the record written to Kafka is the string a#b#c when the write mode is set to text and the column separator is set to #. When the write mode is set to JSON, the value is the string {"col1":"a","col2":"b","col3":"c"}.
   Column Delimiter	When the write mode is configured as text, the source columns are concatenated by using the specified column separator to form the Value of the Kafka record. You can use a single character or multiple characters as the separator. You can also specify a Unicode character in the \u0001 format or an escape character such as \t or \n. The default value is \t.
   Configuration parameters	When you create a KafkaConsumer client, you can specify extended parameters, such as bootstrap.servers, acks, and linger.ms. You can use kafkaConfig to control the data consumption behavior of KafkaConsumer. For a real-time synchronization Kafka writer node, the default value of the acks parameter for KafkaConsumer is all. If you have higher performance requirements, you can override the default value by specifying a different value for acks in the configuration parameters. The possible values for acks are as follows: 0: The producer does not wait for an acknowledgment. 1: The producer waits for an acknowledgment from the leader replica only. all: The producer waits for an acknowledgment from all in-sync replicas.

7. Click the Save icon in the toolbar.