DocsICP FilingConsole
Documentation

Configure a LogHub (SLS) reader

Updated at:
Copy as MD
Product
My Favorites

The LogHub (SLS) reader uses the LogHub (SLS) software development kit (SDK) to extract real-time data from a LogHub (SLS) topic. The reader also supports shard merging and splitting. Shard merging or splitting may cause data duplication, but ensures no data is lost.

Overview

The LogHub (SLS) reader supports the following metadata fields for real-time synchronization:

Field

Type

Description

__time__

STRING

SLS reserved field: The time when the log data was written. The value is a UNIX timestamp in seconds.

__source__

STRING

SLS reserved field: The source device of the log.

__topic__

STRING

SLS reserved field: The name of the topic.

__tag__:__receive_time__

STRING

The time when the log arrived at the server. The server appends this field when public IP address recording is enabled. The value is a UNIX timestamp in seconds.

__tag__:__client_ip__

STRING

The public IP address of the source device. The server appends this field when public IP address recording is enabled.

__tag__:__path__

STRING

The log file path collected by Logtail. Logtail automatically appends this field.

__tag__:__hostname__

STRING

The hostname of the machine from which Logtail collected the data. Logtail automatically appends this field.

Procedure

  1. Log on to the DataWorks console. In the target region, click Data Development and O&M > Data Development in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.

  2. Hover over the 新建 icon and choose Create Node > Data Integration > Real-time Synchronization.

    Alternatively, expand a scheduled workflow, right-click it, and choose Create Node > Data Integration > Real-time Synchronization.

  3. In the Create Node dialog box, set Synchronization Method to Single Table (Topic) to Single Table (Topic) ETL, enter a Name, and select a Path.

  4. Click Confirm.

  5. On the configuration tab of the real-time synchronization node, drag the Input node to the canvas.

  6. Click the LogHub node. In the Node Configuration pane, configure the parameters.

    Parameter

    Description

    Data Source

    Select a configured LogHub (SLS) data source. Only LogHub (SLS) data sources are supported.

    If no data source is configured, click New data source on the right to go to the Management Center > Data Source page and create a data source. For more information, see Configure a LogHub (SLS) data source.

    Logstore

    Select the Logstore from which you want to synchronize data. You can click Data Preview on the right to confirm your choice.

    Advanced Configuration

    You can set whether to split data in the Logstore. If you select Split, you must configure a Split rules.

    The split rule uses the format shardId % X = Y. In this format, shardId is the ID of a data shard, X is the total number of parallel tasks, and Y is the index of the current task (from 0 to X-1). For example, shardId % 5 = 3 indicates that data is distributed across five tasks, and the current task (index 3) processes only the shards for which this condition is true.

    Output Field

    Select the fields to synchronize. For more information about the fields, see the Overview section.

  7. Click the 保存 icon in the toolbar to save the configuration.