Use the logstash-input-datahub plug-in

Prerequisites

Before you begin, ensure that you have:

• Installed the logstash-input-datahub plug-in. For more information, see Install and remove a plug-in.

• Activated DataHub, created a project and a topic, and imported data to the topic. For more information, see Get started with DataHub.

Configure the pipeline

Create a pipeline by following Use configuration files to manage pipelines. When configuring the pipeline, set the parameters described in the Parameters section. After you save and deploy the pipeline, Logstash reads data from DataHub and transfers it to the destination.

The following example shows a complete pipeline configuration:

input {
    datahub {
        access_id  => "<your-access-key-id>"
        access_key => "<your-access-key-secret>"
        endpoint   => "<datahub-endpoint>"
        project_name => "test_project"
        topic_name   => "test_topic"
        interval   => 5
        # cursor: start point per shard (leave blank to consume from the beginning)
        #cursor => {
        #    "0"=>"20000000000000000000000003110091"
        #    "2"=>"20000000000000000000000003110091"
        #    "1"=>"20000000000000000000000003110091"
        #    "4"=>"20000000000000000000000003110091"
        #    "3"=>"20000000000000000000000003110000"
        #}
        shard_ids => []
        pos_file  => "/ssd/1/<logstash-cluster-id>/logstash/data/<filename>"
    }
}
output {
    elasticsearch {
        hosts         => ["http://es-cn-mp91cbxsm000c****.elasticsearch.aliyuncs.com:9200"]
        user          => "elastic"
        password      => "your_password"
        index         => "datahubtest"
        document_type => "_doc"
    }
}

Parameters

The following table describes the parameters supported by logstash-input-datahub.