How to get started with an Alibaba Cloud Logstash cluster-Elasticsearch(ES)-阿里云帮助中心

Background

Before you begin, make sure you understand the following concepts:

Prerequisites

You must have an Alibaba Cloud account.

For more information, see Create an Alibaba Cloud account.
You must have a VPC and a vSwitch.

For more information, see Create a VPC with an IPv4 CIDR block.

Limitations

The source Alibaba Cloud Elasticsearch cluster, Alibaba Cloud Logstash cluster, and destination Alibaba Cloud Elasticsearch cluster must be in the same VPC. If they are in different VPCs, you must configure a NAT Gateway to enable internet access. For more information, see Configure a NAT Gateway for public data transfer.
The versions of the source Alibaba Cloud Elasticsearch cluster, Alibaba Cloud Logstash cluster, and destination Alibaba Cloud Elasticsearch cluster must be compatible. For more information, see Compatibility.

Procedure

Step 1: Prepare the environment

Create a source and a destination Alibaba Cloud Elasticsearch cluster, enable auto-indexing for the destination cluster, and prepare test data.
Step 2: Create an Alibaba Cloud Logstash cluster

Create an Alibaba Cloud Logstash cluster. You can create and run a pipeline only after the cluster state changes to Active.
Step 3: Create and run a pipeline

Create and configure an Alibaba Cloud Logstash pipeline, and then run it to synchronize data.
Step 4: Verify the data synchronization results

Check the synchronization results in the Kibana console of the destination Alibaba Cloud Elasticsearch cluster.

Step 1: Prepare the environment

Create Alibaba Cloud Elasticsearch clusters.
1. Log on to the Alibaba Cloud Elasticsearch console.
2. In the left-side navigation pane, click Elasticsearch Clusters.
3. On the Elasticsearch Clusters page, create two Alibaba Cloud Elasticsearch clusters.
  These two clusters will serve as the input and output for Logstash. For more information, see Create an Alibaba Cloud Elasticsearch cluster. This tutorial uses the following data migration path: Alibaba Cloud Elasticsearch 6.7.0 (Commercial Edition) > Alibaba Cloud Logstash 6.7.0 > Alibaba Cloud Elasticsearch 6.7.0 (Commercial Edition). The provided scripts are compatible only with this path. This tutorial uses the following configurations for the Alibaba Cloud Elasticsearch clusters.
  Note
  
  If you are using a different migration path, see Compatibility to check for compatibility issues. If issues exist, you must upgrade your clusters or purchase new ones.
  
  The default username for accessing an Alibaba Cloud Elasticsearch cluster is elastic, which is used in this tutorial. If you need to use a custom user, you must grant the necessary roles and permissions. For more information, see Control user permissions by using Elasticsearch X-Pack roles.
Enable auto-indexing for the destination Alibaba Cloud Elasticsearch cluster.

For more information, see Configure YML parameters.

Note
To ensure data security, Alibaba Cloud Elasticsearch sets the Auto Indexing feature to Not allowed by default. When Alibaba Cloud Logstash sends data, it creates an index by submitting the data, not by using the create index API. Therefore, before you use Alibaba Cloud Logstash to upload data, you must either set the Auto Indexing feature for your cluster to Permitted or create an index and mappings in advance.

Prepare test data.

Go to the Kibana console of the source Alibaba Cloud Elasticsearch cluster. On the Dev Tools page, open the Console tab and run the following commands to create the index and documents you want to synchronize.

Important

For more information about how to log on to the Kibana console, see Log on to the Kibana console.
The following script is an example for Elasticsearch 6.7 and is for testing purposes only. For sample scripts for Elasticsearch 7.0 or later, see Get started with Elasticsearch.

Create an index my_index with a type my_type.

PUT /my_index
{
    "settings" : {
      "index" : {
        "number_of_shards" : "5",
        "number_of_replicas" : "1"
      }
    },
    "mappings" : {
        "my_type" : {
            "properties" : {
              "post_date": {          
                   "type": "date"       
               },
              "tags": {
                   "type": "keyword"
               },
                "title" : {
                    "type" : "text"
                }
            }
        }
    }
}

Insert a document with ID 1 into the my_index index.

PUT /my_index/my_type/1?pretty
{
  "title": "One", 
  "tags": ["ruby"],
  "post_date":"2009-11-15T13:00:00"
}

Insert a document with ID 2 into the my_index index.

PUT /my_index/my_type/2?pretty
{
  "title": "Two", 
  "tags": ["ruby"],
  "post_date":"2009-11-15T14:00:00"
}

Step 2: Create a Logstash cluster

Go to the Logstash Clusters page.
1. Go to the Logstash Clusters page.
2. In the top navigation bar, select the same region as your destination Alibaba Cloud Elasticsearch cluster.
3. In the left-side navigation pane, click Logstash Clusters.
On the Logstash Clusters page, click Create.
On the purchase page, on the first three configuration pages, configure the cluster launch settings.
This tutorial selects Pay-As-You-Go as the billing method and 6.7 as the version. Leave all other settings at their default values. For more information about the configurations, see Create an Alibaba Cloud Logstash cluster.
Note
- For development and functional testing, we recommend that you purchase a Pay-as-you-go cluster.
- You can receive discounts for purchasing a Subscription cluster.
Click Next: Confirm Order to preview the cluster configuration.

If the configuration does not meet your expectations, click the icon to modify it.
Read and select the terms of service, and then click DataWorks Exclusive Resources (Subscription).
After the success message appears, click DataWorks Exclusive Resources (Subscription).
In the top navigation bar, select the region where the cluster is located. In the left-side navigation pane, click Logstash Clusters. On the Logstash Clusters page, view the newly created cluster.

Step 3: Create and run a pipeline

After the state of the Logstash cluster changes to Normal, you can create and run a pipeline to synchronize data.

On the Logstash Clusters page, click Pipelines in the Actions column of the cluster.
In the Pipelines section, click Create Pipeline.

Enter a Pipeline ID and define the Config Settings.

This tutorial uses the following configuration.

input {
    elasticsearch {
        hosts => ["http://es-cn-0pp1f1y5g000h****.elasticsearch.aliyuncs.com:9200"]
        user => "elastic"
        password => "your_password"
        index => "*,-.monitoring*,-.security*,-.kibana*"
        docinfo => true
    }
}
filter {}
output {
    elasticsearch {
        hosts => ["http://es-cn-mp91cbxsm000c****.elasticsearch.aliyuncs.com:9200"]
        user => "elastic"
        password => "your_password"
        index => "%{[@metadata][_index]}"
        document_type => "%{[@metadata][_type]}"
        document_id => "%{[@metadata][_id]}"
    }
    file_extend {
        path => "/ssd/1/ls-cn-v0h1kzca****/logstash/logs/debug/test"
    }
}

Parameter	Description
`hosts`	The access address of the Alibaba Cloud Elasticsearch service. For input, the address is `http://<Source cluster ID>.elasticsearch.aliyuncs.com:9200`. For output, the address is `http://<Destination cluster ID>.elasticsearch.aliyuncs.com:9200`.
`user`	The username for accessing the Alibaba Cloud Elasticsearch service. The default is `elastic`.
`password`	The password for the user. The password for the `elastic` user is set during cluster creation. If you forget it, you can reset it. For more information, see Reset the access password of a cluster.
`index`	The index to synchronize. Setting this to `,-.monitoring,-.security,-.kibana` synchronizes all indices except for system indices that start with a period (`.`). `%{[@metadata][_index]}` matches the index from the metadata, which means the destination index will have the same name as the source index. Note System indices, which typically store monitoring logs for the Elasticsearch cluster, do not need to be synchronized.
`docinfo`	When set to true, this parameter extracts metadata from the Elasticsearch document, such as index, type, and ID.
`document_type`	Specifies the type of the destination index. Set this parameter to `%{[@metadata][_type]}` to match the `type` from the metadata. This ensures that the destination index has the same type as the source index.
`document_id`	Specify the ID of the synchronized document. Set this to `%{[@metadata][_id]}` to match the ID from the metadata. This means the ID of the synchronized document is the same as the source document ID.
`file_extend`	Optional. Enables the debug logging feature and uses the path parameter to configure the output path for debug logs. We recommend that you configure this parameter because it allows you to view the output directly in the console. If this parameter is not configured, you must check the output at the target destination and then return to the console to make modifications, which is time-consuming. For more information, see Use a Logstash Pipeline to Configure the Debugging Feature. Important Before you use the file_extend parameter, you must install the logstash-output-file_extend plugin. For more information, see Install or uninstall plugins. The path parameter defaults to a system-specified path that you should not modify. You can also get the path by clicking Start Configuration Debug.

For more information about the structure of a configuration file and the supported data types, which may vary by version, see Structure of a Config File.

Click Next step to configure pipeline parameters.

Set Pipeline Workers to the number of CPU cores in your cluster. Leave other parameters at their default values. For detailed parameter descriptions, see Manage pipelines by using configuration files.
Click Save or Save and Deploy.
- Save: Saves the pipeline configuration but does not apply it. After saving, you are returned to the Pipelines page. In the Pipelines section, you can click Deploy Now in the Actions column to restart the instance and apply the configuration.
- Save and Deploy: Saves and deploys the configuration, restarting the instance to apply the changes.
In the success dialog box, click OK.

You can then view the created pipeline in the Pipelines list. After the cluster update is complete and the pipeline Status shows as Running, Alibaba Cloud Logstash starts to synchronize the data.

Step 4: Verify the data synchronization results

After the pipeline is configured and running, you can verify the results in the Kibana console of the destination Alibaba Cloud Elasticsearch cluster.

Log on to the Kibana console of your Elasticsearch cluster.
For instructions, see Log on to the Kibana console.
Note
Examples here use Elasticsearch V6.7.0. Operations may vary slightly for other versions.
In the left navigation menu, choose Management > Dev Tools.

In the Console, run the following command to view the data synchronization results.

GET /my_index/_search
{
  "query": {
    "match_all": {}
  }
}

The expected output is as follows.

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 4,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "my_index",
        "_type" : "doc",
        "_id" : "aKG3bHABnsveUjNW2iXy",
        "_score" : 1.0,
        "_source" : {
          "post_date" : "2009-11-15T14:00:00",
          "@version" : "1",
          "@timestamp" : "2020-02-22T11:45:52.992Z",
          "tags" : [
            "ruby"
          ],
          "title" : "Two"
        }
      },
      ...
    ]
  }
}

If the data on the source and destination is consistent, the data synchronization is successful. You can also run the GET _cat/indices?v command to check if the sizes of the same indexes on the source and destination are consistent, and thereby determine if the data synchronization was successful.

Quick start