Migrate data from Tencent Cloud ES to Alibaba Cloud

Scenario description

The procedure in this topic migrates index data from a Tencent Cloud Elasticsearch cluster to an Alibaba Cloud Elasticsearch cluster. It also applies to cross-account and cross-region data migration between Alibaba Cloud Elasticsearch clusters. When you configure such a migration, replace the Tencent Cloud Elasticsearch configuration in the input section with the configuration of the source Alibaba Cloud Elasticsearch cluster.

Procedure

1. This section covers creating an Alibaba Cloud Elasticsearch instance, an Alibaba Cloud Logstash instance, and a Tencent Cloud Elasticsearch instance. You also enable the automatic index creation feature on the Alibaba Cloud Elasticsearch instance and set up a Tencent Cloud proxy server.

2. This section covers selecting compatible Tencent Cloud Elasticsearch, Alibaba Cloud Elasticsearch, and Logstash instances for the migration.

3. This section covers creating and configuring an Alibaba Cloud Logstash pipeline using a configuration file, with Tencent Cloud Elasticsearch as the input and Alibaba Cloud Elasticsearch as the output.
   Note

   You can also use a self-managed Logstash instance. Install JDK 1.8 or later and the corresponding Logstash version on the server, configure the YML file in the conf directory of Logstash, and then start the service. For more information, see .

4. This section covers verifying in Kibana that the indexes were successfully migrated, by running GET /_cat/indices?v.

Preparations

1. Create an Alibaba Cloud Elasticsearch instance and enable the automatic index creation feature.

   Before you create the instances, confirm the compatible version combinations in Step 1: Select a migration plan. For instructions, see Create an Alibaba Cloud Elasticsearch instance and Quick access and configuration.

2. Create an Alibaba Cloud Logstash instance.

   For more information, see Create an Alibaba Cloud Logstash instance.

   Alibaba Cloud Logstash instances are deployed in a virtual private cloud (VPC). During the migration, Logstash must connect to the Internet to communicate with Tencent Cloud Elasticsearch. Therefore, you must configure a NAT gateway to provide Internet access. For more information, see Configure a NAT gateway for public data transmission.

   Note

   For a self-managed Logstash instance, purchase an ECS instance in the same VPC as your Alibaba Cloud Elasticsearch instance. If you already have a compliant ECS instance, you do not need to purchase another one. You must associate the instance with an elastic IP address (EIP).

3. Create a Tencent Cloud Elasticsearch instance and set up a Tencent Cloud proxy server.

   Tencent Cloud Elasticsearch does not support Direct Internet Access. Therefore, you must prepare a Cloud Virtual Machine (CVM) server to set up an NGINX proxy. For more information, see Access a Tencent Cloud Elasticsearch cluster from the internet.

Step 1: Select a migration plan

Tencent Cloud Elasticsearch and Alibaba Cloud Elasticsearch use different versions, so you must select compatible versions for the migration. The steps that follow in this topic describe the Alibaba Cloud Logstash path. If you select a self-managed Logstash version, follow the separate procedure for self-managed Logstash clusters that the note in the Procedure section links to. This topic supports the following version plans, and compatibility is not guaranteed for other plans:

Step 2: Create and configure a Logstash pipeline

1. Go to the Logstash Clusters page in the Alibaba Cloud Elasticsearch console.

2. Go to the target instance.

   1. In the top menu bar, select the region where the target instance resides.

   2. On the Logstash Clusters page, click the ID of the target instance.

3. In the left-side navigation pane, click Pipelines.

4. Click Create Pipeline.

5. On the Create Pipeline page, enter a Pipeline ID and configure the pipeline.

   This topic uses the following pipeline configuration.

   input {
        elasticsearch {
          hosts => "http://xxxxxxxxx:9200"
          user  => "elastic"
          index => "*"
          password => "xxxxxx"
          docinfo => true
        }
      }
   output {
         elasticsearch {
           hosts => "http://xxxxxxx.elasticsearch.aliyuncs.com:9200"
           user => "elastic"
           password => "xxxxxx"
           index => "%{[@metadata][_index]}"
           document_type => "%{[@metadata][_type]}"
           document_id => "%{[@metadata][_id]}"
     }
   }

   Parameter	Description
   hosts	The endpoint used to access the Elasticsearch service. In the input section, set this parameter to http://<Public endpoint of the Tencent Cloud proxy service>:<Port>. In the output section, set this parameter to http://<Alibaba Cloud Elasticsearch instance ID>.elasticsearch.aliyuncs.com:9200.
   user	The username used to access the Elasticsearch service. The default value is elastic.
   password	The password of the user. For Alibaba Cloud Elasticsearch, the password of the elastic user is set when you create the instance. If you forget the password, you can reset it. For the usage notes and procedure for resetting a password, see Reset the access password of an instance.
   index	In the input section, this parameter specifies the name of the index to migrate. Set it to * to synchronize all indexes. In the output section, this parameter specifies the name of the migrated index. Set it to %{[@metadata][_index]} to match the index in the metadata, so that the migrated index has the same name as the source index.
   document_type	The type of the migrated index. Set this parameter to %{[@metadata][_type]} to match the type in the metadata, so that the migrated index has the same type as the source index.
   document_id	The ID of the migrated document. Set this parameter to %{[@metadata][_id]} to match the ID in the metadata, so that the migrated document has the same ID as the source document.
   docinfo	If you set this parameter to true, Logstash extracts the metadata of Elasticsearch documents, such as the index, type, and ID.

   For more information about Config settings, see Logstash configuration files.

   The preceding configuration synchronizes all indexes from the source Elasticsearch cluster to the destination cluster. You can also synchronize only specified indexes. The following example shows the input section of the pipeline configuration for that case.

   input {
        elasticsearch {
          hosts => "http://xxxxxxxx:9200"
          user  => "elastic"
          index => "alicloudtest"
          password => "xxxxxxx"
          query => '{ "query":{ "query_string": { "query": "*" } } }'
          docinfo => true
        }

   The Elasticsearch input plugin reads document data from an Elasticsearch cluster based on the query statement that you configure. This approach suits operations such as bulk-importing test logs. By default, synchronization stops automatically after all data is read. However, an Alibaba Cloud Logstash instance must keep its process running, and the process restarts after it stops. In single-task scenarios where the input section uses the elasticsearch input plugin, this behavior causes data to be written repeatedly. You can avoid duplicate writes by configuring a scheduled task with a long interval. For example, trigger the task at 13:20 on March 5 each year, and stop the pipeline after the first run. Implement this by using the cron syntax with the schedule parameter. For more information, see Scheduling in the Logstash documentation.

   For example, to run the task at 13:20 on March 5:

   schedule => "20 13 5 3 *"

6. Click Next step and configure the pipeline parameters.

   Parameter	Description
   Pipeline Workers	The number of worker threads used to run the filter and output stages of the pipeline in parallel. If events are backlogged or the CPU is not saturated, increase the number of threads to make better use of the CPU processing capacity. Default value: the number of CPU cores of the instance.
   Pipeline Batch Size	The maximum number of events that a single worker thread can collect from the input stage before it attempts to run the filter and output stages. A larger pipeline batch size may cause larger memory overhead. You can set the LS_HEAP_SIZE variable to increase the JVM heap size so that this value takes full effect. Default value: 125.
   Pipeline Batch Delay	The time in milliseconds to wait for each event in a batch before an undersized batch is dispatched to a pipeline worker thread. Default value: 50 ms.
   Queue Type	The internal queuing model used for event buffering. Valid values: MEMORY: the default value. A traditional memory-based queue. PERSISTED: a disk-based ACKed queue (persistent queue).
   Queue Max Bytes	The maximum data volume that the queue can store, in MB. The value must be an integer from 1 to 2^53-1. Default value: 1024MB. Make sure that this value is smaller than the total disk capacity.
   Queue Checkpoint Writes	When the persistent queue is enabled, this parameter specifies the maximum number of events that can be written before a checkpoint is forced. A value of 0 indicates no limit. Default value: 1024.

   Warning

   After you complete the configuration, you must save and deploy it for the configuration to take effect. Saving and deploying triggers an instance restart. Proceed only when the restart will not affect your business.

7. Click Save or Save and Deploy.

• Save: saves the pipeline configuration but does not apply it. After you save, the system returns to the Pipelines page. In the Pipelines section of that page, in the Actions column, click Deploy Now to trigger an instance restart and apply the configuration.

• Save and Deploy: saves the pipeline configuration and deploys it. Deploying triggers an instance restart and applies the configuration.

Step 3: Verify the result

1. Log on to the Kibana console of the target Alibaba Cloud Elasticsearch instance.

   For more information, see Log on to the Kibana console.

2. In the left-side navigation pane, click Dev Tools.

3. In Console, run the GET /_cat/indices?v command to verify that the indexes were successfully migrated.

References

• Troubleshoot Logstash data write issues.

• To migrate data from a Tencent Cloud ES cluster to an Alibaba Cloud ES cluster by using the ES snapshot mechanism, see Migrate Elasticsearch indexes from Amazon OpenSearch Service to Alibaba Cloud.