Elasticsearch migration method selection-Elasticsearch(ES)-阿里云帮助中心

You can use various methods, such as Logstash, reindex, Object Storage Service (OSS), and Data Online Migration, to migrate data between Alibaba Cloud Elasticsearch instances, migrate Elasticsearch data to Openstore storage, migrate data from a self-managed Elasticsearch cluster to Alibaba Cloud Elasticsearch, or migrate data from a third-party Elasticsearch service to Alibaba Cloud Elasticsearch. This topic compares data migration methods, their applicable scenarios, and their limitations to help you select the right one for your business.

Important

In October 2020, Alibaba Cloud Elasticsearch upgraded its network architecture. Instances created before this date use the legacy network architecture, while instances created in or after October 2020 use the new network architecture. Cross-cluster operations, such as remote reindex, cross-cluster search, and cross-cluster replication, are not supported between instances on the new and legacy network architectures. To perform these operations, ensure that both instances use the same network architecture. For instances in the China (Zhangjiakou) and overseas regions, the exact time of the network architecture upgrade varies. To verify network connectivity for cross-cluster operations, submit a ticket to contact Alibaba Cloud Elasticsearch technical support. To check the network architecture of an instance, log on to the Alibaba Cloud Elasticsearch console and find the Basic Information section on the Control Architecture Type page. Cloud-native Control Architecture (v3) indicates the new network architecture, and Basic Control Architecture (v2) indicates the legacy network architecture.
Do not migrate system indexes. These indexes are prefixed with a period (.), such as .monitoring, .kibana, or .security. Migrating these indexes may cause Kibana to fail.

Migration between Alibaba Cloud Elasticsearch instances

Migration method	Applicable scenarios	Limitations	Example
OSS snapshot	Migrating large datasets (GB, TB, or PB scale). Migrating snapshot data between instances in the same or different Alibaba Cloud accounts, within the same region or across regions. Note For cross-region data migration, you can use Elasticsearch snapshot and restore commands together with the cross-region replication (CRR) feature of OSS.	Migrating incremental data by using an OSS snapshot may require service downtime. When restoring incremental data on the new cluster, you must close the index. To use the cross-cluster OSS repository feature of Alibaba Cloud Elasticsearch, the source and target instances must meet the following requirements: Located in the same region. Belong to the same Alibaba Cloud account or RAM user. The version of the source instance must be earlier than or equal to the version of the target instance. For more information about version compatibility, see Compatibility matrixes. When using an API to migrate snapshot data across accounts, you must use the same AccessKey for both the source and target. Specifically, when adding the repository, use the AccessKey of the account that owns the Object Storage Service (OSS) bucket.	Back up and restore data by using a cross-cluster OSS repository (for automatic snapshots) Create manual snapshots and restore data from manual snapshots (for manual snapshots)
Logstash	Migrating full data or syncing incremental data when real-time performance is not critical. Important For cross-account and cross-region migrations where the Elasticsearch and Logstash instances are not in the same VPC, you must configure a NAT gateway for the Logstash instance to transmit data over the public network. For more information, see Migrate data from Tencent Cloud ES to Alibaba Cloud Elasticsearch. Migrating only query results. Filtering data before migration. Migrating data across major versions, such as from 5.x to 6.x or 7.x. For more information about version compatibility, see Compatibility matrixes.	The source Elasticsearch instance, Logstash instance, and target Elasticsearch instance must be in the same VPC. If not, you must configure a NAT gateway to enable public network connectivity. For more information, see Configure a NAT gateway for data transmission over the Internet. The versions of the source Elasticsearch, Logstash, and target Elasticsearch instances must be compatible. For more information, see Compatibility matrixes. To sync incremental data, ensure that the data IDs are consistent between the source and target, and configure a scheduled task.	Migrate data between Alibaba Cloud Elasticsearch clusters Use the reindex API to migrate data in a multi-type index of an earlier version
reindex	Migrating small datasets where migration speed is not a high priority. Migrating only query results (results from a query statement run in Kibana).	The two Elasticsearch clusters must use the same network architecture. For more information, see the notes at the beginning of this topic.	Use the reindex API to migrate data
elasticsearch-dump tool	Migrating small datasets with a limited number of indexes.	The source cluster, target cluster, and the server where the elasticsearch-dump tool is installed must have network connectivity.	Use elasticsearch-dump to migrate data
OSS snapshot and reindex	Migrating both full and incremental data. Migrating across major versions where the file format may be incompatible and direct snapshot restore is not supported. For example, to migrate from 6.x to 8.x, you can first restore a snapshot to a 7.x instance, reindex the data on the 7.x instance, and then restore a snapshot from the 7.x instance to the 8.x instance.	The intermediate version must be compatible with both the source and target versions. For more information, see Elastic version changes and compatibility.	Create manual snapshots and restore data from manual snapshots (for manual snapshots) Use the reindex API to migrate data
Data Online Migration	For smooth, highly available, and zero-downtime upgrades from an Alibaba Cloud Elasticsearch V2 instance to V3. This method migrates data while preserving the instance endpoint after the upgrade.		Upgrade instance architecture

Migration to Openstore storage

Migration method	Applicable scenarios	Limitations	Example
OSS snapshot	Migrating data from a self-managed or Alibaba Cloud Elasticsearch cluster to Openstore hot/cold tiered storage.	The Object Storage Service (OSS) bucket must be of the Standard storage class. The Archive storage class is not supported. The region of the OSS bucket must be the same as the region of the Elasticsearch instance.	Migrate Elasticsearch data to Openstore hot/cold tiered storage
Manual backup	Migrating data from a self-managed or Alibaba Cloud Elasticsearch cluster to Openstore intelligent hybrid storage.	The region of the OSS bucket must be the same as the region of the Elasticsearch instance.	Migrate Elasticsearch data to Openstore intelligent hybrid storage

Migration from self-managed to Alibaba Cloud Elasticsearch

Migration method	Applicable scenarios	Limitations	Example
Data Online Migration	Migrating data from a self-managed Elasticsearch cluster deployed on ECS instances within an IDC. For smooth, highly available, zero-downtime migrations from a self-managed Elasticsearch cluster.	Cross-region and cross-VPC data migration are not supported. Cross-major-version migration is not supported (for example, from 7.x to 8.x). Indexes in the self-managed Elasticsearch cluster must have at least one replica.	Migrate data from a self-managed Elasticsearch cluster to Alibaba Cloud Elasticsearch
OSS snapshot	Migrating large datasets (GB, TB, or PB scale). Migrating snapshot data in the same or different Alibaba Cloud accounts, within the same region or across regions.	You must obtain the elasticsearch-repository-oss plug-in that matches the source Elasticsearch version. Migrating incremental data by using an OSS snapshot may require service downtime. When restoring incremental data on the new cluster, you must close the index.	Use OSS to migrate data from a self-managed Elasticsearch cluster to an Alibaba Cloud Elasticsearch cluster
Logstash	Data migrations where real-time performance is not critical. Migrating only query results. Filtering data before migration. Migrating data across major versions, such as from 5.x to 6.x or 7.x. For more information about version compatibility, see Compatibility matrixes.	The source Elasticsearch instance, Logstash instance, and target Elasticsearch instance must be in the same VPC. If not, you must configure a NAT gateway to enable public network connectivity. For more information, see Configure a NAT gateway for data transmission over the Internet. The versions of the source Elasticsearch, Logstash, and target Elasticsearch instances must be compatible. For more information, see Compatibility matrixes. To sync incremental data, ensure that the data IDs are consistent between the source and target, and configure a scheduled task.	Use Alibaba Cloud Logstash to migrate data from a self-managed Elasticsearch cluster to an Alibaba Cloud Elasticsearch cluster Use the reindex API to migrate data in a multi-type index of an earlier version
reindex	Migrating small datasets where migration speed is not a high priority. Migrating only query results (results from a query statement run in Kibana). Migrating from an earlier-version self-managed Elasticsearch cluster to a later-version Alibaba Cloud Elasticsearch cluster, such as from 6.x to 8.x. This requires using PrivateLink to establish network connectivity for remote reindexing.	The two Elasticsearch clusters must use the same network architecture. For more information, see the notes at the beginning of this topic.	Use the reindex API to migrate data from a self-managed Elasticsearch cluster to an Alibaba Cloud Elasticsearch cluster
elasticsearch-dump tool	Migrating small datasets with a limited number of indexes.	The source cluster, the target cluster, and the server where the elasticsearch-dump tool is installed must have network connectivity.	Use elasticsearch-dump to migrate data

Migration from third-party to Alibaba Cloud Elasticsearch

Migration method	Applicable scenarios	Limitations	Example
Logstash	Migrating full data. Migrating only data that matches specific queries. Migrating data between Alibaba Cloud Elasticsearch instances across different accounts and regions.	You must configure a NAT gateway for the Logstash instance to enable public network connectivity. For more information, see Configure a NAT gateway for data transmission over the Internet. The versions of the source Elasticsearch, Logstash, and target Elasticsearch instances must be compatible. For more information, see Compatibility matrixes.	Migrate data from Tencent Cloud ES to Alibaba Cloud Elasticsearch
OSS snapshot	Migrating large datasets (GB, TB, or PB scale).	Migrating incremental data by using an OSS snapshot may require service downtime. When restoring incremental data on the new cluster, you must close the index.	Migrate Elasticsearch index data from Amazon OpenSearch Service to Alibaba Cloud Elasticsearch Note This method is not limited to specific cloud vendors and relies on the Elasticsearch snapshot mechanism. For example, you can use Alibaba Cloud Data Online Migration (Migrate data) to migrate Elasticsearch snapshot data from Tencent Cloud COS to Alibaba Cloud OSS, and then restore the data from the OSS repository. For more information, see Create manual snapshots and restore data from manual snapshots. Elasticsearch snapshot migration supports restoring data only to the same or a later version. For more information, see Version compatibility of data restoration from snapshots.
elasticsearch-dump tool	Migrating small datasets with a limited number of indexes.	The source cluster, the target cluster, and the server where the elasticsearch-dump tool is installed must have network connectivity.	Use elasticsearch-dump to migrate data