Migrate self-managed Elasticsearch cluster to Alibaba Cloud Elasticsearch, migrate self-managed Elasticsearch, data migration-Elasticsearch(ES)-阿里云帮助中心

Migrate data from a self-managed Elasticsearch cluster to Alibaba Cloud Elasticsearch by creating an OSS snapshot repository, backing up your indexes, and restoring the snapshots to the target cluster.

Limitations

The OSS bucket must meet the following requirements:
- The storage class must be Standard.
- The bucket must be in the same region as the Alibaba Cloud Elasticsearch cluster.
Cluster version: You can migrate data from an earlier minor version to a later minor version within the same major Elasticsearch version (for example, from 8.15.1 to 8.17.1). Migrating data across major versions (for example, from 7.7.1 to 8.17.1) is not recommended.

Note
Alibaba Cloud Elasticsearch is fully compatible with open-source Elasticsearch. However, data types can change between major open-source versions. For example, the string type in 5.x was split into text and keyword in 6.x. Migrating data across major versions can cause field type recognition errors.

Prerequisites

Create an Alibaba Cloud Elasticsearch cluster and an OSS bucket. The Alibaba Cloud Elasticsearch cluster must have the same major version as your self-managed cluster.
Ensure the RAM user has the AliyunOSSFullAccess permission policy. Grant permissions to a RAM user.

Background

This example uses an index named products_info to demonstrate the migration.

Create an index and insert data

Create an index:

curl -X PUT "http://localhost:9200/products_info" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "productName": { "type": "text" },
      "annual_rate": { "type": "text" },
      "describe": { "type": "text" }
    }
  }
}'

Insert data:

curl -X POST "http://localhost:9200/products_info/_bulk" -H 'Content-Type: application/json' -d'
{ "index": { "_id": "1" } }
{ "productName": "Financial Product A", "annual_rate": "3.2200%", "describe": "180-day fixed-term investment, 20,000 minimum, stable returns, optional message notifications." }
{ "index": { "_id": "2" } }
{ "productName": "Financial Product B", "annual_rate": "3.1100%", "describe": "90-day investment product, 10,000 minimum, daily return notifications." }
{ "index": { "_id": "3" } }
{ "productName": "Financial Product C", "annual_rate": "3.3500%", "describe": "270-day investment product, 40,000 minimum, instant daily return notifications." }
{ "index": { "_id": "4" } }
{ "productName": "Financial Product D", "annual_rate": "3.1200%", "describe": "90-day investment product, 12,000 minimum, daily return notifications." }
{ "index": { "_id": "5" } }
{ "productName": "Financial Product E", "annual_rate": "3.0100%", "describe": "Recommended 30-day investment product, 8,000 minimum, daily return notifications." }
{ "index": { "_id": "6" } }
{ "productName": "Financial Product F", "annual_rate": "2.7500%", "describe": "Popular short-term product, 3-day term, no fees, 500 minimum, get return notifications via SMS." }
'

Run curl -X GET "http://localhost:9200/products_info/_search?pretty". If you receive the following response, the index data is ready:

Sample response

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 6,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "products_info",
        "_type" : "_doc",
        "_id" : "kRjec5gBZDsv98SA0Bmx",
        "_score" : 1.0,
        "_source" : {
          "productName" : "Financial Product A",
          "annual_rate" : "3.2200%",
          "describe" : "180-day fixed-term investment, 20,000 minimum, stable returns, optional message notifications."
        }
      },
      {
        "_index" : "products_info",
        "_type" : "_doc",
        "_id" : "khjec5gBZDsv98SA0Bmy",
        "_score" : 1.0,
        "_source" : {
          "productName" : "Financial Product B",
          "annual_rate" : "3.1100%",
          "describe" : "90-day investment product, 10,000 minimum, daily return notifications."
        }
      },
      {
        "_index" : "products_info",
        "_type" : "_doc",
        "_id" : "kxjec5gBZDsv98SA0Bmy",
        "_score" : 1.0,
        "_source" : {
          "productName" : "Financial Product C",
          "annual_rate" : "3.3500%",
          "describe" : "270-day investment product, 40,000 minimum, instant daily return notifications."
        }
      },
      {
        "_index" : "products_info",
        "_type" : "_doc",
        "_id" : "lBjec5gBZDsv98SA0Bmy",
        "_score" : 1.0,
        "_source" : {
          "productName" : "Financial Product D",
          "annual_rate" : "3.1200%",
          "describe" : "90-day investment product, 12,000 minimum, daily return notifications."
        }
      },
      {
        "_index" : "products_info",
        "_type" : "_doc",
        "_id" : "lRjec5gBZDsv98SA0Bmy",
        "_score" : 1.0,
        "_source" : {
          "productName" : "Financial Product E",
          "annual_rate" : "3.0100%",
          "describe" : "Recommended 30-day investment product, 8,000 minimum, daily return notifications."
        }
      },
      {
        "_index" : "products_info",
        "_type" : "_doc",
        "_id" : "lhjec5gBZDsv98SA0Bmy",
        "_score" : 1.0,
        "_source" : {
          "productName" : "Financial Product F",
          "annual_rate" : "2.7500%",
          "describe" : "Popular short-term product, 3-day term, no fees, 500 minimum, get return notifications via SMS."
        }
      }
    ]
  }
}

Solution overview

Best suited for large datasets (over 500 GB). After the initial full backup, only incremental changes are synchronized, reducing overall migration time. Works well for cloud migration, cross-version upgrades, and disaster recovery.

Data migration

Step 1: Install the elasticsearch-repository-oss plugin

Connect to the ECS instance that hosts your self-managed Elasticsearch cluster and download the elasticsearch-repository-oss plugin. This example uses version 8.17.0.
```
wget https://github.com/aliyun/elasticsearch-repository-oss/releases/download/v8.17.0/elasticsearch-repository-oss-8.17.0.zip
```
For other versions, see Other versions of the elasticsearch-repository-oss plugin.

Install the plug-in on your Elasticsearch cluster.

# Install the plug-in by unzipping it to the plugins directory of each node in your Elasticsearch installation.
sudo unzip -d /usr/local/elasticsearch-8.17.0/plugins/elasticsearch-repository-oss elasticsearch-repository-oss-8.17.0.zip

Start the nodes in your self-managed Elasticsearch cluster.
```
cd /usr/local/elasticsearch-8.17.0
./bin/elasticsearch -d
```

Step 2: Create a snapshot repository for the self-managed cluster

On the server that hosts your self-managed Elasticsearch cluster, run the following command to create a snapshot repository:

Note

This command applies to Elasticsearch 8.x. For other versions, see Manual backup and restoration.

curl -X PUT "http://localhost:9200/_snapshot/<yourBackupName>" -H 'Content-Type: application/json' -d'
{
    "type": "oss",
    "settings": {
        "oss.client.endpoint": "oss-cn-shanghai.aliyuncs.com",
        "oss.client.access_key_id": "xxxx",
        "oss.client.secret_access_key": "xxxx",
        "oss.client.bucket": "<yourBucketName>",
        "oss.client.base_path":"snapshot/",
        "oss.client.compress": true
    }
}
'

Parameter	Description
<yourBackupName>	A custom name for the repository.
type	The repository type. Set to oss.
oss.client.endpoint	The OSS bucket endpoint. Regions and endpoints. Note If the ECS instance for your self-managed Elasticsearch cluster is in the same region as the OSS bucket, use the internal endpoint. Otherwise, use the public endpoint.
oss.client.access_key_id	The AccessKey ID of the account used to create the OSS bucket. Obtain an AccessKey pair.
oss.client.secret_access_key	The AccessKey secret of the account used to create the OSS bucket. Obtain an AccessKey pair.
oss.client.bucket	The name of the OSS bucket.
oss.client.base_path	The storage path for snapshots.
oss.client.compress	Specifies whether to enable compression: `true`: Enable compression to save storage space. `false`: Disable compression.

A successful creation returns the response {"acknowledged": true}.

Step 3: Create an index snapshot

On your self-managed Elasticsearch cluster, create a snapshot named snapshot_1 to back up the products_info index.

curl -X PUT localhost:9200/_snapshot/<yourBackupName>/snapshot_1?pretty -H "Content-Type: application/json" -d'
{
"indices": "products_info"
}'

<yourBackupName> is the repository name you specified when creating the snapshot repository.

A successful creation returns the response {"accepted": true}.

Run curl -X GET localhost:9200/_snapshot/<yourBackupName>/snapshot_1/_status to check the progress. When state is SUCCESS, the snapshot is complete.

Step 4: Create a repository on Alibaba Cloud Elasticsearch

In your Alibaba Cloud Elasticsearch cluster, log on to the Kibana console. In the left-side navigation pane, click the 展开图标 icon and choose Management > Developer Tools.

In the Shell, create a repository with the same settings as the one for your self-managed cluster:

PUT _snapshot/<yourBackupName>/
{
    "type": "oss",
    "settings": {
        "endpoint": "oss-cn-shanghai.aliyuncs.com",
        "access_key_id": "xxxx",
        "secret_access_key": "xxxxxx",
        "bucket": "<yourBucketName>",
        "compress": true,
        "chunk_size": "500mb",
        "base_path": "snapshot/"
    }
}

The <yourBackupName>, <yourBucketName>, and <base_path> values must match those used for the snapshot repository on your self-managed Elasticsearch cluster.

Step 5: Restore the snapshot on Alibaba Cloud Elasticsearch

In the Kibana console, restore snapshot_1 to the cluster. This restores all indexes except monitoring and security audit indexes, and skips unavailable indexes.

POST _snapshot/<yourBackupName>/snapshot_1/_restore
{"indices":"*,-.monitoring*,-.security_audit*","ignore_unavailable":"true"}

A successful request returns the response {"accepted": true}.

Run GET products_info/_recovery to check the restoration progress. When stage is DONE, the index has been restored.

Note

In Elasticsearch 7.10.x and 7.17.x, set cluster.restore.ignore_revision.enable to true to allow restoring snapshots with a higher revision number. The revision number is the third part of the version (for example, 1 in 7.10.1).

Example: After setting cluster.restore.ignore_revision.enable to true on a 7.10.0 cluster, you can restore snapshots from 7.10.2:

PUT _cluster/settings
{
  "persistent": {
    "cluster.restore.ignore_revision.enable": true
  }
}

Step 6: Verify the results

In the Kibana console, run the following commands to verify that the data has been restored:

View the restored indexes:
```
GET /_cat/indices?v
```

View data in the restored index:

GET /products_info/_search

Expected result:

{
  "took": 3,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 6,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "products_info",
        "_id": "1",
        "_score": 1,
        "_source": {
          "productName": "Financial Product A",
          "annual_rate": "3.2200%",
          "describe": "180-day fixed-term investment, 20,000 minimum, stable returns, optional message notifications."
        }
      },
      {
        "_index": "products_info",
        "_id": "2",
        "_score": 1,
        "_source": {
          "productName": "Financial Product B",
          "annual_rate": "3.1100%",
          "describe": "90-day investment product, 10,000 minimum, daily return notifications."
        }
      },
      {
        "_index": "products_info",
        "_id": "3",
        "_score": 1,
        "_source": {
          "productName": "Financial Product C",
          "annual_rate": "3.3500%",
          "describe": "270-day investment product, 40,000 minimum, instant daily return notifications."
        }
      },
      {
        "_index": "products_info",
        "_id": "4",
        "_score": 1,
        "_source": {
          "productName": "Financial Product D",
          "annual_rate": "3.1200%",
          "describe": "90-day investment product, 12,000 minimum, daily return notifications."
        }
      },
      {
        "_index": "products_info",
        "_id": "5",
        "_score": 1,
        "_source": {
          "productName": "Financial Product E",
          "annual_rate": "3.0100%",
          "describe": "Recommended 30-day investment product, 8,000 minimum, daily return notifications."
        }
      },
      {
        "_index": "products_info",
        "_id": "6",
        "_score": 1,
        "_source": {
          "productName": "Financial Product F",
          "annual_rate": "2.7500%",
          "describe": "Popular short-term product, 3-day term, no fees, 500 minimum, get return notifications via SMS."
        }
      }
    ]
  }
}

FAQ

Other versions of the elasticsearch-repository-oss plugin

Download the plugin from GitHub. If your exact version is unavailable, download a similar minor version within the same major version, update the values in plugin-descriptor.properties, repackage the plugin, and install it.

version=Required plug-in version
elasticsearch.version=Version of your self-managed Elasticsearch cluster

Note
The plug-in version must match the version of your self-managed Elasticsearch cluster.
java.version=1.8
Note
- Different Elasticsearch versions require different JDK versions. For more information, see the official Elasticsearch and plugin requirements.
- Different open-source Elasticsearch versions have different compilation methods. You must compile the elasticsearch-repository-oss plugin to match your version. For example, a self-managed 7.6.2 cluster requires JDK 1.8 or later. The corresponding plugin is elasticsearch-repository-oss-7.6.2.