Migrate data from Elasticsearch to MaxCompute

Procedure

1. Create a source table in Elasticsearch. For more information, see Use DataWorks to synchronize MaxCompute data to Alibaba Cloud Elasticsearch.

2. Create a destination table in MaxCompute.

   1. Log on to the DataWorks console. In the target region, click Data Development and O&M > Data Development in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.

   2. Right-click the workflow that you created and choose Create Table > MaxCompute > Table.

   3. In the Create Table dialog box, enter a Name and click Create.

      Note

      If multiple MaxCompute compute resources are associated with DataStudio, select the required MaxCompute Engine Instance.

   4. At the top of the table editor page, click DDL.

   5. In the DDL dialog box, enter the following statement and click Generate Table Schema.

      create table elastic2mc_bankdata 
      (
      age             string,
      job             string,
      marital         string,
      education       string,
      default         string,
      housing         string,
      loan            string,
      contact         string,
      month           string,
      day_of_week     string
      );

   6. Click Submit to Production Environment.

3. Synchronize the data.

   1. Go to the data analytics page. Right-click the specified workflow and choose Create Node > Data Integration > Offline synchronization.

   2. In the New node dialog box, enter a Name and click OK.

   3. In the top navigation bar, choose icon.

   4. In script mode, click icon.

   5. In import Template dialog box SOURCE type, data source, target type and data source, and click confirm.

   6. Configure the script.

      For more information about script parameters, see Elasticsearch Reader.

      {
       "type": "job",
       "steps": [
       {
       "stepType": "elasticsearch",
       "parameter": {
       "retryCount": 3,
       "column": [
       "age",
       "job",
       "marital",
       "education",
       "default",
       "housing",
       "loan",
       "contact",
       "month",
       "day_of_week",
       "duration",
       "campaign",
       "pdays",
       "previous",
       "poutcome",
       "emp_var_rate",
       "cons_price_idx",
       "cons_conf_idx",
       "euribor3m",
       "nr_employed",
       "y"
       ],
       "scroll": "1m",
       "index": "es_index",
       "pageSize": 1,
       "sort": {
       "age": "asc"
      },
       "type": "elasticsearch",
       "connTimeOut": 1000,
       "retrySleepTime": 1000,
       "endpoint": "http://es-cn-xxxx.xxxx.xxxx.xxxx.com:9200",
       "password": "xxxx",
       "search": {
       "match_all": {}
       },
       "readTimeOut": 5000,
       "username": "xxxx"
       },
       "name": "Reader",
       "category": "reader"
       },
       {
       "stepType": "odps",
       "parameter": {
       "partition": "",
       "truncate": true,
       "compress": false,
       "datasource": "odps_source",// The name of the MaxCompute data source.
       "column": [
       "age",
       "job",
       "marital",
       "education",
       "default",
       "housing",
       "loan",
       "contact",
       "month",
       "day_of_week",
       "duration",
       "campaign",
       "pdays",
       "previous",
       "poutcome",
       "emp_var_rate",
       "cons_price_idx",
       "cons_conf_idx",
       "euribor3m",
       "nr_employed",
       "y"
       ],
       "emptyAsNull": false,
       "table": "elastic2mc_bankdata"
       },
       "name": "Writer",
       "category": "writer"
       }
       ],
       "version": "2.0",
       "order": {
       "hops": [
       {
       "from": "Reader",
       "to": "Writer"
       }
       ]
       },
       "setting": {
       "errorLimit": {
       "record": "0"
       },
       "speed": {
       "throttle": false,
       "concurrent": 1,
       "dmu": 1
       }
       }
      }

      Note

      You can view the public endpoint and port number of your Alibaba Cloud Elasticsearch cluster on the Basic Information page.

   7. Click the Run icon to execute the code.

   8. You can view the results in the Run log.

4. Verify the result.

   1. Right-click the workflow and choose new > MaxCompute > ODPS SQL.

   2. In create a node dialog box, enter node name, and click submit.

   3. In the ODPS SQL node editor, enter the following statement.

      SELECT * FROM elastic2mc_bankdata;

   4. Click icon to run the code.

   5. You can operation Log view the results.