Import full data from ApsaraDB RDS

Usage notes

Lindorm Tunnel Service (LTS) no longer provides full data import from ApsaraDB RDS from March 10, 2023. If you purchased LTS after March 10, 2023, this feature is no longer available. If you purchased LTS before March 10, 2023, you can still use this feature.

Prerequisites

• The LTS instance was purchased before March 10, 2023.

• You have purchased LTS. You have configured the account name and password for the LTS console.
• Your LTS, destination ApsaraDB for HBase cluster, and source ApsaraDB RDS cluster are connected.
• You have added an ApsaraDB for HBase data source or ApsaraDB Phoenix data source.
• You have added an ApsaraDB RDS data source.

Supported editions

Create a task

1. Log on to the ApsaraDB for HBase console. Go to the LTS console. In the left-side navigation pane, select Data Import > RDS Full Data Import.
2. Click Create Migration Task. On the page that appears, specify the RDS data source and the destination ApsaraDB for HBase or Phoenix data source, and enter the mapping information about the table that you want to migrate.

   The page contains the following configuration items:

   • Channel Name: Enter the migration task name, such as job1.

   • RDS Data Source: Select the RDS data source that you have added.

   • Destination Data Source (HBase, Phoenix): Select the destination HBase instance.

   • Table Mapping: Configure in JSON format. The reader section uses multiple querySql statements to implement data sharding (for example, splitting by ID range). The writer section uses columns to define the mapping between HBase column families and columns, and supports expressions such as concat.

   After the configuration is complete, click Create Sync Channel.

3. View the task progress.

   The task progress page shows that the channel status in the Channel Overview section is SUCCEEDED, with migration progress at 2/2 and read/write details both at 1.6 kB. In the Subtask Details section, both DataXTask subtasks have the status SUCCEEDED, migrating 14 and 8 records respectively, for a total of 22 records.

4. After the migration task is complete, view the HBase table.

   
   idg6                                     column=f1:b, timestamp=1579420817743, value=\xEF\xBF\xBD\xEF\xBF\xBD^\xEF\xBF\xBD\xEF\xBF\xBD\xDF\xB4I
   idg6                                     column=f1:col1, timestamp=1579420817743, value=title16
   idg6                                     column=f1:col2, timestamp=1579420817743, value=title1
   idg6                                     column=f1:d, timestamp=1579420817743, value=0.234234234234234
   idg6                                     column=f1:date, timestamp=1579420817743, value=2019-01-02
   idg6                                     column=f1:datetime, timestamp=1579420817743, value=2019-01-02 20:14:55
   idg6                                     column=f1:dcm, timestamp=1579420817743, value=
   idg6                                     column=f1:f, timestamp=1579420817743, value=1.5
   idg6                                     column=f1:id, timestamp=1579420817743, value=6
   idg6                                     column=f1:te, timestamp=1579420817743, value=
   idg6                                     column=f1:time, timestamp=1579420817743, value=20:14:55
   idg6                                     column=f1:title, timestamp=1579420817743, value=title1
   idg6                                     column=f1:ts, timestamp=1579420817743, value=2019-01-02 20:14:55
   idg7                                     column=f1:b, timestamp=1579420817743, value=\xEF\xBF\xBD\xEF\xBF\xBD^\xEF\xBF\xBD\xEF\xBF\xBD\xDF\xB4I
   idg7                                     column=f1:col1, timestamp=1579420817743, value=title17
   idg7                                     column=f1:col2, timestamp=1579420817743, value=title1
   idg7                                     column=f1:d, timestamp=1579420817743, value=0.234234234234234
   idg7                                     column=f1:date, timestamp=1579420817743, value=2019-01-02
   idg7                                     column=f1:datetime, timestamp=1579420817743, value=2019-01-02 20:14:55
   idg7                                     column=f1:dcm, timestamp=1579420817743, value=
   idg7                                     column=f1:f, timestamp=1579420817743, value=1.5
   idg7                                     column=f1:id, timestamp=1579420817743, value=7
   idg7                                     column=f1:te, timestamp=1579420817743, value=
   idg7                                     column=f1:time, timestamp=1579420817743, value=20:14:55
   idg7                                     column=f1:title, timestamp=1579420817743, value=title1
   idg7                                     column=f1:ts, timestamp=1579420817743, value=2019-01-02 20:14:55
   

Map a MySQL table to an HBase table

{
  "reader": {
    "querySql": [
      "select id, title, content from rds.test where id < 8",
      "select id, title, content from rds.test where id >= 8"
    ]
  },
  "writer": {
    "columns": [
      {
        "name": "f1:col1",
        "value": "{{ concat(title, id) }}"
      },
      {
        "name": "f1:col2",
        "value": "content",
        "type": "string" 
      },
      {
        "name": "f1:*"
      }
    ],
    "rowkey": {
      "value": "{{ concat('idg', id) }}"
    },
    "tableName": "default:t1"
  }
}

{
  "name": "cf1:hhh",
  "value": "{{ concat(title, id) }}"
}

{
    "name": "cf1:*",
}

Map a MySQL table to a Phoenix table

{
  "reader": {
    "querySql": [
      "select id, title, ts, datetime, date, time, b, f, d from rds.test where id < 8",
      "select id, title, ts, datetime, date, time, b, f, d from rds.test where id >= 8"
    ]
  },
  "writer": {
    "columns": [
      {
        "isPk": true,
        "name": "id"
      },
      {
        "name": "title",
        "value": "title" 
      },
      {
        "name": "ts"
      },
      {
        "name": "datetime"
      },
      {
        "name": "date"
      },
      {
        "name": "time"
      },
      {
        "name": "b"
      },
      {
        "name": "f"
      },
      {
        "name": "d"
      }
    ],
    "tableName": "dtstest"
  }
}