RDS incremental import

Usage notes

The RDS incremental import feature was discontinued on March 10, 2023. Lindorm Tunnel Service (LTS) instances purchased after this date do not support this feature. LTS instances purchased before March 10, 2023 can continue to use this feature.

Prerequisites

• The LTS instance was purchased before March 10, 2023.

• Purchase the data migration and synchronization service for Lindorm Tunnel Service (LTS) and configure the username and password for the LTS console.

• Establish a network connection between your LTS instance and the HBase migration cluster.

• Add an HBase cluster data source.

• Activate your Data Transmission Service (DTS) subscription and add a DTS data source.

Applicable versions

• Self-managed HBase 1.x and 2.x

• EMR HBase

• ApsaraDB for HBase Standard Edition and ApsaraDB for Lindorm (standalone and cluster editions)

• ApsaraDB for HBase Phoenix

Procedure

1. Log on to the Lindorm Tunnel Service (LTS) console. For more information, see Log on to LTS.

2. In the left-side navigation pane, choose Import to Lindorm/HBase > RDS Real-time Change Tracking to open the DTS Incremental Data Subscription page.

3. Click Create Channel, configure the ApsaraDB RDS data source and Destination data source (HBase, Phoenix), and then enter the mapping information for the tables to migrate.

4. Click Create Synchronization Channel.

Parameters

• HBase table mapping

  {
    "mapping": [
      {
        "columns": [
          {
            "name": "cf1:hhh",
            "value": "{{ concat(title, id) }}"
          },
          {
            "name": "cf1:title",
            "value": "title"
          },
          {
            "name": "cf1:*"
          }
        ],
        "config": {
          "skipDelete": true
        },
        "rowkey": {
          "value": "{{ concat('idg', id) }}"
        },
        "srcTableName": "hhh_test.test",
        "targetTableName": "default:_test"
      }
    ]
  }
                      

  Parameter	Description	Required
  mapping[y].srcTableName	The name of the ApsaraDB RDS source table.	Yes
  mapping[y].targetTableName	The name of the HBase destination table.	Yes
  mapping[y].columns	The column mappings between the ApsaraDB RDS table and the HBase table.	Yes
  mapping[y].columns[x].name	The name of the corresponding column in the HBase table.	Yes
  mapping[y].columns[x].value	The expression used to generate the value for the corresponding HBase column. It uses Jtwig syntax and allows you to perform simple calculations on columns from the source table.	Yes
  mapping[y].config	The synchronization policy for the table.	No
  mapping[y].rowkey	The rule for generating the rowkey of the HBase table.	Yes

  • Simple expressions are supported, as shown in the following example:

    {
      "name": "cf1:hhh",
      "value": "{{ concat(title, id) }}"
    }
                                

  • Dynamic columns are supported. Unmapped columns use the default mapping.

    {
        "name": "cf1:*",
    }
                                

  • You can specify a start time for the subscription. The Data Transmission Service (DTS) subscription channel contains only data after the specified timestamp.

    {
      "config": {
          "startOffset":1569463200 // Unit: seconds
      },
      "mapping": [
          "srcTableName": "hhh_test.test",
          "targetTableName": "default:test",
          "columns": [
            {
              "name": "cf1:*"
            }
          ],
          "config": {
            "skipDelete": true
          },
          "rowkey": {
            "value": "{{ concat('idg', id) }}"
          }
        }
      ]
    }
                                

  • DML support

    Actions	Supported	Notes
    INSERT	Yes	Corresponds to the PUT operation in HBase.
    UPDATE	Yes	Corresponds to the PUT operation in HBase.
    DELETE	Yes	You can configure whether to synchronize DELETE operations from the source; by default, they are not synchronized.

• Phoenix table mapping

  {
    "mapping": [
      {
        "srcTableName": "hhh_test.phoenix_test",
        "targetTableName": "phoenix_test",
        "config": {
          "skipDelete": true
        },
        "columns": [
          {
            "name": "id",
            "isPk": true
          },
          {
            "name": "title",
            "value": "title"
          },
          {
            "name": "ts",
            "value": "ts"
          },
          {
            "name": "datetime",
            "value": "datetime"
          }
        ]
      }
    ]
  }
                      

  Parameter	Description	Required
  mapping[y].srcTableName	The name of the ApsaraDB RDS source table.	Yes
  mapping[y].targetTableName	The name of the Phoenix destination table.	Yes
  mapping[y].columns	The column mappings between the ApsaraDB RDS table and the Phoenix table.	Yes
  mapping[y].columns[x].name	The name of the column in the Phoenix table.	Yes
  mapping[y].columns[x].value	The name of the corresponding column in the ApsaraDB RDS table.	Yes
  mapping[y].columns[x].isPk	Specifies if the column is a primary key column.	Yes
  mapping[y].config	The synchronization policy for the table.	No
  mapping[y].rowkey	The rule for generating the rowkey of the HBase table.	Yes