RDS incremental import

Usage notes

The RDS incremental import feature was discontinued on March 10, 2023. Lindorm Tunnel Service (LTS) instances purchased after this date do not support this feature. LTS instances purchased before March 10, 2023 can continue to use this feature.

Prerequisites

• The LTS instance was purchased before March 10, 2023.

• You have purchased Lindorm Tunnel Service (LTS) and configured the username and password for the LTS web UI.
• You have established a network connection between LTS and the target HBase migration cluster.
• You have added the HBase cluster as a data source.
• You have activated Data Transmission Service (DTS) and added a DTS data source.

Applicable versions

• self-managed HBase 1.x and 2.x
• EMR HBase
• ApsaraDB for HBase Standard Edition and ApsaraDB for HBase Performance-enhanced Edition (single-node and cluster instances)
• ApsaraDB for HBase Phoenix

Create a channel

1. Log on to the LTS web UI. In the left-side navigation pane, choose Import to Lindorm/HBase > RDS Real-time Change Tracking. This opens the DTS Incremental Data Subscription page with sections for Running Channels and Finished Channels.
2. Click Create Channel. Set the ApsaraDB RDS data source and the Target data source (HBase, Phoenix), and then provide the table mapping information. Enter a Channel Name (for example, test). In the table mapping text box, configure the JSON parameters, such as startOffset (the start offset timestamp), the columns mapping within mapping (for example, f1:*), and skipDelete (whether to skip delete operations). When you are finished, click Create Sync Channel.
3. Click Create Sync Channel.

HBase table mapping

{
  "mapping": [
    {
      "columns": [
        {
          "name": "cf1:hhh",
          "value": "{{ concat(title, id) }}"
        },
        {
          "name": "cf1:title",
          "value": "title"
        },
        {
          "name": "cf1:*"
        }
      ],
      "config": {
        "skipDelete": true
      },
      "rowkey": {
        "value": "{{ concat('idg', id) }}"
      },
      "srcTableName": "hhh_test.test",
      "targetTableName": "default:_test"
    }
  ]
}

• Simple expressions are supported. For example:

  {
    "name": "cf1:hhh",
    "value": "{{ concat(title, id) }}"
  }

• Wildcard column mapping is supported, allowing you to map all source columns to a column family without defining them individually.

  {
      "name": "cf1:*",
  }

• You can specify a start time for the subscription. The DTS subscription channel then includes data generated after the specified timestamp.

  {
    "config": {
        "startOffset":1569463200 // In seconds
    },
    "mapping": [
        "srcTableName": "hhh_test.test",
        "targetTableName": "default:test",
        "columns": [
          {
            "name": "cf1:*"
          }
        ],
        "config": {
          "skipDelete": true
        },
        "rowkey": {
          "value": "{{ concat('idg', id) }}"
        }
      }
    ]
  }

• The supported DML operations are listed in the following table.

  Actions	Supported	Description
  INSERT	Yes	Corresponds to a PUT operation in HBase.
  UPDATE	Yes	Corresponds to a PUT operation in HBase.
  DELETE	Yes	Configurable. DELETE operations are not replicated by default.

Phoenix table mapping

{
  "mapping": [
    {
      "srcTableName": "hhh_test.phoenix_test",
      "targetTableName": "phoenix_test",
      "config": {
        "skipDelete": true
      },
      "columns": [
        {
          "name": "id",
          "isPk": true
        },
        {
          "name": "title",
          "value": "title"
        },
        {
          "name": "ts",
          "value": "ts"
        },
        {
          "name": "datetime",
          "value": "datetime"
        }
      ]
    }
  ]
}