Import incremental data from SLS

Note

The incremental data import feature from SLS was discontinued on March 10, 2023. LTS instances purchased after this date cannot use the incremental data import feature from SLS. LTS instances purchased on or before this date can still use this feature.

Prerequisites

• You have purchased an LTS instance, configured your account credentials, and logged on to the LTS console.

• You have created an HBase data source in the LTS console.

• You have created a loghub data source in the LTS console.

Supported HBase versions

• Self-managed HBase 1.x and 2.x with Kerberos disabled.

• E-MapReduce HBase.

• ApsaraDB for HBase Standard Edition and ApsaraDB for HBase Performance-enhanced Edition in cluster mode. The single-node mode is not supported.

Create a job

1. Log on to the LTS console. In the left-side navigation pane, choose Import to Lindorm/HBase > SLS Real-time Data Replication.

2. Click Create Job. Set the Source Cluster to loghub (SLS) and the Target Data Source (HBase, Phoenix) to your HBase data source, and configure the field mapping.

3. Click Create. You can then view the job's details.

Parameters

{
  "reader": {
    "columns": [
      "__client_ip__",
      "C_Source",
      "id",
      "name"
    ],
    "consumerSize": 2, // The number of consumers that subscribe to data from the loghub. Default: 1.
    "logstore": "bds-test"
  },
  "writer": {
    "columns": [
      {
        "name": "f:col1",
        "value": "{{ concat('xx', name) }}" // Expressions are supported.
      },
      {
        "name": "f:col2",
        "value": "__client_ip__" // Column name mapping.
      },
      {
        "name": "f:*" // Columns that are not explicitly matched are handled by the default mapping.
      }
    ],
    "rowkey": {
      "value": "{{ concat('idg',id) }}"
    },
    "tableName": "default:sls"
  }
}

{
  "name": "cf1:hhh",
  "value": "{{ concat(title, id) }}"
}