SDK overview

Overview

DataHub offers two SDK types: Low-Level and High-Level.

• The Low-Level SDK implements DataHub server-side APIs, primarily for resource management.

• The High-Level SDK (client library) has two modules: Producer and Consumer. It wraps Low-Level SDK read/write APIs for easier data ingestion and consumption.

Low-Level SDK

Each Low-Level SDK API maps to a DataHub RESTful API. For example, the create-project API:

  POST /projects/<ProjectName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: application/json
  Content-Length: xxx
  {
      "Comment": "test project"
  }

The corresponding Java API is:

/**
* Create a datahub project.
*
* @param projectName The name of the project.
* @param comment     The comment of the project.
* @throws DatahubClientException Throws DatahubClientException
*/
CreateProjectResult createProject(String projectName, String comment);

High-Level SDK

The Low-Level SDK also handles data ingestion and consumption, but these operations require combining multiple APIs. Misuse can cause failures or excessive load. The High-Level SDK wraps these APIs to simplify read/write operations and improve stability.

Producer

The following compares data ingestion with the Low-Level SDK versus the Producer.

Using the Low-Level SDK:

1. Initialize the client.

2. Call GetTopic or ListSchema to retrieve schema information.

3. Call ListShard to get the list of shards and filter for those in the ACTIVE state.

4. Accumulate data and send in batches.

5. Select a shard and call PutRecordsByShard to write data.

6. If the request succeeds, repeat step 4.

7. If the request fails, retry step 5 for retryable errors or handle non-retryable ones accordingly.

8. Periodically call ListShard to update shard information.

Using the Producer (High-Level SDK for asynchronous sending):

1. Initialize the Producer and register a callback.

2. Call sendAsync to send data.

Advantages of the Producer over the Low-Level SDK:

• Simpler logic and improved usability.

• Built-in retry mechanism with configurable retries for exceptions such as network jitters.

• Automatically adapts to scaling in/out.

Consumer

Low-Level SDK consumption is complex:

• Each shard has a checkpoint that must be periodically committed and retrieved on startup.

• A single shard cannot be consumed by multiple nodes, making multi-node shard allocation difficult.

• For strict order requirements, handling shard splits/merges becomes complex. For example, if shard0 splits into shard1 and shard2, both inherit a portion of the hash range. Consumption must complete shard0 before starting on shard1 and shard2 to preserve order.

Low-Level SDK consumption process:

1. Initialize the client.

2. Call GetTopic or ListSchema to retrieve schema information.

3. Call ListShard to get shard information and manually assign shards to tasks.

4. Call OpenOffsetSession to start consumption and retrieve the stored offset.

5. Use the offset to call GetCursor and obtain the cursor.

6. Use the cursor to call GetRecords and fetch data.

7. If no data is returned, sleep and repeat step 6.

8. If data is returned, process the data and update the cursor, then repeat step 6.

9. Periodically call ListShard and CommitOffset to update shard information and checkpoints.

High-Level SDK consumption process using Consumer:

1. Initialize the Consumer, which automatically assigns shards to nodes and retrieves saved checkpoints.

2. Call Read to read data.

3. If no data is returned, the client waits and repeats step 2.

4. If data is returned, process it and repeat step 2.

Advantages of Consumer over the Low-Level SDK:

• Simpler logic and easier to use.

• Built-in retry mechanism with configurable retries for exceptions such as network jitters.

• Automatically adapts to scaling in/out and maintains consumption order.

• Dynamically allocates shards as nodes join or leave.

• Automatic checkpoint management without manual handling.

SDK availability

Language	Low-Level SDK	High-Level SDK
Java	Supported	Supported
C++	Supported	Supported
Python	Supported	Supported
Go	Supported	Not supported