API Reference

DataHub API specifications, signature calculations, and available API operations for projects, topics, shards, connectors, and subscriptions.

API reference

I. API specifications

1. Common request headers

Request header	Description
x-datahub-client-version	API version.
x-datahub-security-token	The security token, if any.
Date	Request time in standard GMT format: EEE, dd MMM yyyy HH:mm:ss z.
Authorization	Request signature.
Content-Type	Request serialization type.

2. Common response headers

Response header	Description
Content-Type	Serialization type of the response.
Content-Length	Message body size.
x-datahub-request-id	Globally unique request ID (GUID).

3. Error codes

Error code	Description	Remarks
InvalidParameter	One or more parameters are invalid.
InvalidCursor	The cursor is invalid.
NoSuchXXX	The requested resource does not exist.
XXXAlreadyExist	The requested resource already exists.
Unauthorized	Authentication failed.	Invalid AccessKey pair or request timestamp mismatch.
NoPermission	You are not authorized to perform this operation.	The RAM user lacks the required permissions.
OperationDenied	The operation is forbidden.	The operation is forbidden. For example, you cannot delete a project that contains topics.
LimitExceeded	Traffic exceeds defined limits.	Server limits exceeded, such as QPS or traffic limits.
InvalidShardOperation	A shard split or merge operation failed.
MalformedRecord	The data format is invalid.
OffsetReseted	The consumer offset was reset.
OffsetSessionChanged	The subscription ID is occupied by another user.
SubscriptionOffline	The subscription was canceled.
InternalServerError	An internal error occurred.
Other error codes	Other non-retryable error codes, which may be removed in the future.

4. Unified error response format

Parameter	Description
ErrorCode	Returned error code.
ErrorMessage	Returned error message.

Sample error response:

    HTTP/1.1 403
    x-datahub-request-id: 2018050817492199d6650a00000039
    Content-Type: application/json
    Content-Length: xxx

    {
        "ErrorCode": "Unauthorized",
        "ErrorMessage": "Authroize failed"
    }

5. Limits

Parameter	Description
ProjectName	The project name must be 3 to 32 characters in length and can contain only letters, digits, and underscores (_). It must start with a letter. The value is not case-sensitive.
TopicName	The topic name must be 3 to 128 characters in length and can contain only letters, digits, and underscores (_). It must start with a letter. The value is not case-sensitive.

II. Signature calculations for the Authorization header

Authorization = "DATAHUB " + AccessId + ":" + Signature
Signature = base64(hmac-sha1(AccessKey,
            HTTPMethod + "\n"
            + Content-Type + "\n"
            + Date + "\n"
            + CanonicalizedDataHubHeaders + "\n"
            + CanonicalizedResource))

AccessKey: AccessKey secret for signature calculation.
HTTPMethod: HTTP method, such as PUT, GET, POST, HEAD, or DELETE.
\n: the line feed.
Content-Type: request serialization type. Typically application/json.
Date: request time in GMT. Example: Sun, 22 Nov 2015 08:16:38 GMT.
CanonicalizedDataHubHeaders: HTTP request headers prefixed with x-datahub-, sorted alphabetically.
CanonicalizedResource: URL of the requested DataHub resource. Sort parameters alphabetically if present.

Construction of the CanonicalizedDataHubHeaders string

All HTTP request headers prefixed with x-datahub- are called canonicalized DataHub headers. To construct the CanonicalizedDataHubHeaders string:

Convert all x-datahub- header names to lowercase. Example: X-DATAHUB-Client-Version:1.1 becomes x-datahub-client-version:1.1.
For STS-based access, add the token as x-datahub-security-token:token.
Sort all headers alphabetically.
Remove spaces around the colon (:) between header names and values. Example: x-datahub-client-versionn : 1.1 becomes x-datahub-client-version:1.1.
Join each header-value pair with \n to form the CanonicalizedDataHubHeaders string.

Construction of the CanonicalizedResource string

The DataHub resources specified in the request are called canonicalized resources. To construct the CanonicalizedResource string:

Set the CanonicalizedResource string to an empty string.
Enter the URL of the DataHub resource that you want to access, such as /projects/<ProjectName>/topics/<TopicName>.
If the URL contains one or more parameters, arrange them in alphabetical order and separate them with ampersands (&). Add a question mark (?) and the parameters at the end of the string. Example: /projects/<ProjectName>/topics/<TopicName>/connectors/sink_odps?donetime

Calculation of the signature string

Encode the signature string in UTF-8, then use it with the AccessKey secret to calculate the signature.
Calculate the HMAC-SHA1 signature per RFC 2104, using the AccessKey secret as the key.
Headers prefixed with x-datahub- must meet these requirements:
Header names must be lowercase.
Headers are sorted alphabetically.
No spaces around the colon (:) between header name and value.
Each header is followed by a line feed (\n). If no headers exist, use an empty string.

Examples

Request

Calculation formula of the signature string

Signature string

POST /projects/test_project/topics/test_topic HTTP/1.1 Host: https://dh-cn-hangzhou.aliyuncs.com

User-Agent: customer x-datahub-client-version: 1.1 Content-Type: application/json Date: Thu, 10 Jan 2019 07:28:29 GMT

Signature = base64(hmac-sha1(AccessKey,HTTPMethod + "\n" + Content-Type + "\n" + Date + "\n" + CanonicalizedDataHubHeaders+ CanonicalizedResource))

POST\napplication/json\nThu, 10 Jan 2019 07:28:29 GMT\nx-datahub-client-version:1.1\n/projects/test_project/topics/test_topic

Calculate the signature in Python:

import base64
import hmac
import sha
h = hmac.new("****your accessKey*****",
"POST\napplication/json\nThu, 10 Jan 2019 07:28:29 GMT\nx-datahub-client-version:1.1\n/projects/test_project/topics/test_topic", sha)
Signature = base64.b64encode(h.digest())
print("Signature: %s" % Signature)

Sample request headers:

The value of the Authorization header is in the DATAHUB AccessId:Signature format.

POST /projects/test_project/topics/test_topic HTTP/1.1
Authorization: DATAHUB AccessId:Signature
Content-Type: application/json
Date: Thu, 10 Jan 2019 07:28:29 GMT
Host: http://dh-cn-hangzhou.aliyuncs.com
User-Agent: customer
x-datahub-client-version: 1.1

Calculate the signature in Java 8:

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.util.Base64;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import java.util.SortedMap;
import java.util.TreeMap;

public abstract class Authorization {
    private static final String DEFAULT_ENCODING = "UTF-8";
    private static final String DEFAULT_HASH = "HmacSHA1";
    private static final String X_DATAHUB_PREFIX = "x-datahub";
    private static final String HEADER_CONTENT_TYPE = "Content-Type";
    private static final String HEADER_DATE = "Date";

    static public String getAkAuthorization(Request request) {
        String canonicalURL = request.getUrlPath();
        String canonicalQueryString = request.getQueryStrings();
        String canonicalHeaderString = getCanonicalHeaders(
                getSortedHeadersToSign(request.getHeaders()));

        String canonicalRequest = request.getMethod().toUpperCase() + "\n" +
                canonicalHeaderString + "\n" + canonicalURL;
        if (canonicalQueryString != null && !canonicalQueryString.isEmpty()) {
            canonicalRequest += "?" + canonicalQueryString;
        }

        String signature = HMAC1Sign(request.getAccessKey(), canonicalRequest);

        return "DATAHUB " + request.getAccessId() + ":" + signature;
    }

    static private String HMAC1Sign(String accessKey, String canonicalRequest) {
        try {
            SecretKeySpec signingKey = new SecretKeySpec(accessKey.getBytes(), DEFAULT_HASH);
            Mac mac = Mac.getInstance(DEFAULT_HASH);
            mac.init(signingKey);
            return Base64.getEncoder().encodeToString(
                    mac.doFinal(canonicalRequest.getBytes(DEFAULT_ENCODING))
            ).trim();
        } catch (Exception e) {
            throw new RuntimeException(e.getMessage(), e);
        }
    }

    static private String getCanonicalHeaders(Map<String, String> headers) {
        StringBuilder sb = new StringBuilder();
        Iterator<Map.Entry<String, String>> pairs = headers.entrySet().iterator();
        while (pairs.hasNext()) {
            Map.Entry<String, String> pair = pairs.next();
            if (pair.getKey().startsWith(X_DATAHUB_PREFIX)) {
                sb.append(pair.getKey());
                sb.append(":");
                sb.append(pair.getValue());
            } else {
                sb.append(pair.getValue());
            }

            if (pairs.hasNext()) {
                sb.append("\n");
            }
        }
        return sb.toString();
    }

    static private SortedMap<String, String> getSortedHeadersToSign(Map<String, String> headers) {
        SortedMap<String, String> sortedHeaders = new TreeMap<>();

        for (Map.Entry<String, String> entry : headers.entrySet()) {
            String lowerKey = entry.getKey().toLowerCase();
            if (lowerKey.equalsIgnoreCase(HEADER_CONTENT_TYPE) ||
                    lowerKey.equalsIgnoreCase(HEADER_DATE) ||
                    lowerKey.startsWith(X_DATAHUB_PREFIX)) {
                if (!entry.getValue().isEmpty()) {
                    sortedHeaders.put(lowerKey, entry.getValue());
                }
            }
        }

        if (!sortedHeaders.containsKey(HEADER_CONTENT_TYPE.toLowerCase())) {
            sortedHeaders.put(HEADER_CONTENT_TYPE.toLowerCase(), "");
        }

        return sortedHeaders;
    }

    public static class Request {
        private String accessId;
        private String accessKey;
        private String urlPath;
        private String method;
        private Map<String, String> headers;
        private String queryStrings;

        public String getAccessId() {
            return accessId;
        }

        public Request setAccessId(String accessId) {
            this.accessId = accessId;
            return this;
        }

        public String getAccessKey() {
            return accessKey;
        }

        public Request setAccessKey(String accessKey) {
            this.accessKey = accessKey;
            return this;
        }

        public String getUrlPath() {
            return urlPath;
        }

        public Request setUrlPath(String urlPath) {
            this.urlPath = urlPath;
            return this;
        }

        public String getMethod() {
            return method;
        }

        public Request setMethod(String method) {
            this.method = method;
            return this;
        }

        public Map<String, String> getHeaders() {
            return headers;
        }

        public Request setHeaders(Map<String, String> headers) {
            this.headers = headers;
            return this;
        }

        public String getQueryStrings() {
            return queryStrings;
        }

        public Request setQueryStrings(String queryStrings) {
            this.queryStrings = queryStrings;
            return this;
        }
    }

    public static void main(String[] args) {
        Map<String, String> headers = new HashMap<>();
        headers.put("Date", "Thu, 10 Jan 2019 07:28:29 GMT");
        headers.put("x-datahub-client-version", "1.1");
        headers.put("Content-type", "application/json");

        String accessId = "testKeyID";
        String accessKey = "testKeySecret";
        String method = "POST";
        String path = "/projects/test_project/topics/test_topic";
        String canonicalQueryString = ""; // Parameters are arranged in alphabetical order and separated by ampersands (&). Example: a=x&b=y.

        Authorization.Request authRequest = new Authorization.Request()
                .setAccessId(accessId)
                .setAccessKey(accessKey)
                .setMethod(method.toUpperCase())
                .setUrlPath(path)
                .setHeaders(headers)
                .setQueryStrings(canonicalQueryString);
        System.out.println(Authorization.getAkAuthorization(authRequest));
    }
}

III. API operations

Create a project

Request

Request syntax

  POST /projects/<ProjectName> HTTP/1.1

Request parameters

Parameter	Type	Description
Comment	String	Project description. Maximum: 1,024 bytes.

Response

Response syntax

  HTTP/1.1 201 Created

Examples

Sample requests

  POST /projects/<ProjectName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: application/json
  Content-Length: xxx

  {
      "Comment": "test project"
  }

Sample success responses

  HTTP/1.1 201 Created
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Length: 0

Query the information about a project

Request

Request syntax

  GET /projects/<ProjectName> HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
CreateTime	long	Project creation time. Unit: seconds.
LastModifyTime	long	Last update time. Unit: seconds.
Comment	String	Project description.

Examples

Sample requests

  GET /projects/<ProjectName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Content-Length: xxx

  {
      "Comment": "test project",
      "CreateTime": 1525763481,
      "LastModifyTime": 1525763481
  }

Query the information about projects

Request

Request syntax

  GET /projects HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
ProjectNames	List	The names of the projects.

Examples

Sample requests

  GET /projects HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
      "ProjectNames": [
          "project1",
          "projcet2"
      ]
  }

Update a project

Request

Request syntax

  PUT /projects/<ProjectName> HTTP/1.1

Request parameters

Parameter	Type	Description
Comment	String	Project description.

Response

Response syntax

  HTTP/1.1 200 OK

Examples

Sample requests

  PUT /projects/<ProjectName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: application/json
  Content-Length: xxx

  {
      "Comment": "update comment"
  }

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Length: 0

Delete a project

Request

Request syntax

  DELETE /projects/<ProjectName> HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Examples

Sample requests

  DELETE /projects/<ProjectName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Length: 0

Create a topic

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName> HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	Operation type.
ShardCount	int	The initial number of shards.
Lifecycle	int	The time to live (TTL) of data.
RecordType	String	Record type. Valid values: BLOB (unstructured data) and TUPLE (structured data).
RecordSchema	String	Record schema. Required when RecordType is TUPLE. Not required for BLOB.
Comment	String	Topic description.
ExpandMode	String	Extension mode. Set to extend to enable; otherwise omit.

Response

Response syntax

  HTTP/1.1 201 Created

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: application/json
  Content-Length: xxx

  {
      "Action": "create",
      "ShardCount": 1,
      "Lifecycle": 1,
      "RecordType": "TUPLE",
      "RecordSchema": "{\"fields\":[{\"name\":\"field1\",\"type\":\"STRING\"},{\"name\":\"field2\",\"type\":\"BIGINT\"}]}}",
      "Comment": "create topic", 
      "ExpandMode": "extend"
  }

Sample success responses

  HTTP/1.1 201 Created
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Content-Length: 0

Query the information about a topic

Request

Request syntax

  GET /projects/<ProjectName>/topics/<TopicName> HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
ShardCount	int	The initial number of shards.
Lifecycle	int	The TTL of data.
RecordType	String	Record type. Valid values: BLOB (unstructured data) and TUPLE (structured data).
RecordSchema	String	Record schema. Required when RecordType is TUPLE. Not required for BLOB.
Comment	String	Topic description.
CreateTime	long	The time when the topic was created.
LastModifyTime	long	The time when the topic was last updated.

Examples

Sample requests

  GET /projects/<ProjectName>/topics/<TopicName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Sample success responses

  HTTP/1.1 201 Created
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Content-Length: xxx

  {
      "ShardCount": 1,
      "Lifecycle": 1,
      "RecordType": "TUPLE",
      "RecordSchema": "{\"fields\":[{\"name\":\"field1\",\"type\":\"STRING\"},{\"name\":\"field2\",\"type\":\"BIGINT\"}]}",
      "Comment": "create topic",
      "CreateTime": 1525763481,
      "LastModifyTime": 1525763481
  }

Query the information about topics

Request

Request syntax

  GET /projects/<ProjectName>/topics HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
TopicNames	List	The names of the topics.

Examples

Sample requests

  GET /projects/<ProjectNames>/topics HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
      "TopicNames": [
          "topic1",
          "topic2"
      ]
  }

Update a topic

Request

Request syntax

  PUT /projects/<ProjectName>/topics/<TopicName> HTTP/1.1

Request parameters

Parameter	Type	Description
Comment	String	Topic description.

Response

Response syntax

  HTTP/1.1 200 OK

Examples

Sample requests

  PUT /projects/<ProjectName>/topics/<TopicName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: application/json
  Content-Length: xxx
  
  {
          "Comment": "update comment"
  }

Delete a topic

Request

Request syntax

  DELETE /projects/<ProjectName>/topics/<TopicName> HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Examples

Sample requests

  DELETE /projects/<ProjectName>/topics/<TopicName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Length: 0

Query the information about shards

Request

Request syntax

  GET /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
ShardId	String	The shard ID.
State	String	Shard state. Examples: OPENING, ACTIVE, CLOSED.
BeginHashKey	String	The start hash key of the hash key range.
EndHashKey	String	The end hash key of the hash key range.
ParentShardIds	List	Parent shard IDs.

Examples

Sample requests

  GET /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
      "Shards": [
              {
                      "ShardId": "0",
                      "State": "ACTIVE",
                      "BeginHashKey":"00000000000000000000000000000000",
                      "EndHashKey":"FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF",
                      "ParentShardIds:[]
              }
      ]
  }

Split a shard

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	Operation type. Set to split.
ShardId	String	The ID of the shard that you want to split.
SplitKey	String	Split key. Formula: SplitKey = BeginHashKey + (EndHashKey - BeginHashKey)/2.

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
NewShards	List	New shard details.

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: application/json
  Content-Length: xxx
  
  {
          "Action": "split",
          "ShardId": "0",
          "SplitKye": "7FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF"
  }

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
          "NewShards": [
                  {
                          "ShardId": "1",
                          "BeginHashKey":"00000000000000000000000000000000",
                          "EndHashKey":"7FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF"
                  },
                  {
                          "ShardId":"0",
                          "BeginHashKey":"7FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF",
                          "EndHashKey":"FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF"
                  }
          ]
  }

Merge shards

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	Operation type. Set to merge.
ShardId	String	The ID of the shard that you want to merge.
AdjacentShardId	String	ID of the adjacent shard to merge.

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
ShardId	String	The ID of the new shard.
BeginHashKey	String	The start hash key of the hash key range.
EndHashKey	String	The end hash key of the hash key range.

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: application/json
  Content-Length: xxx
  
  {
          "Action": "merge",
          "ShardId": "0",
          "AdjacentShardId": "1"
  }

Sample success responses

  HTTP/1.1 200 OK
        x-datahub-request-id: 2018050817492199d6650a00000039
        Content-Type: applicaton/json
        Conent-Length: xxx
        
        {
                "ShardId":"2",
                "BeginHashKey":"00000000000000000000000000000000",
                "EndHashKey":"FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF"
        }

Query a cursor

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/shards/<ShardId> HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	Operation type. Set to cursor.
Type	String	Cursor query method. Valid values: OLDEST, LATEST, SYSTEM_TIME, and SEQUENCE.
SystemTime	Int64	Cursor system time. Required when Type is SYSTEM_TIME. Unit: milliseconds.
Sequence	Int64	Cursor sequence number. Required when Type is SEQUENCE.

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
Cursor	String	Cursor value.
RecordTime	Int64	Time when the data was written to DataHub. Unit: milliseconds.
Sequence	Int64	Sequence number of the data. Unique within a shard.

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/shards/<ShardId> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: application/json
  Content-Length: xxx
  
  {
          "Action": "cursor",
          "Type": "SEQUENCE",
          "Sequence": 1
  }

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
          "Cursor": "30005af19b3800000000000000000000",
          "RecordTime": 1525783352873,
          "Sequence": 1
  }

Write data without specifying shards

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	Operation type. Set to pub.
ShardId	String	The shard ID.
Attributes	Map<String, String>	The user attributes.
Data		Data to write. For BLOB: a Base64-encoded string. For TUPLE: an array of strings.

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
FailedRecordCount	Int	Number of failed records.
FailedRecords	Array	Details of failed records.

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
          "Action": "pub",
          "Records": [
                  {
                          "ShardId": "0",
                          "Attributes": {
                                  "attr1": "value1",
                                  "attr2": "value2"
                          },
                          "Data": ["A","B","3","4"]
                  }
          ]
  }

  // BLOB
  {
          "Action": "pub",
          "Record": [
                  {
                          "ShardId": "0",
                          "Attributes": {
                                  "attr1": "value1",
                                  "attr2": "value2"
                          },
                          "Data": "Base64String"
                  }
          ]
  }

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
      "FailedRecordCount": 1,
      "FailedRecords": [
                      {
                              "Index": 0,
                              "ErrorCode": "errorCode",
                              "ErrorMessage": "errormsg"
                      }
      ]
  }

Read data

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/shards/<ShardId> HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	Operation type. Set to sub.
Cursor	String	The start cursor that is used to read records.
Limit	Int	The number of records to read at a time.

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
NextCursor	String	Cursor for reading the next record.
SystemTime	Int64	Time when the record was written to DataHub. Unit: milliseconds.
Cursor	String	Cursor corresponding to the record.
Sequence	Int64	Sequence number of the record in DataHub.
Attributes	Map	The user attributes.
Data		Requested data.

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "sub",
          "Cursor": "30005af19b3800000000000000000000",
          "Limit": 1
  }

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
      "NextCursor": "30005af19b3800000000000000090001",
      "Records": [
                      {
                              "Cursor": "30005af19b3800000000000000000000",
                              "SystemTime": 1525783352873,
                              "Sequence": 1,
                              "Attributes": {
                                      "key1": "value1",
                                      "key2": "value2"
                              },
                              "Data": ["AAA", "100"]
                      }
      ]
  }

Add a field

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName> HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	Operation type. Set to appendfield.
FieldName	String	The field name.
FieldType	String	The data type of the field. Example: STRING and BIGINT.

Response

Response syntax

  HTTP/1.1 200 OK

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "appendfield",
          "FieldName": "field1",
          "FieldType": "BIGINT"
  }

Create a connector

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1

Request parameters

Parameter	Type	Description
Type	String	The connector type. Example: SINK_ODPS.
ColumnFields	Array	The fields that you want to synchronize.
Config	Map	Connector configuration.

Response

Response syntax

  HTTP/1.1 201 Created

Examples

Sample request for creating a connector of the SINK_ODPS type

  POST /projects/<ProjectName>/topics/<TopicName>/connectors/sink_odps HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Type": "SINK_ODPS",
          "ColumnFields": ["field1", "field2"],
          "Config": {
                  "Project": "odpsProject",
                  "Topic": "odpsTopic",
                  "OdpsEndpoint": "xxx",
                  "TunnelEndpoint": "xxx",
                  "AccessId": "xxx",
                  "AccessKey": "xxx",
                  "PartitionMode": "SYSTEM_TIME",
                  "TimeRange": 60,
                  "PartitionConfig": {
                          "pt": "%Y%m%d",
                          "ct": "%H%M"
                  }
          }
  }

Query the information about a connector

Request

Request syntax

  GET /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Query the information about connectors

Request

Request syntax

  GET /projects/<ProjectName>/topics/<TopicName>/connectors HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Examples

Sample requests

  GET /projects/<ProjectName>/topics/<TopicName>/connectors HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
          "Connectors": [
                  "sink_odps", "sink_ads"
          ]
  }

Delete a connector

Request

Request syntax

  DELETE /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Reload a connector

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	Operation type. Set to reload.
ShardId	String	Shard ID to reload. If empty, reloads all shards for the connector.

Response

Response syntax

  HTTP/1.1 200 OK

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "reload"
  }

Query the shard state for a connector

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	Operation type. Set to status.
ShardId	String	The shard ID.

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
CurrentSequence	Int64	The sequence number of the consumer offset.
State	Enum	The shard state.
LastErrorMessage	String	Error message if the shard state is not CONTEXT_EXECUTING.
DiscardCount	Int64	Total records dropped since the connector started.

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "status",
          "ShardId": "0"
  }

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
          "State": "CONTEXT_EXECUTING",
          "CurrentSequence": 10,
          "DiscardCount": 0,
          "LastErrorMessage": ""
  }

Add a field for a connector

Request

Request syntax

POST /projects//topics//connectors/ HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	Operation type. Set to appendfield.
FieldName	String	The field name.

Response

Response syntax

  HTTP/1.1 200 OK

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "appendfiled",
          "FieldName": "field1"
  }

Create a subscription

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/subscriptions HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	Operation type. Set to create.
Comment	String	Subscription description.

Response

Response syntax

  HTTP/1.1 201 Created

Response parameters

Parameter	Type	Description
SubId	String	The subscription ID.

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/subscriptions HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "create",
          "Comment": "xxxx"
  }

Sample success responses

  HTTP/1.1 200 OK
        x-datahub-request-id: 2018050817492199d6650a00000039
         Content-Type: applicaton/json
  Conent-Length: xxx

        {
          "SubId": "1542078393028fzsZx"
  }

Query the information about a subscription

Request

Request syntax

  GET /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId> HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
SubId	String	The subscription ID.
State	Int	Subscription state. Valid values: 0 (online) and 1 (offline).
Comment	String	Subscription description.

Examples

Sample requests

  GET /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "SubId": "xxxx",
          "Comment": "xxxx"
          "State": 1
  }

Query the information about subscriptions

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/subscriptions HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	Operation type. Set to list.
PageIndex	Int	The page number.
PageSize	Int	The number of entries per page.

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
TotalCount	Int	The total number of entries returned.

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/subscriptions HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "list",
          "PageIndex": 1,
          "PageSize": 10
  }

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Subscriptions": [
                  {
                           "Comment": "xxxx",
                            "State": 1,
                            "SubId": "1542079169844gH8HM"
                  },
                  {
                          "Comment": "xxxx",
                          "State": 1,
                          "SubId": "1542078393028fzsZx"
                  }
          ],
          "TotalCount": 2
  }

Delete a subscription

Request

Request syntax

  DELETE /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId> HTTP/1.1

Examples

Sample requests

  DELETE /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Update the subscription status

Request

Request syntax

  PUT /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId> HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Examples

Sample requests

  PUT /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "State": 0
  }

Create a session for a consumer offset

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId>/offsets HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	Operation type. Set to open.
ShardIds	Array	The IDs of the shards.

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
Timestamp	Int64	Consumer offset timestamp. Unit: milliseconds.
Sequence	Int64	The sequence number of the consumer offset.
Version	Int64	The session version.
SessionId	String	The session ID.

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId>/offsets HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "open",
          "ShardIds": ["0"]
  }

Sample success responses

  HTTP/1.1 200 OK
        x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Offsets": {
                  "0": {
                          "Timestamp": 1000,
                          "Sequence": 1,
                          "Version": 1,
                          "SessionId": "xxx"
                  }
          }
  }

Query the information about a consumer offset

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId>/offsets HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	Operation type. Set to get.
ShardIds	Array	The IDs of the shards.

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
Timestamp	Int64	Consumer offset timestamp. Unit: milliseconds.
Sequence	Int64	The sequence number of the consumer offset.
Version	Int64	The session version.
SessionId	String	The session ID.

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId>/offsets HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "get",
          "ShardIds": ["0"]
  }

Sample success responses

  HTTP/1.1 200 OK
        x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Offsets": {
                  "0": {
                          "Timestamp": 1000,
                          "Sequence": 1,
                          "Version": 1,
                          "SessionId": "xxx"
                  }
          }
  }

Commit a consumer offset

Request

Request syntax

  PUT /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId>/offsets HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	Operation type. Set to commit.
Timestamp	Int64	Consumer offset timestamp. Unit: milliseconds.
Sequence	Int64	The sequence number of the consumer offset.
Version	Int64	The session version.
SessionId	Int64	The session ID.

Response

Response syntax

  HTTP/1.1 200 OK

Examples

Sample requests

  PUT /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId>/offsets HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "commit",
          "Offsets": {
                  "0": {
                          "Timestamp": 1000,
                          "Sequence": 1,
                          "Version": 1,
                          "SessionId": 1
                  }
          }
  }