DocsICP FilingConsole
Documentation
Home Page DataHub User Guide CLI

CLI

Updated at:
Copy as MD
Product
My Favorites

Use the command-line tool to access your DataHub projects and run commands.

Prerequisites

  • Java 8 or later is installed.

Install and configure the client

  1. Download and extract the command-line tool package datahub_console.tar.gz.

  2. Extracting the package creates the bin, conf, and lib directories.

  3. Navigate to the conf directory and enter the ak and endpoint information in the datahub.properties file as follows:

    datahub.accessid=
datahub.accesskey=
datahub.endpoint=

    Parameter details:

    Parameter

    Required

    Description

    Example

    datahub.accessid

    Yes

    The AccessKey ID of your Alibaba Cloud account or RAM user.

    N/A

    datahub.accesskey

    Yes

    The AccessKey Secret for the specified AccessKey ID.

    N/A

    datahub.endpoint

    Yes

    The endpoint for the DataHub service.

    Set the endpoint according to the region and network type of your DataHub project. For a list of available endpoints, see DataHub Endpoints.

    https://dh-cn-hangzhou.aliyuncs.com

Run the client

Start the client using one of the following methods:

  • Option 1: In the bin directory of the client installation path, double-click the datahubcmd.bat file (for Windows) to start the client. The client starts successfully when the DataHub=> prompt is displayed.

  • Option 2: In your system's command-line window, navigate to the bin directory of the client installation path. Run the datahubcmd command (for Windows) or sh datahubcmd.sh (for Linux or macOS). The client starts successfully when the DataHub=> prompt is displayed.

Command help

To get help for commands, use one of the following methods.

  • Option 1: View help information from within the client.

    • View all commands:

      help

    • View related commands by keyword. For example, to get a list of topics:

      DataHub=>help lt
NAME
        lt - List topic

SYNOPSYS
        lt [-p] string

OPTIONS
        -p  string
                projectName
                [Mandatory]

  • Option 2: From your system's command-line window, navigate to the bin directory in the client installation path and run the following command to view help for all commands.

    ...\bin>datahubcmd help

Command reference

Project

  • Create a project

    cp -p test_project  -c test_comment

    • -p: The project name.

    • -c: A description for the project.

  • Delete a project

    Important

    Before deleting a project, you must delete all its resources, such as topics, subscriptions, and data synchronization tasks. Otherwise, the operation will fail.

    dp -p test_project

    • -p: The project name.

  • List projects

    lp

Topic

  • Create a topic

    ct -p test_project -t test_topic -m TUPLE -f [(name,string,true)] -s 3 -l 3 -c test_comment

    • -p: The project name.

    • -t: The topic name.

    • -m: The topic type. Valid values are BLOB or TUPLE.

    • -f: The schema for a tuple topic. The format is [(fieldName,fieldType,isNull)]. Separate multiple fields with commas.

    • -s: The number of shards.

    • -l: The data lifecycle, in days. The value must be between 1 and 7.

    • -c: A description for the topic.

  • Delete a topic

    dt -p test_project -t test_topic

    • -p: The project name.

    • -t: The topic name.

  • Get topic information

    gt -p test_project -t test_topic

    • -p: The project name.

    • -t: The topic name.

  • Export a topic schema to a JSON file

    gts -f filepath -p test_project -t test_topic

    • -f: The path where the JSON file is saved.

    • -p: The project name.

    • -t: The topic name.

  • List topics

    lt -p test_project

    • -p: The project name.

  • Create a topic from a JSON file

    rtt -s 3 -l 3 -c test_comment -f filepath -p test_project -t test_topic

    • -s: The number of shards.

    • -l: The data lifecycle in days. The valid range is 1 to 7.

    • -f: The file path.

    • -p: The project name.

    • -t: The topic name.

  • Update the topic lifecycle

    utl -p test_project -t test_topic -l 3 -c test_comment

    • -p: The project name.

    • -t: The topic name.

    • -l: The topic lifecycle.

    • -c: A description for the topic.

  • Add a topic field

    anf -p test_project -t test_topic -l [(f1,STRING)]

    • -p: The project name.

    • -t: The topic name.

    • -l: The new field to add. The format is [(fieldName,fieldType)].

Connector

  • Create an ODPS connector

    coc -p test_project -t test_topic -m SYSTEM_TIME -e odpsEndpoint -op odpsProject -ot odpsTable -oa odpsAccessId -ok odpsAccessKey -tr 60 -c (field1,field2) -tf ds hh mm

    • -p: The project name.

    • -t: The topic name.

    • -m: The synchronization type. For syncing to ODPS, the supported types are SYSTEM_TIME, USER_DEFINE, EVENT_TIME, and META_TIME.

    • -e: The ODPS endpoint. You must use a classic network endpoint.

    • -op: The ODPS project name.

    • -oa: The AccessKey ID for accessing ODPS.

    • -ok: The AccessKey Secret for accessing ODPS.

    • -tr: The partition time interval in minutes. The default value is 60.

    • -tf: The partition format. Use ds for daily, ds hh for hourly, and ds hh mm for minute-level partitions.

  • Add a field to an ODPS connector

    acf -p test_project -t test_topic -c connectorId -f fieldName

    • -p: The project name.

    • -t: The topic name.

    • -c: The connector ID. You can find this on the Data Synchronization tab.

    • -f: The name of the new field.

  • Create a MySQL/RDS connector

    cdc -p test_project -t test_topic -h host -po 3306 -ty mysql -d mysql_database -ta msyql_table -u username -pa password -ht IGNORE -n (field1,field2)

    • -p: The project name.

    • -t: The topic name.

    • -h: The host. You must use a classic network endpoint.

    • -po: The port.

    • -ty: The sink type. Valid values are:

      • SINK_MYSQL creates a connector to sync data to MySQL.

      • SINK_ADS creates a connector to sync data to ADS.

    • -d: The database name.

    • -ta: The table name.

    • -u: The username.

    • -pa: The password.

    • -ht: The insert mode. Valid values are:

      • IGNORE

      • OVERWRITE

    • -n: The fields to synchronize. Example: (field1,field2).

  • Create a DataHub connector

    cdhc -p test_project -t test_topic -sp sinkProject -st sinkTopic -m AK -i accessid k accessKey

    • -p: The project name.

    • -t: The topic name.

    • -sp: The sink project where data will be imported.

    • -st: The sink topic where data will be imported.

    • -m: The authentication type

    • AK: Authenticates using an AccessKey. Requires accessId and accessKey.

    • STS: Authenticates using Security Token Service (STS).

  • Create an FC connector

    cfc -p test_project -t test_topic -e endpoint -s service -f function -au AK -i accessId -k accessKey -n (field1,field2)

    • -p: The project name.

    • -t: The topic name.

    • -e: The Function Compute (FC) endpoint. You must use a classic network endpoint.

    • -s: The FC service name.

    • -f: The FC function name.

    • -au: The authentication method

    • AK: Authenticates using an AccessKey. Requires accessId and accessKey.

    • STS: Authenticates by using STS.

    • -n: The fields to synchronize. Example: (field1,field2).

  • Create a Hologres connector

    chc -p test_project -t test_topic -e endpoint -cl (field,field2) -au AK -hp holoProject -ht holoTopic -i accessId -k accessKey -m Delimiter -l 1 -b false -n (field1,field2)

    • -p: The project name.

    • -t: The topic name.

    • -e: The endpoint.

    • -cl: The fields to synchronize to Hologres.

    • -au: The authentication method. Only AK authentication is currently supported for Hologres connectors.

    • -m: The parsing type:

      • Delimiter: Requires you to specify the lineDelimiter, parseData, and columnDelimiter properties.

      • InformaticaJson: Requires you to specify the parseData property.

  • Create an OTS connector

    cotsc -p test_project -t test_topic -i accessId -k accessKey -it instanceId -m AK -t table -wm PUT -c (field1,field2)

    • -p: The project name.

    • -t: The topic name.

    • -it: The Tablestore (OTS) instance name.

    • -m: The authentication type. The default is STS.

    • AK: Authenticates by using an AccessKey. You must provide an accessId and accessKey.

    • STS: Authenticates by using STS.

    • -t: The OTS table name.

    • -wm: The write mode. Two modes are supported:

      • PUT

      • UPDATE

    • -c: The fields to synchronize. Example: (field1,field2).

  • Create an Elasticsearch connector

    cec -p test_project -t test_es -e endpoint:9200 -i f1 -u user -w password -n (f1,f2) -ty (f1,f2) -r (f1,f2) -c (f1,f2) -ti "1970-01-01 00:00:00"

    • -p: The project name.

    • -t: The topic name.

    • -e: The Elasticsearch endpoint. Must be an internal endpoint and port in the format internal_endpoint:internal_port.

    • -i: The index. For more information about indexes, see Create a connector to synchronize data to Elasticsearch.

    • -u: The username for accessing Elasticsearch.

    • -w: The password for accessing Elasticsearch.

    • -n: The ID property column.

    • -ty: The type property column.

    • -r: The router property column.

    • -c: The fields to import.

    • -ti: The synchronization start time, in yyyy-mm-dd hh:mm:ss format.

  • Create an OSS connector

    csc -p test_project -t test_topic -b bucket -e endpoint -pr ossPrefix -tf ossTimeFormat -tr timeRange -c (f1,f2)

    • -p: The project name.

    • -t: The topic name.

    • -b: The Object Storage Service (OSS) bucket name.

    • -e: The OSS endpoint name.

    • -pr: The directory prefix for data synchronized to OSS.

    • -tf: The synchronization time format. For example, %Y%m%d%H%M specifies minute-level partitions.

    • -tr: The time interval for partitions.

    • -c: The fields to synchronize.

  • Delete connectors

    dc -p test_project -t test_topic -c connectorId

    • -p: The project name.

    • -t: The topic name.

    • -c: The connector ID. You can find this on the Data Synchronization tab.

  • Get connector details

    gc -p test_project -t test_topic -c connectorId

    • -p: The project name.

    • -t: The topic name.

    • -c: The connector ID. You can find this on the Data Synchronization tab.

  • List connectors in a topic

    lc -p test_project -t test_topic

    • -p: The project name.

    • -t: The topic name.

  • Restart a connector

    rc -p test_project -t test_topic -c connectorId

    • -p: The project name.

    • -t: The topic name.

    • -c: The connector ID. You can find this on the Data Synchronization tab.

  • Update a connector's AccessKey

    uca -p test_project -t test_topic -c connectorId  -a accessId -k accessKey

    • -p: The project name.

    • -t: The topic name.

    • -c: The connector ID. You can find this on the Data Synchronization tab.

Shard

  • Merge shards

    ms -p test_project -t test_topic -s shardId -a adjacentShardId

    • -p: The project name.

    • -t: The topic name.

    • -s: The ID of the shard to merge.

    • -a: The ID of the adjacent shard to merge with.

  • Split a shard

    ss -p test_project -t test_topic -s shardId

    • -p: The project name.

    • -t: The topic name.

    • -s: The ID of the shard to split.

  • List shards in a topic

    ls -p test_project -t topicName

    • -p: The project name.

    • -t: The topic name.

  • Get shard synchronization status

    gcs -p test_project -t test_topic -s shardId -c connectorId

    • -p: The project name.

    • -t: The topic name.

    • -s: The shard ID.

    • -c: The connector ID. You can find this on the Data Synchronization tab.

  • Get the consumption offset for a shard

    gso -p test_project -t test_topic -s subid -i shardId

    • -p: The project name.

    • -t: The topic name.

    • -s: The subscription ID.

    • -i: The shard ID.

Subscription

  • Create a subscription

    css -p test_project -t test_topic -c comment

    • -p: The project name.

    • -t: The topic name.

    • -c: A description for the subscription.

  • Delete a subscription

    dsc -p test_project -t test_topic -s subId

    • -p: The project name.

    • -t: The topic name.

    • -s: The subscription ID.

  • List subscriptions

    lss -p test_project -t test_topic

    • -p: The project name.

    • -t: The topic name.

Upload and download data

  • Upload data

    uf -f filepath -p test_topic -t test_topic -m "," -n 1000

    • -f: The file path. Note: For Windows paths, you must escape backslashes. Example: D:\\test\\test.txt.

    • -p: The project name.

    • -t: The topic name.

    • -m: The text delimiter. Commas (,) and spaces are supported.

    • -n: The batch size for each upload. The default is 1000.

    Example: Upload a CSV file

    This example shows how to upload a CSV file to DataHub. The file has the following format:

    1. 0,qe614c760fuk8judu01tn5x055rpt1,true,100.1,14321111111
2. 1,znv1py74o8ynn87k66o32ao4x875wi,true,100.1,14321111111
3. 2,7nm0mtpgo1q0ubuljjjx9b000ybltl,true,100.1,14321111111
4. 3,10t0n6pvonnan16279w848ukko5f6l,true,100.1,14321111111
5. 4,0ub584kw88s6dczd0mta7itmta10jo,true,100.1,14321111111
6. 5,1ltfpf0jt7fhvf0oy4lo8m3z62c940,true,100.1,14321111111
7. 6,zpqsfxqy9379lmcehd7q8kftntrozb,true,100.1,14321111111
8. 7,ce1ga9aln346xcj761c3iytshyzuxg,true,100.1,14321111111
9. 8,k5j2id9a0ko90cykl40s6ojq6gruyi,true,100.1,14321111111
10. 9,ns2zcx9bdip5y0aqd1tdicf7bkdmsm,true,100.1,14321111111
11. 10,54rs9cm1xau2fk66pzyz62tf9tsse4,true,100.1,14321111111

    In the preceding CSV file, each line is a record, and fields are separated by commas (,). The file is saved at the local path /temp/test.csv. The DataHub topic has the following schema:

    Field name

    Type

    id

    BIGINT

    name

    STRING

    gender

    BOOLEAN

    salary

    DOUBLE

    my_time

    TIMESTAMP

    Run the following command:

    uf -f /temp/test.csv -p test_topic -t test_topic -m "," -n 1000

  • Download data

    down -p test_project -t test_topic -s shardId -d subId -f filePath -ti "1970-01-01 00:00:00" -l 100 -g 0

    • -f: The path to save the downloaded file. For Windows paths, you must escape backslashes (for example, D:\\test\\test.txt).

    • -p: The project name.

    • -t: The topic name.

    • -s: The shard ID.

    • -d: The subscription ID.

    • -ti: The timestamp to start reading data from. The client will read records created at or after this time. The format is yyyy-mm-dd hh:mm:ss.

    • -l: The number of records to read in each batch.

    • -g: Specifies whether to read continuously.

      • 0: Reads only one batch of records and then stops.

      • 1: Reads continuously.

FAQ

  • Script fails to start on Windows: This can occur if the script path contains parentheses. To resolve this, ensure the full path to the script does not contain any parentheses.

Previous：DataHub-DataWorks Connection service-linked roleNext：Set up CloudMonitor alerts