Create an OSS sync task

Prerequisites

1. Prepare an OSS bucket. DataHub supports syncing data to OSS. Before you create a sync task, create a destination bucket in the OSS console. 2. Grant permissions to a service-linked role. When you configure a sync task, you can use a static cloud account with an AccessId/AccessKey or a temporary account through STS for data synchronization. If you select the STS option, the system automatically creates a DataHub service-linked role. DataHub uses this role to access OSS and sync data.

3. Synchronization details. 1) DataHub supports syncing TUPLE and BLOB data to OSS.

• TUPLE: Data is stored in CSV format. Columns within each record are separated by commas, and records are separated by a line break ('\N').

• BLOB: Data is appended to files. If you need to partition your data in OSS, you must insert data delimiters when writing data to DataHub.

2) File format: File names for synced objects in OSS are generated from information such as the ConnectorId. You cannot customize file names. 3) Subdirectory: DataHub creates a subdirectory based on the data's write time. The default time zone is Asia/Shanghai. For more options, create the task by using the SDK.

Create a sync task

1. Navigate to Project List > Project Details > Topic Details.

2. Click + Sync in the upper-right corner. In the New connector dialog box, configure the Endpoint, Bucket, Prefix, Time interval, Time format, Import fields, and Authentication mode parameters. Then, click Create.

The following list describes the console configuration parameters. For advanced options, use the SDK.

1. Endpoint: The OSS service endpoint. Use the domain name for the classic network. For internal networks, only HTTP is supported. HTTPS is not supported.

2. Import fields: Select specific columns from your DataHub topic to sync to OSS.

3. Prefix: The prefix for the destination directory in the OSS bucket.

4. Time format and Time interval: DataHub uses the data's write time to name the subdirectory under the specified prefix. The time interval defines the time range of data for each subdirectory. The value must be between 15 and 1,440 minutes (1 day).

Example

1. Create an OSS bucket. In the OSS console, create a destination bucket. For this example, the bucket name is datahub2oss, the access permission is set to Private, and the storage class is Standard storage.

2. Create a DataHub topic. In this example, create a TUPLE topic. The topic is named wurao_tuple_test_001 and has 5 shards. The schema includes three non-null fields: field1 (STRING), field2 (BIGINT), and field3 (DOUBLE).

3. Create the sync task. In the New connector dialog box, set Endpoint to http://oss-cn-hangzhou-internal.aliyuncs.com:80, Bucket to datahub2oss, and Prefix to datahub_pre. Set Time interval to 15 minutes. For Import fields, select field1, field2, and field3. For Authentication mode, select STS. Then, click Create.

4. Write TUPLE data to DataHub. The first record is f_1, 1, 1, and the second is f_2, 2, 2. These values correspond to the fields field1, field2, and field3, respectively.

5. Verify the synced data. File paths for synced objects in OSS follow the format prefix/subdirectory/filename. For example, a file path could be datahub_pre/202003102045/{ConnectorId}, where datahub_pre is the prefix and 202003102045 is the subdirectory named according to the time interval. Download the file to view its contents. The synced TUPLE data is in CSV format:

   $cat 3d828972-b9f2-4733-8c00-e46bc4c876f5_4-45400-45401-1583844969437
   f_1,"1","1.0"
   f_2,"2","2.0"