Use DataHub to migrate log data to MaxCompute-MaxCompute(MaxCompute)-阿里云帮助中心

This topic describes how to use DataHub to migrate log data to MaxCompute.

Prerequisites

The following permissions are granted to the account authorized to access MaxCompute:

CreateInstance permission on MaxCompute projects
Permissions to view, modify, and update MaxCompute tables

For more information, see MaxCompute permissions.

Background information

DataHub is a platform that is designed to process streaming data. After data is uploaded to DataHub, the data is stored in a table for real-time processing. DataHub executes scheduled tasks within five minutes to synchronize the data to a MaxCompute table for offline computing.

To periodically archive streaming data in DataHub to MaxCompute, you need only to create and configure a DataHub connector.

Data type mappings

MaxCompute	DataHub
BIGINT	BIGINT
STRING	STRING
BOOLEAN	BOOLEAN
DOUBLE	DOUBLE
DATETIME	TIMESTAMP
DECIMAL	DECIMAL
TINYINT	TINIINT
SMALLINT	SMALLINT
INT	INTEGER
FLOAT	FLOAT
BLOB	STRING
MAP	Not supported
ARRAY	Not supported

Procedure

On the odpscmd client, create a table that is used to store the data synchronized from DataHub. For example, you can execute the following statement to create a table:
```
 CREATE TABLE test(f1 string, f2 string, f3 double) partitioned by (ds string);
```
Create a project in the DataHub console.
1. Log on to the DataHub console. In the upper-left corner, select a region.
2. In the left-side navigation pane, click Projects.
3. On the Projects page, click Create Project in the upper-right corner.
4. On the Create Project page, enter a Name and Description, and then click Create.
Create a topic.
1. On the Projects page, find your project and click View in the Actions column.
2. On the project details page, click Create Topic in the upper-right corner.
3. In the Create Topic panel, select Import MaxCompute Tables for Creation Type and configure other parameters.
  
  Configure the Project Name, Table Name, AccessKey ID, and AccessKey Secret parameters, and then click Next Step.
4. Click Next to complete the topic information.
  Note
  
  The schema of the DataHub topic must exactly match the schema of the target MaxCompute table, including field names, data types, and order. The connector creation fails if any of these conditions are not met.
  
  You can synchronize DataHub topics of the TUPLE and BLOB types to MaxCompute tables.
  
  By default, you can create up to 20 topics. To increase this limit, submit a ticket.
  
  The owner of a DataHub topic or the Creator account has the permissions to manage a DataConnector. For example, you can create or delete a DataConnector.
Write data to the newly created topic.
1. On the Topics tab for your project, find the new topic and click View in the Actions column.
2. On the topic details page, click Connector in the upper-right corner.
3. On the Create Connector page, click MaxCompute, configure the required parameters, and then click Create.
View DataConnector details.
1. In the left-side navigation pane, click Projects.
2. On the Projects page, find your project and click View in the Actions column.
3. In the Topics section, find your topic and click View in the Actions column.
4. On the topic details page, click the Connector tab to see the connector you created.
5. Find the newly created DataConnector and click View to view DataConnector details.
  
  By default, DataHub delivers data to a MaxCompute offline table every 5 minutes or when the data size reaches 60 MB. Sync Offset shows the number of records synchronized. The connector details page displays information such as Status (for example, EXECUTING), Sync Latency (in seconds), Endpoint, Project, Table, and partitioning mode.
Execute the following statement to check whether the log data is migrated to MaxCompute:
```
SELECT * FROM test;
```
A result similar to the following indicates that the data was delivered successfully. After you run the query, the result table contains three columns: f1, f2, and f3. The three newly inserted records are, for example, f1='60', f2='Samsung-11', f3=909.0, f1='61', f2='Samsung-12', f3=606.0, and f1='62', f2='Samsung-13', f3=2000.0.