Migrate log data to MaxCompute with Simple Log Service
You can use the data shipping feature of Simple Log Service to migrate log data to MaxCompute for subsequent processing.
Introduction
Simple Log Service allows you to ship real-time ingested log data to MaxCompute. After you enable this feature, Simple Log Service periodically ships log data from a Logstore to the corresponding MaxCompute table for further processing.
Prerequisites
An Alibaba Cloud account is used for this operation.
Procedure
-
In the MaxCompute client (odpscmd), run the following statement to create a table named
loghubto receive data from Simple Log Service.CREATE TABLE loghub ( id BIGINT, name STRING, salenum BIGINT ) PARTITIONED BY (dt string); -
Create a
projectand aLogstorein Simple Log Service.-
Log in to the Log Service console.
-
In the Projects area, click Create Project, and configure the parameters in the Create Project dialog box.
For more information about the parameters, see Manage a project.
-
In the Projects section, click the target Project to open the log management page.
-
On the Log Management page, click the
icon next to Logstores on the left and configure the parameters in the Create Logstore dialog box that appears.For more information about the parameters, see Create a Logstore.
-
Click the name of the Logstore to view its log data.
If you are prompted that indexing is not enabled, enable indexing in the upper-right corner of the page.
-
-
On the Log Management page, click the
icon to the left of the target Logstores, and select . -
Click the
icon next to MaxCompute, and configure the parameters in the Data Shipping to MaxCompute dialog box that appears.The following table describes the parameters that you need to configure.
Parameter
Description
Task Name
A custom name for the data shipping job to facilitate management.
Project Name
The MaxCompute project that contains the destination table. The current project is selected by default. You can select another project from the drop-down list.
MaxCompute Table Name
-
The name of the MaxCompute table. Enter a name for a new table or select an existing table.
-
The MaxCompute table must contain at least one data column and at least one partition key column.
MaxCompute Common Column
-
On the left, enter the log field names that you want to map to the MaxCompute data columns. On the right, enter or select the corresponding MaxCompute column names and types.
-
Each log field can be mapped to only one MaxCompute column, either a data column or a partition key column. Field redundancy is not supported. If a log field is mapped twice, the second mapping receives a NULL value. If a NULL value appears in a partition key column, data shipping for the record fails.
MaxCompute Partition Key Column
On the left, enter the log field names that you want to map to the MaxCompute partition key columns. On the right, enter or select the corresponding MaxCompute column names and types.
Important-
Data shipping to MaxCompute is a batch operation. Configure partition key columns and their types carefully. Ensure that the number of partitions processed in a single synchronization task is less than 512. The value of a partition key field cannot be empty or contain MaxCompute reserved characters such as forward slashes (/).
-
A single MaxCompute table supports a maximum of 60,000 partitions. After the limit is reached, no more data can be written to the table. To help you manage this limit, Simple Log Service supports up to three partition key columns when you ship data to a MaxCompute table. Select custom fields as partition key columns with caution and ensure that their values are enumerable.
Time Partition Format
The output format of the date for
__partition_time__. Example:yyyy-MM-dd.Note-
Do not ship data from different Logstores to the same MaxCompute table. Otherwise, partition conflicts or data loss may occur.
-
In international regions, you cannot ship data from LogHub to MaxCompute. You can use DataWorks to ship data. For more information, see Migrate log data to MaxCompute by using DataWorks Data Integration.
-
For more information about the new version of the data shipping to MaxCompute feature, see Create a job to ship data to MaxCompute (new version).
-
-
After you finish the configuration, click OK to start data shipping.
After the job starts, Simple Log Service periodically runs offline data shipping jobs.
-
Return to the Simple Log Service console to view the status of the data shipping job.
-
Run the following statement in MaxCompute to query the table data and verify the data shipping result:
SELECT id,name,salenum,dt FROM loghub WHERE substr(dt,1,18) = '2019_10_25';If the following result is returned, the data has been shipped successfully.
The result set contains four columns: id, name, salenum, and dt, with a total of 19 rows. The results include sales data from two time partitions:
2019_10_25_10_00and2019_10_25_10_30. This confirms that the data has been shipped to the destination table.