Migrate log data from Simple Log Service to MaxCompute

Method 1: Use the root account AccessKey pair

Use the AccessKey ID and AccessKey secret of Account A to add the LogHub data source. This grants Account B access to all SLS projects under Account A.

Method 2: Use a RAM user AccessKey pair

Use the AccessKey ID and AccessKey secret of a RAM user (for example, RAM User A1) under Account A. This method allows you to restrict access to specific SLS projects.

Grant full access

Attach the AliyunLogFullAccess and AliyunLogReadOnlyAccess system policies to RAM User A1. For more information, see Create a RAM user and authorize the RAM user to access Simple Log Service.

These system policies grant access to all SLS projects under Account A.

Grant project-level access

To restrict access to specific SLS projects, attach a custom policy. Log on to the RAM console with Account A. In the left-side navigation pane, choose Permissions > Policies, and click Create Policy.

For more information about custom permissions, see RAM overview and Overview.

The following example policy grants access only to project_name1 and project_name2:

{
    "Version": "1",
    "Statement": [
        {
            "Action": [
                "log:Get*",
                "log:List*",
                "log:CreateConsumerGroup",
                "log:UpdateConsumerGroup",
                "log:DeleteConsumerGroup",
                "log:ListConsumerGroup",
                "log:ConsumerGroupUpdateCheckPoint",
                "log:ConsumerGroupHeartBeat",
                "log:GetConsumerGroupCheckPoint"
            ],
            "Resource": [
                "acs:log:*:*:project/project_name1",
                "acs:log:*:*:project/project_name1/*",
                "acs:log:*:*:project/project_name2",
                "acs:log:*:*:project/project_name2/*"
            ],
            "Effect": "Allow"
        }
    ]
}

Option A: Codeless UI

1. In the Configure Network Connections and Resource Group step, configure the source and destination.

   Parameter	Description
   Source	Select LogHub.
   Data Source Name	Select the LogHub data source that you added.
   Resource Group	Select the exclusive resource group for Data Integration.
   Destination	Select MaxCompute.
   Data Source Name	Select the MaxCompute data source that you added.

   

2. Test the network connectivity between the exclusive resource group and both the source and destination. After the test succeeds, click Next.

3. Configure the source Logstore and destination table settings.

   Click Data preview to view sample data. The preview shows only a small number of LogHub records. The displayed data may differ from the actual data to synchronize because of the start time and end time that you specified.

   Source parameters

   Parameter	Description
   Logstore	The Logstore to read data from.
   Log start time	The start time of data consumption. Defines the left boundary of a left-closed, right-open time range. Format: yyyyMMddHHmmss. Example: 20180111013000. Supports DataWorks scheduling parameters.
   Log end time	The end time of data consumption. Defines the right boundary of a left-closed, right-open time range. Format: yyyyMMddHHmmss. Example: 20180111013010. Supports DataWorks scheduling parameters.
   Number of batches	The number of data records to read at a time. Default value: 256.

4. In the Field Mapping section, configure mappings between source fields and destination fields.

5. In the Channel Control section, configure Synchronization rate and Policy for Dirty Data Records.

6. In the right-side navigation pane, click Properties. Configure settings such as rerun properties, a resource group for scheduling, and ancestor tasks.

   When configuring ancestor tasks, select Add Root Node.

7. Verify all settings and click the icon in the top toolbar.

8. Run the batch synchronization task. Run once Click the icon in the top toolbar. Run on a schedule

   1. Click the Properties tab in the right-side navigation pane and configure time properties such as the scheduling cycle.

   2. Click the and icons in the top toolbar in sequence to commit the task to the scheduling system. The scheduling system runs the task periodically starting from the next day based on the configured properties.

   Specify values for any custom parameters before running the task.

Option B: Code editor

1. On the configuration tab of the batch synchronization task, click the Conversion script icon in the top toolbar.

   

2. In the Tips message, click OK to switch to the code editor.

3. Click the Import Template icon in the top toolbar.

   

4. In the Import Template dialog box, configure the Source type, Data source, Target type, and Data source parameters, and click Confirmation.

5. Modify the JSON configuration based on your requirements. The following example shows a complete configuration:

   Code editor parameters

   Section	Parameter	Description	Default
   reader	datasource	LogHub data source name. Must match the name of the added data source.	-
   reader	logstore	Logstore name to read data from.	-
   reader	beginDateTime	Start time of data consumption. Left-closed, right-open boundary.	-
   reader	endDateTime	End time of data consumption. Left-closed, right-open boundary.	-
   reader	batchSize	Number of records to read at a time.	256
   reader	column	Source fields to synchronize. Supports column names, constants, variables, and functions.	-
   writer	datasource	MaxCompute data source name. Must match the name of the added data source.	-
   writer	table	Destination table name.	-
   writer	truncate	Whether to clear the destination table before writing.	-
   writer	partition	Partition information for the destination table.	-
   writer	column	List of destination column names to write to.	-
   setting	mbps	Maximum transmission rate in MB/s.	-
   setting	concurrent	Maximum number of parallel threads.	-

      {
          "type": "job",
          "version": "1.0",
          "configuration": {
              "reader": {
                  "plugin": "loghub",
                  "parameter": {
                      "datasource": "loghub_lzz",         // LogHub data source name. Must match the data source you added.
                      "logstore": "logstore-ut2",          // Logstore name to read data from.
                      "beginDateTime": "${startTime}",     // Start time (left-closed, right-open).
                      "endDateTime": "${endTime}",         // End time (left-closed, right-open).
                      "batchSize": 256,                    // Records per read. Default: 256.
                      "splitPk": "",
                      "column": [                          // Source fields to synchronize.
                          "key1",
                          "key2",
                          "key3"
                      ]
                  }
              },
              "writer": {
                  "plugin": "odps",
                  "parameter": {
                      "datasource": "odps_source",         // MaxCompute data source name. Must match the data source you added.
                      "table": "test",                     // Destination table name.
                      "truncate": true,                    // Whether to clear the destination table before writing.
                      "partition": "",                     // Partition information for the destination table.
                      "column": [                          // Destination columns to write to.
                          "key1",
                          "key2",
                          "key3"
                      ]
                  }
              },
              "setting": {
                  "speed": {
                      "mbps": 8,                           // Maximum transmission rate (MB/s).
                      "concurrent": 7                      // Maximum number of parallel threads.
                  }
              }
          }
      }