Synchronize SLS data by using the data synchronization feature

Step 1: Create data source and data link

1. Log on to the Simple Log Service console.

2. In the Project list, click the target project. In the left-side navigation pane, click , go to the Logstores tab, and then expand the target Logstore.

3. Click Data Processing > Export, and then click + next to AnalyticDB.

4. In the Shipping Note dialog box that appears, select Create in AnalyticDB for MySQL Console.

   Direct delivery sends all new data to the destination. To process data (for example, by filtering, format conversion, or data enrichment) or deliver historical data, use the Data Processing feature. This involves routing data to a temporary Logstore and then configuring direct delivery from that Logstore.

5. On the AnalyticDB for MySQL Log Synchronization page, configure the parameters in the Source and Destination Settings, Destination Database and Table Settings, and Synchronization Settings sections.

   • The following table describes the parameters in the Source and Destination Settings section.

     Parameter	Description
     Job Name	The name of the data link. The system generates a default name from the data source type and the current time. Change the name as needed.
     Simple Log Service Project	The SLS project.
     Simple Log Service Logstore	The SLS Logstore.
     Destination AnalyticDB for MySQL Cluster	Select the AnalyticDB for MySQL cluster.
     AnalyticDB for MySQL Account	The database account for the AnalyticDB for MySQL cluster.
     AnalyticDB for MySQL Password	The password for the database account of the AnalyticDB for MySQL cluster.

   • Destination Database and Table Settings parameters:

     Parameter	Description
     Database Name	The name of the database in the AnalyticDB for MySQL cluster.
     Table Name	The name of the data table in the AnalyticDB for MySQL cluster.
     Source Data Preview	Click View Latest 10 Logstore Data Entries to view the 10 latest data entries from the source SLS Logstore.
     Schema Field Mapping	The system automatically populates the Destination Field and Source Field with the fields from the AnalyticDB for MySQL cluster table. If the mapping between the Destination Field and Source Field is incorrect, modify it manually. For example, if a field in the AnalyticDB for MySQL cluster table is named name and the corresponding field in the source SLS data is named user_name, the system automatically populates both Destination Field and Source Field with name. In this case, you must manually change the Source Field to user_name.

   • The following table describes the parameters in the Synchronization Settings section.

     Parameter	Description
     Start Offset	When the sync task starts, it begins consuming SLS data from the specified time. For example, if you set Start Offset to 2024-04-09 13:10, the system starts consuming data from the first entry after 13:10 on April 9, 2024.
     Dirty Data Processing Mode	During data sync, if the data type of a field in the destination table does not match the data type of the corresponding field from the source SLS data, the sync will fail. For example, if the source data is abc but the destination field type is int, a conversion error occurs and the sync fails. The dirty data handling mode can be one of the following: Stop Synchronization (Default): The sync task stops. You must then change the field type in the destination table or select a different handling mode, and then restart the sync task. Treat as Null: The dirty data is written as a NULL value to the destination table, and the original dirty data is discarded. For example, if a row of SLS data has three fields (col1, col2, and col3) and the col2 field contains dirty data, the value for col2 is converted to NULL and written to the table. The data for the col1 and col3 fields is written normally.
     Convert UNIX Timestamp into Datetime	If a source SLS field is a Unix timestamp (for example, 1710604800) and the destination field type is DATETIME or TIMESTAMP, enable this feature to convert the timestamp. After you enable it, you can select Timestamp Accurate to Seconds, Timestamp Accurate to Milliseconds, or Timestamp Accurate to Microseconds based on the precision of your SLS data.
     Job Resource Group	Select the job-specific resource group to run the incremental sync task. Important This parameter is required only if the AnalyticDB for MySQL cluster is of Enterprise Edition, Basic Edition, or Data Lakehouse Edition.
     ACUs for Incremental Synchronization	The initial number of ACUs for the incremental sync task. This value must be within the range of [1, Maximum compute resources of the job-specific resource group]. After the data link is created, AnalyticDB for MySQL automatically scales the number of ACUs used to run the incremental sync task based on the business workload. The number of ACUs can be scaled up to 64 or down to 1. Important This parameter is required only if the AnalyticDB for MySQL cluster is of Enterprise Edition, Basic Edition, or Data Lakehouse Edition.

6. After configuring the parameters, click Submit.

   The system then automatically creates an SLS data source and a data link in AnalyticDB for MySQL and redirects you to the SLS/Kafka Data Synchronization page in the AnalyticDB for MySQL console.

Step 2: Start the sync task

1. In the Actions column of the target data link, click Start.

2. After the task starts, click Search in the upper-right corner to refresh its status. The task has started successfully when its status changes to Running.

Step 3: Manage the sync task

You can perform the following operations in the Actions column:

Step 1 (Optional): Configure RAM authorization

To synchronize cross-account SLS data to AnalyticDB for MySQL, you must create a RAM role in the source account, grant it the required permissions, and then modify its trust policy.

1. Create a RAM role. For more information, see Create a RAM role for a trusted Alibaba Cloud account.

   Note

   For the Select Trusted Alibaba Cloud Account parameter, select Other Alibaba Cloud Account and enter the ID of the Alibaba Cloud account that owns the AnalyticDB for MySQL cluster. To obtain the account ID, log on to the Account Center. On the Overview page, find your Account ID.

2. Use precise authorization to grant the AliyunAnalyticDBAccessingLogRolePolicy permission to the RAM role.

3. Modify the trust policy of the RAM role to allow the AnalyticDB for MySQL cluster that belongs to the specified Alibaba Cloud account to assume this RAM role.

   {
     "Statement": [
       {
         "Action": "sts:AssumeRole",
         "Effect": "Allow",
         "Principal": {
           "RAM": [
               "acs:ram::<Alibaba Cloud account ID>:root"
           ],
           "Service": [
               "<Alibaba Cloud account ID>@ads.aliyuncs.com"
           ]
         }
       }
     ],
     "Version": "1"
   }

   Note

   The Alibaba Cloud account ID is the ID of the account that owns the AnalyticDB for MySQL cluster, which you specified in Step 1. Do not include the angle brackets (<>) when you configure the policy.

Step 2: Create a data source

1. Log on to the AnalyticDB for MySQL console. In the upper-left corner of the console, select a region. In the left-side navigation pane, click Clusters. Find the cluster that you want to manage and click the cluster ID.

2. In the navigation pane on the left, click Simple Log Service Data Ingestion > Data Sources.

3. In the upper-right corner, click Create Data Source.

4. On the Create Data Source page, configure the parameters. The following table describes the parameters:

   Parameter	Description
   Data Source Type	Select SLS.
   Data Source Name	The system generates a name based on the data source type and the current time. Change the name as needed.
   Data Source Description	A description of the data source, such as its application scenario or business limits.
   Deployment Mode	Only Alibaba Cloud Instance is supported.
   Region of Simple Log Service Project	The region where the SLS project is located. Note Only the region where the AnalyticDB for MySQL cluster is located is supported.
   Across Alibaba Cloud Accounts	An AnalyticDB for MySQL cluster can synchronize SLS data from the same Alibaba Cloud account or from another Alibaba Cloud account (cross-account). No: Synchronize SLS data from the current Alibaba Cloud account to the AnalyticDB for MySQL cluster. Yes: Synchronize SLS data from another Alibaba Cloud account to the AnalyticDB for MySQL cluster. When you select Yes, configure RAM authorization and specify the Alibaba Cloud Account and RAM Role parameters. For more information about how to configure RAM authorization, see Configure RAM authorization. Note Alibaba Cloud Account: The ID of the Alibaba Cloud account to which the SLS project belongs. RAM Role: The RAM role that belongs to the Alibaba Cloud account of the SLS project. This is the RAM role created in Step 1 of Configure RAM authorization.
   Simple Log Service Project	The source SLS project.
   Simple Log Service Logstore	The source SLS Logstore.

5. After configuring the parameters, click Create.

Step 3: Create a sync link

1. In the navigation pane on the left, click Simple Log Service Data Ingestion > Data Synchronization.

2. In the upper-right corner, click Create Synchronization Job.

3. On the Create Synchronization Job page, configure Source and Destination Settings, Destination Database and Table Settings, and Synchronization Settings. The following table describes the parameters.

   • Source and Destination Settings parameters:

     Parameter	Description
     Job Name	The name of the data link. The system generates a default name from the data source type and the current time. Change the name as needed.
     Data Source	Select an existing SLS data source or create a new one.
     Destination Type	For Enterprise Edition, Basic Edition, and Data Lakehouse Edition clusters, select Data Warehouse-ADB Storage. You do not need to specify this parameter for Data Warehouse Edition clusters.
     AnalyticDB for MySQL Account	The database account for the AnalyticDB for MySQL cluster.
     AnalyticDB for MySQL Password	The password for the database account of the AnalyticDB for MySQL cluster.

   • Destination Database and Table Settings parameters:

     Parameter	Description
     Database Name	The name of the database in the AnalyticDB for MySQL cluster.
     Table Name	The name of the data table in the AnalyticDB for MySQL cluster.
     Source Data Preview	Click View Latest 10 Logstore Data Entries to view the 10 latest data entries from the source SLS Logstore.
     Schema Field Mapping	The system automatically populates the Destination Field and Source Field with the fields from the AnalyticDB for MySQL cluster table. If the mapping between the Destination Field and Source Field is incorrect, modify it manually. For example, if a field in the AnalyticDB for MySQL cluster table is named name and the corresponding field in the source SLS data is named user_name, the system automatically populates both Destination Field and Source Field with name. In this case, you must manually change the Source Field to user_name.

   • Synchronization Settings parameters:

     Parameter	Description
     Start Offset	When the sync task starts, it begins consuming SLS data from the specified time. For example, if you set Start Offset to 2024-04-09 13:10, the system starts consuming data from the first entry after 13:10 on April 9, 2024.
     Dirty Data Processing Mode	During data sync, if the data type of a field in the destination table does not match the data type of the corresponding field from the source SLS data, the sync will fail. For example, if the source data is abc but the destination field type is int, a conversion error occurs and the sync fails. The dirty data handling mode can be one of the following: Stop Synchronization (Default): The sync task stops. You must then change the field type in the destination table or select a different handling mode, and then restart the sync task. Treat as Null: The dirty data is written as a NULL value to the destination table, and the original dirty data is discarded. For example, if a row of SLS data has three fields (col1, col2, and col3) and the col2 field contains dirty data, the value for col2 is converted to NULL and written to the table. The data for the col1 and col3 fields is written normally.
     Convert UNIX Timestamp into Datetime	If a source SLS field is a Unix timestamp (for example, 1710604800) and the destination field type is DATETIME or TIMESTAMP, enable this feature to convert the timestamp. After you enable it, you can select Timestamp Accurate to Seconds, Timestamp Accurate to Milliseconds, or Timestamp Accurate to Microseconds based on the precision of your SLS data.

4. After you configure the parameters, click Submit.

Step 4: Start the data sync task

1. On the Data Synchronization page, select the created data sync task. In the Actions column, click Start.

2. Click Search in the upper-right corner. The sync task has started successfully when the status changes to Running.

Step 5: Manage data sources

On the Data Synchronization page, you can perform the following actions in the Actions column.