Create a Scheduled SQL task to import data between Logstores

Procedure

1. Log on to the Log Service console for the project that contains the source Logstore. Use an Alibaba Cloud account or a RAM user with Scheduled SQL permissions.

2. In the Projects section, click the project that contains the source Logstore.

3. In the left-side navigation pane, click Log Storage. In the Logstore list, click the Logstore to analyze. This is the source Logstore for the Scheduled SQL task.

4. Enter a query and analysis statement, and then click Last 15 Minutes to set a time range for the query and analysis.

   Note

   Preview the task to verify your query and analysis statement returns expected data.

5. On the Graph tab, click Save as Scheduled SQL Job.

6. Create a Scheduled SQL task.

   1. In the Compute Settings wizard, configure the parameters and click Next.

      Parameter	Description
      Task Name	Unique task name. You can use the default name.
      Write Mode	Select Import Data from Logstore to Logstore. Processes source Logstore data and writes results to a destination Logstore.
      SQL Code	The query and analysis statement from Step 4 is displayed. To use a different statement, enter it, select a time range, and click Preview to confirm the results. When the Scheduled SQL task runs, Simple Log Service executes the statement to analyze data.
      Target Warning Do not configure the destination store as the current source store (same-source configuration). Otherwise, logs may be written in a loop, which incurs additional storage and traffic costs. You are responsible for the resource consumption and costs incurred. Same account Parameter Description Destination Region Select the region of the Destination Project. Destination Project Select the Destination Project. Target Store Select the destination Logstore. Write Authorization Select a Default Role or a Custom Role for writing data to the destination Logstore. Default Role: Grants permissions to run SQL analysis on source Logstores or MetricStores and write results to destination Logstores or MetricStores. Create a Scheduled SQL task by using a default role. Custom Role: Create a custom role and policy for fine-grained access control. Grant a custom RAM role the permissions to write data to a destination Logstore. Cross-account Parameter Description Destination Region Select another region and agree to the Data Cross-border Compliance Commitment. Destination Project Enter the destination project name, such as test-project. Target Store Enter the destination Logstore name, such as test-logstore. Write Authorization Select Custom Role. Grant a custom RAM role the permissions to write data to a destination Logstore.
      SQL Execution Authorization	Select a Default Role or a Custom Role for querying and analyzing data in the source Logstore. Default Role: Grants permissions to run SQL analysis on source Logstores or MetricStores and write results to destination Logstores or MetricStores. Create a Scheduled SQL task by using a default role. Custom Role: Create a custom role and policy for fine-grained access control. Grant a custom RAM role the permissions to analyze data in a source Logstore.

   2. In the Scheduling Settings wizard, configure the parameters and click OK.

      Parameter	Description
      Scheduling Interval	How often the task runs. Each run generates an execution instance. The scheduling interval determines the scheduled time of each instance. Fixed Interval: The task is scheduled at a fixed interval. Cron: Schedules the task based on a Cron expression. The Cron expression runs on a 24-hour clock with a minimum precision of 1 minute. For example, 0 0/1 * * * runs the task every hour, starting from 00:00. To configure a time zone, you must select the Cron mode. For a list of common time zones, see Time zone formats.
      Scheduling Time Range	The time range for scheduling instances. Valid values: From Specific Time: the time when the first instance is scheduled to run. Specific Time Range: The start and end times to schedule instances. The Scheduled SQL task runs only within this time range. Note Instances run only within this range. No new instances are generated outside it. The scheduling time range corresponds to __time__. Reserved fields.
      SQL Time Window	The logs within this time window are analyzed per run. The SQL time window works with the scheduling interval. The time window cannot exceed 5x the Scheduling Interval and cannot exceed one day. Time expression syntax. For example, if Scheduling Interval is a fixed interval of 10 minutes, Start At is 2021-04-01 00:00:00, Delay Task is 30 seconds, and SQL Time Window is [@m-10m,@m), the system generates the first execution instance at 00:00:30 and analyzes logs in the time range of [23:50:00~00:00:00). Scheduling and execution scenarios. Note The SQL time window corresponds to __time__. Reserved fields. If you do not specify the __time__ field in the SQL code, the __time__ field of logs written to the destination Logstore uses the start time of the scheduling window by default.
      SQL Timeout	The retry threshold when SQL analysis fails. If the retry duration or count exceeds the maximum, the instance fails. Manually retry failed instances based on failure cause. Retry a Scheduled SQL task instance.
      Delay Task	Delay before each run. Valid values: 0 to 120. Unit: seconds. Use this to accommodate delayed data writes.

7. After creation, view the task in the Job Management > Scheduled SQL section. Manage Scheduled SQL tasks.

SDK examples

Create a Scheduled SQL task