Create a data transformation job (new version)

Prerequisites

• Data collected in SLS. Data collection overview.

• If you use a RAM user, grant data transformation permissions. Authorize a RAM user to perform data transformation operations.

Procedure

1. Log on to the Simple Log Service console.

2. Go to the data transformation page.

   1. In the Projects section, click the project you want.

   2. On the Log Storage > Logstores tab, click the logstore you want.

   3. On the query and analysis page, click Data Transformation.

3. In the upper-right corner of the page, select a time range.

   After selecting a time range, verify that logs appear on the Raw Logs tab.

4. In the editor, enter an SPL statement.

   For more information about the SPL syntax, see SPL syntax.

5. Preview data.

   • On the Raw Data tab, click Add Test Data. You can also manually enter test data on the Test Data tab.

6. Create a data transformation job.

   1. Click Save as Transformation Job (New Version).

   2. In the Create Data Transformation Job (New Version) panel, configure the following parameters and click OK.

   Parameter	Description
   Task Name	The name of the data transformation job.
   Display Name	The display name of the job.
   Job Description	The description of the job.
   Authorization Method	Grant the job read access to the source logstore using one of these methods: Default Role: The job assumes the AliyunLogETLRole system role to read data from the source logstore. Click Authorize the system role AliyunLogETLRole and complete the authorization as prompted. For more information, see Access data using a default role. Important If you use a RAM user, an Alibaba Cloud account must complete the authorization first. If your Alibaba Cloud account is already authorized, you can skip this step. Custom Role: The job assumes a custom role to read data from the source logstore. You must first grant the custom role permissions to read data from the source logstore, and then enter the ARN of the role in the Role ARN field. For more information, see Access data using a custom role. AccessKey: For security reasons, you can no longer use an AccessKey pair (AK/SK) to create jobs.
   Storage Destination
   Destination Name	The storage destination name. Includes configurations such as the Project and logstore.
   Destination Region	The region where the destination Project is located.
   Destination Project	The destination Project for transformation results. A Project specified in your SPL statement overrides this setting. Dynamic destination Project/logstore output. Important The Project that you dynamically specify in the SPL statement must match the region and authorization that you configure here.
   Target Store	The destination logstore for transformation results. A logstore specified in your SPL statement overrides this setting. Dynamic destination Project/logstore output. Important The logstore that you dynamically specify in the SPL statement must match the region, authorization, and Project that you configure here. The destination logstore cannot be the same as the source logstore. Warning Do not configure the destination store as the current source store (same-source configuration). Otherwise, logs may be written in a loop, which incurs additional storage and traffic costs. You are responsible for the resource consumption and costs incurred.
   Authorization Method	Grant the job write access to the destination logstore using one of these methods: Default Role: The job assumes the AliyunLogETLRole system role to write results to the destination logstore. Click Authorize the system role AliyunLogETLRole and complete the authorization as prompted. For more information, see Access data using a default role. Important If you use a RAM user, an Alibaba Cloud account must complete the authorization first. If your Alibaba Cloud account is already authorized, you can skip this step. Custom Role: The job assumes a custom role to write results to the destination logstore. You must first grant the custom role permissions to write data to the destination logstore, and then enter the ARN of the role in the Role ARN field. For more information, see Access data using a custom role. AccessKey: For security reasons, you can no longer use an AccessKey pair (AK/SK) to create jobs.
   Write to Result Set	The dataset to write to the destination logstore. Multiple datasets can target one destination, and multiple destinations can share a dataset. Dataset description.
   Processing scope
   Time Range (Data Receiving Time)	The time range for the job: All: Processes all data from the first log entry until manually stopped. From Specific Time: Processes data from the specified start time until manually stopped. Specific Time Range: Processes data within the specified time window and stops automatically at the end time.
   Advanced Options
   Advanced Parameter Settings	Store sensitive information (such as database passwords) as key-value pairs, and reference them in your statement using res_local("key"). Click + to add multiple key-value pairs. For example, config.vpc.vpc_id.test1:vpc-uf6mskb0b****n9yj specifies the ID of the VPC where the RDS instance resides.

What to do next

After creating the job:

• View, modify, or stop jobs on the Data Transformation Overview page. Manage data transformation (new version) jobs.

• Query and analyze data in the destination logstore. Quick start for query and analysis.