Create a data transformation job

Procedure

1. Log on to the Simple Log Service console.

2. Go to the data transformation page.

   1. In the Projects section, click the project you want.

   2. On the Log Storage > Logstores tab, click the logstore you want.

   3. On the query and analysis page, click Data Transformation.

3. In the upper-right corner, select a time range for the data.

   Make sure that logs appear on the Raw Logs tab after you select the time range.

4. In the editor, enter a data transformation statement.

   For more information about the statement syntax, see Data transformation syntax.

5. Preview the data.

   1. Click Quick.

      Simple Log Service supports Quick and Advanced preview modes. For more information, see Preview and debug data.

   2. Click Preview Data.

      View the preview results.

      • If the data transformation fails due to an incorrect statement or permission errors, follow the instructions on the page to resolve the issue.

      • If the results are correct, proceed to step 6.

6. Create a data transformation job.

   1. Click Save as Transformation Job (Old Version).

   2. In the Create Data Transformation Job panel, configure the following parameters and click OK.

      Parameter	Description
      Job name	The name of the data transformation job.
      Display name	The display name of the data transformation job.
      Job description	The description of the data transformation job.
      Authorization method	The authorization method for reading data from the source Logstore. Default Role: The job assumes the AliyunLogETLRole system role to read data from the source Logstore. Custom Role: The job assumes a custom role to read data from the source Logstore. You must first grant the custom role permissions to read data from the source Logstore. Then, enter the ARN of the custom role in the Role ARN field. For more information, see Use a custom role to access data.
      Storage destination
      Destination name	The name of the storage destination. A storage destination includes configurations such as the project and Logstore. You can create multiple storage destinations to store transformation results in different Logstores. Note the following: You can use the name parameter in the e_output() or e_coutput() function within your transformation statement to define the Destination name. For more information, see e_output. If the transformation statement does not include the e_output() function, the job writes results to the first configured destination Logstore by default. If you have only one destination Logstore, the e_output() function is typically not needed. If you specify the name, project, and logstore parameters in the e_output() or e_coutput() function, those settings override the Destination project and Destination logstore configurations in the UI.
      Destination region	The region of the destination project. Cross-region data transformation uses HTTPS to ensure data privacy. Cross-region data transformation transmits data over the public internet, which may cause latency due to network instability. Select the DCDN Acceleration checkbox to accelerate cross-region transmission. Before you use this feature, make sure that acceleration is enabled for the corresponding project. For more information, see Accelerate log collection. Note Cross-region data transformation is billed based on outbound internet traffic (after compression). For more information, see Pay-by-feature.
      Destination project	The project that stores the transformed data.
      Destination logstore	The Logstore that stores the transformation results. Warning Do not set the destination Logstore to the same as the source Logstore. This misconfiguration causes an infinite data loop and incurs additional storage and traffic costs. You are responsible for any charges resulting from this error.
      Authorization method	The authorization method for writing data to the destination Logstore. Default Role: The job assumes the AliyunLogETLRole system role to write results to the destination Logstore. Custom Role: The job assumes a custom role to write results to the destination Logstore. You must first grant the custom role permissions to write data to the destination Logstore. Then, enter the ARN of the custom role in the Role ARN field. For more information, see Use a custom role to access data.
      Processing scope
      Time range	The time range of data to transform. Note This time range is based on the time when logs are received by the service. All: Processes all logs from the earliest available time point in the Logstore until you manually stop the job. From Specific Time: Processes logs from a specified start time until you manually stop the job. Specific Time Range: Processes logs within a specified time range. The job stops automatically when it reaches the end of the range.
      Advanced options
      Advanced parameter configuration	For sensitive information such as database passwords used in transformation statements, save the information as key-value pairs in a secret and reference it in the statement by using res_local("key"). Click + to add multiple key-value pairs. For example, config.vpc.vpc_id.test1:vpc-uf6mskb0b****n9yj can represent the ID of the VPC where an RDS instance is located.