Process data

Prerequisite

First, complete the steps described in Synchronize data.

Step 1: Build a data processing workflow

After the data is loaded in the Synchronize data stage, the next step is to process it into user profile data.

1. In the Data Studio left-side navigation pane, click the icon. In the Project Directory section, find and click the User_profile_analysis_Spark workflow you created to open its canvas.

2. From the component list on the left, drag EMR Spark SQL nodes onto the canvas and name them as described below.

   This tutorial uses the following nodes:

   Node type	Node name	Description
   EMR Spark SQL	dwd_log_info_di_spark	Processes data from the ods_raw_log_d_spark table using Spark SQL and writes the result to the dwd_log_info_di_spark table.
   EMR Spark SQL	dws_user_info_all_di_spark	Joins the detail log table dwd_log_info_di_spark and the user table ods_user_info_d_spark on the uid column to generate an aggregated user log table. Aggregates data from the user information table (ods_user_info_d_spark) and the processed log data table (dwd_log_info_di_spark), and writes the result to the dws_user_info_all_di_spark table.
   EMR Spark SQL	ads_user_info_1d_spark	Further processes data from the dws_user_info_all_di_spark table and writes the result to the ads_user_info_1d_spark table to generate basic user profiles.

3. Manually drag to connect the nodes and configure the upstream node for each node. The final result is as follows:

   
   Note

   You can set upstream and downstream dependencies between nodes by manually connecting them in the workflow, or use code parsing to automatically identify node dependencies within internal nodes. This tutorial uses the manual connection method. For more information about code parsing, see Automatic parsing mechanism.

Step 2: Configure data processing nodes

Use EMR Spark SQL nodes to process the user information table and the detail log table, and generate the user profile table ads_user_info_1d_spark.

Step 3: Process data

1. Process the data.

   On the workflow toolbar, click Run. Set the values for the parameter variables defined in each node for this run (this tutorial uses 20250223; you can modify as needed). Click OK and wait for the run to complete.

2. Query the data processing result.

   After all nodes run successfully, run the following SQL queries to verify that the tables contain the expected data.

   1. Verify the result of the dwd_log_info_di_spark table.

      -- You need to update the partition filter condition to the actual business date of your current operation. For example, if the task runs on 20250223, the business date is 20250222, i.e., the day before the task run date.
      SELECT * FROM dwd_log_info_di_spark WHERE dt='${bizdate}';

   2. Verify the result of the dws_user_info_all_di_spark table.

      -- You need to update the partition filter condition to the actual business date of your current operation. For example, if the task runs on 20250223, the business date is 20250222, i.e., the day before the task run date.
      SELECT * FROM dws_user_info_all_di_spark WHERE dt='${bizdate}';

   3. Verify the result of the ads_user_info_1d_spark table.

      -- You need to update the partition filter condition to the actual business date of your current operation. For example, if the task runs on 20250223, the business date is 20250222, i.e., the day before the task run date.
      SELECT * FROM ads_user_info_1d_spark WHERE dt='${bizdate}';

Step 4: Deploy the workflow

Deploy the workflow to the production environment so that tasks can be automatically scheduled.

1. In the left navigation bar of Data Studio, click to go to the DataStudio page. Then, in the Project Directory area, find the created workflow and click the workflow to open the workflow orchestration page.

2. Click Publish in the node toolbar to open the Publish panel.

3. Click Start Release Production. In the confirmation dialog box that appears, select a deployment method based on your requirements:

   • Full deployment: Deploys the current workflow and all its internal task nodes.

   • Incremental deployment: Deploys only the current workflow and the internal task nodes that have been modified since the last deployment. This is suitable for iterative optimizations and minor updates.

4. After you confirm the deployment method, the system automatically executes the deployment process, deploying the workflow and selected task nodes to the development and production environments in sequence. To complete the deployment to the production environment, you must click Confirm Release.

Step 5: Run tasks in the production environment

After a task is deployed, an instance is generated to run on the next day. You can use Supplementary data to backfill data for the deployed workflow and check whether the task can run in the production environment. For more information, see Data Backfill Instance O&M.

1. After the tasks are deployed, click Operation and Maintenance Center in the upper-right corner.

   Alternatively, click the icon in the upper-left corner and choose All Products > Data Development and O&M > Operation and Maintenance Center (Workflow).

2. In the left-side navigation pane, click Auto Triggered Task O&M > Auto Triggered Node to go to the Auto Triggered Node page, and click the workshop_start_spark virtual node.

3. In the DAG on the right, right-click the workshop_start_spark node and choose Supplementary data > Current and Descendant Nodes Retroactively.

4. Select the tasks for which you want to backfill data, set the business date, and click Submit and Redirect.

5. On the backfill data page, click Refresh until all SQL tasks have run successfully.

Next steps

• Analyze data: Visualize the processed data in charts by using the Data Analysis module to extract key insights and identify business trends.

• Monitor data quality: Configure data quality monitoring for the tables generated during data processing to proactively identify and block dirty data before it spreads.

• Manage data assets: View the tables created during this workflow in the Data Map module and use lineage to explore the relationships between them.

• Create data services: Share and consume the processed data through standardized API endpoints by using the Data Service module.