Easily build an ETL offline data processing system with CloudFlow and Function Compute

Background information

The widespread use of new technologies such as cloud computing, artificial intelligence, and the Internet of Things (IoT) has led to explosive data growth. This growth creates a higher demand for data processing capabilities. Data is now a key asset, and the ability to collect and process it has become a core competitive advantage. Examples include application service monitoring, operational data analysis, and data filtering and pre-processing for deep learning. The ability to process this data directly affects service operational efficiency. You can use existing ETL systems for these tasks. However, you may want to build your own service in many cases. For example:

• Your data processing tasks run at irregular intervals. You want to avoid consuming resources when no tasks are running.

• Your data processing needs involve only a few simple steps. A self-built service is faster to implement.

• Your data processing workflow has many custom steps. Because existing systems are not flexible enough, you must build your own service to meet your business requirements.

• You do not want to spend much effort setting up and maintaining various open-source data processing modules. However, you still need good performance for high-concurrency data processing requests.

If you have these requirements or want to implement a highly flexible, highly reliable, low-cost, and high-performance offline data processing system, the serverless solution in this topic is an ideal choice.

Scenarios

Suppose you have a batch of data to process. The data values are either data_1 or data_2. The goal is to count the occurrences of each data type and store the results in a data warehouse. When the data volume is large or the data sources are different, processing all the data in a single process is difficult. In this scenario, the combination of CloudFlow and Function Compute provides an efficient solution.

To demonstrate the core data processing capabilities, this example uses Alibaba Cloud Object Storage Service (OSS) as the data warehouse. OSS represents various types of databases and other storage systems.

The following solution shows how to use CloudFlow and Function Compute to implement a low-cost, highly elastic data processing system. In this system, Function Compute dynamically provides computing resources for data processing and statistical analysis based on the data volume. CloudFlow orchestrates the logic for complex upstream and downstream business processes.

How it works

A typical data processing task focuses on the following aspects:

Data source: The source of the data to be processed. Typically, data comes from sources such as databases or text files, such as log files. In this example, a function generates a small amount of data for demonstration. In a real-world scenario, you can use any custom data source as the input for the system.

Processing framework/pattern: MapReduce. This example uses CloudFlow to implement the framework.

Destination: Data warehouse. This example uses OSS as the data warehouse, which is the destination for the processed data.

Data processing flow:

First, the raw data is randomly divided into 3 to 5 shards. Each shard contains two types of data. Next, statistical analysis is performed on each shard, and the intermediate results are stored (Map). Finally, the statistical results from all shards are aggregated, summed, and the final result is stored (Reduce). The workflow is as follows:

1. Retrieve data from the data source.

2. Divide the data into shards based on a rule or at random.

3. Use MapReduce to process the data in parallel.

4. Store the results in the final destination.

The following diagram shows the system modules and their interactions using Alibaba Cloud services.

Prerequisites

• Activate Function Compute

• Activate Object Storage Service

• Activate CloudFlow

Procedure

1. Deploy CloudFlow and its functions from the Function Compute Application Center with a single click.

   1. Go to the ETLDataProcessing template.

   2. Click One-click Deploy to open the Function Compute Application Center and start the deployment.

      Deploy with Serverless Devs CLI:

      1. Install the Serverless Devs CLI developer tool and configure the authorization information.

      2. Initialize the project:

         s init ETLDataProcessing –d ETLDataProcessing

      3. Enter the project directory and deploy the project:

         cd ETLDataProcessing && s deploy – y

   3. On the Create Application page, configure the following parameters and click Create and Deploy Default Environment.

      Configure the key parameters as follows. You can keep the default values for the other parameters.

      Parameter Name	Parameter Description
      Basic Information
      Deployment Type	Select Direct Deployment.
      Role Name	If you use an Alibaba Cloud account and are creating an application in the Application Center for the first time, click Go to Authorization. You will be redirected to the Quick Role Creation page. Create the AliyunFCServerlessDevsRole role, and then click Agree to Authorization. If you use a Resource Access Management (RAM) user, provide the authorization link to the primary Alibaba Cloud account as prompted. After authorization is complete, click Authorization Completed. Note If the page displays the message Cannot get role, contact the primary Alibaba Cloud account holder to grant the AliRAMReadOnlyAccess and AliyunFCFullAccess permissions to your RAM user. For more information, see Grant permissions to a RAM user.
      Advanced Configuration
      Region	Select the region where the application will be deployed.
      Workflow Execution Role	The service role for workflow execution. Create a role and grant it the AliyunFCInvocationAccess policy beforehand.
      Function Service Role	The service role that Function Compute uses to access other Alibaba Cloud services. If you have no special requirements, use the default service role, AliyunFCDefaultRole.
      Object Storage Service Bucket Name	Enter the name of a bucket that is in the same region as the workflow and function.

      After 1 to 2 minutes, the application is deployed. The system automatically creates three functions and one workflow named etl-data-processing-2q1i. You can log on to the Function Compute console and the CloudFlow console to view the results.

      • shards-spliter: Reads the data source, splits the source data into shards based on a rule, and returns the shards to the workflow.

      • mapper: The Map function in the MapReduce framework. This function filters, cleans, and processes the data from an incoming shard. To speed up processing, multiple instances of this function run in parallel, one for each shard. After each map function is complete, it stores the result in a specific directory in OSS.

      • reducer: The Reduce function in the MapReduce framework. This function integrates and merges the results from the map functions and pushes the final result to OSS.

      You can obtain the complete code for this example from the application in the Function Compute console.

2. Verify the results.

   1. Log on to the CloudFlow console. In the top menu bar, select a region.

   2. In the workflow list, click the etl-data-processing-2q1i workflow. On the workflow details page, select the Execution History tab and click Start Execution.

      After the execution is complete, check the workflow input and output.

      On the execution details page, select the Input and Output tab. The Execution Input is {"key": "hello world"} and the Execution Output is {"ReducingStatus": "operation success"}.

   3. Log on to the Object Storage Service (OSS) console to view the sharded content and the merged results.

      In the file list of the OSS console, you can see the shard files generated by the MapReduce job (map_shard_0, map_shard_1, map_shard_2, and map_shard_3), the merged result file (reduced_result), and the related directories dst/ and src/.