Prepare the environment

Create an EMR cluster

Create an EMR cluster and integrate it with DataWorks so that you can run data processing tasks on the cluster. Set the following key parameters when creating the cluster:

For detailed instructions on creating an EMR cluster, see Create a cluster.

Step 1: Create a workspace

If you already have a workspace in the China (Shanghai) region, you can use it and skip this step.

1. Log on to the DataWorks console. In the top-left corner, switch the region to China (Shanghai).

2. In the left-side navigation pane, click Work space. On the Workspaces page, click Create Workspace to create a workspace in standard mode, which isolates the development and production environments. For details, see Create a workspace.

Step 2: Create a serverless resource group

A serverless resource group handles data synchronization and scheduling in this tutorial. Purchase one and complete the initial setup.

1. Purchase a serverless resource group.

   1. Log on to the DataWorks console, switch the region at the top to China (Shanghai), and click Resource Group in the left navigation bar to go to the Resource Groups page.

   2. Click Create Resource Group. On the buy page, set Region and availability zone to China (Shanghai) and specify a Resource Group Name. Configure other parameters as prompted and complete the payment. For billing information, see Billing of serverless resource groups.

      Note

      If no VPC or vSwitch is available in the current region, click the corresponding link in the parameter description to create them. For more information about VPCs and vSwitches, see What is VPC?

2. Associate the resource group with a DataWorks workspace.

   A newly purchased serverless resource group must be associated with a workspace before use.

   Log on to the DataWorks console. In the top navigation bar, switch the region to China (Shanghai). Find the serverless resource group that you purchased, and in the Operation column, click Associate Workspace. Find your workspace and click Associate.

3. Configure public network access for the resource group.

   The test data is downloaded from the internet. Serverless resource groups lack public network access by default, so you must configure an Internet NAT Gateway and an elastic IP address (EIP) for the associated VPC.

   1. Go to the Internet NAT Gateway page in the VPC console. In the top navigation bar, switch the region to China (Shanghai).

   2. Click Create Internet NAT Gateway and configure the parameters. The following table describes the key parameters for this tutorial. You can keep the default values for other parameters.

      Parameter	Value
      Region	China (Shanghai).
      VPC	Select the VPC and vSwitch that are associated with the resource group. To find this information, go to the DataWorks console, switch to the correct region, and click Resource Groups in the left-side navigation pane. Find your resource group, then click Network Settings in the Operation column. The VPC and Switch are in the Data Scheduling & Data Integration section. For more information about VPCs and vSwitches, see What is VPC?
      Associate vSwitch
      Access mode	Full-cone mode for VPC (SNAT).
      EIP	Purchase New EIP.
      Service-linked role creation	If you are creating an Internet NAT Gateway for the first time, you must create a service-linked role. Click Create Service-Linked Role.

   3. Click Buy Now. Read and agree to the service agreement, and then click Activate to complete the purchase.

For more information about how to add and use serverless resource groups, see Use serverless resource groups.

Step 3: Register EMR cluster and initialize resource group

You must register the EMR cluster with your DataWorks workspace before you can use it.

1. Go to the EMR cluster registration page.

   1. Go to the Management Center page.

      Log on to the DataWorks console. After you switch the region to China (Shanghai), choose More > Management Center in the left-side navigation pane. In the drop-down list, select the target workspace and click Go to Management Center.

   2. In the left-side navigation pane, click Clusters. On the Clusters page, click Register Cluster and set Select Cluster Type to E-MapReduce.

2. Register the EMR cluster.

   On the Register EMR Cluster page, configure the cluster information. The following table describes the key parameters.

   Parameter	Value
   Alibaba Cloud Account to Which Cluster Belongs	Current Alibaba Cloud Account.
   Cluster Type	Data Lake.
   Default Access Identity	Cluster Account: hadoop.
   Pass Proxy User Information	Pass.

3. Initialize the resource group.

   1. On the Clusters page, find the registered EMR cluster and click Initialize Resource Group in the upper-right corner.

   2. Find the target resource group and click Initialization.

   3. After the initialization is complete, click Confirm.

   Important

   Ensure the initialization is successful; otherwise, tasks may fail. If it fails, review the error message and follow the prompts to run a connectivity diagnosis.

For detailed instructions on how to register an EMR cluster, see DataStudio (legacy version): Associate an EMR compute resource.