Create and use Data Engineering-Cloudera Data Platform(CDP)-阿里云帮助中心

This topic uses a simple example to describe how to create and use Data Engineering. You will learn how to create and initialize a virtual Spark cluster and how to create and submit a Spark job.

Prerequisites

The Data-Service cluster Management Console is deployed in a private network. You must use a Secure Shell (SSH) tunnel to access the console. For more information, see Access the web UI of a Cloudera Manager service component through an SSH tunnel.

Procedure

Step 1. Create a user and assign permissions in FreeIPA

Add a user in FreeIPA. In the User login field, enter a username. This username is used to log on to the Management Console of the Data-Service cluster. In the New Password field, enter a password. You can fill in the other fields as needed.
Log on to the Management Console of the Data-Service cluster as the new user.
At this point, the user does not have any permissions. Log on to the Management Console again as the administrator `admin` to assign the PowerUser role to the new user.
Log out of the administrator account. Log on to the Management Console again as the new user to continue.

Step 2. Create CDE Services in the CDE console

In the CDE console, click Enable CDE Service. Enter the required Name and Environment information. Then, click Enable. The CDE service starts within a few minutes.

Step 3. Create a virtual Data Engineering cluster

After the CDE Service is started, click Create DE Cluster to create a virtual Spark cluster. Enter a cluster name, select the CDE Service and Spark version, and then click Create.

Step 4. Initialize the cluster

Log on to the `cdp-ds-1-1` node and download the initialization script: cdp-cde-utils.sh. Then, grant the execute permission to the script. If the `cdp-ds-1-1` node cannot connect to the external network, you can download the script on the `cdp-utility-1-1` node and copy it to the `cdp-ds-1-1` node. The following script provides an example of how to download the script on the `cdp-utility-1-1` node and copy it to the `cdp-ds-1-1` node.
```
# Log on to the cdp-utility-1-1 node using SSH.
# Download the initialization script.
wget https://docs.cloudera.com/data-engineering/1.4.0/cdp-cde-utils.sh
# Grant the execute permission to the script.
chmod +x cdp-cde-utils.sh
# Copy the initialization script to the cdp-ds-1-1 node.
scp cdp-cde-utils.sh cdp-ds-1-1:/root
```
Obtain the domain name of the virtual cluster.
1. Go to the CDE Service management page and click Cluster Details to open the details page.
2. Click JOBS API URL. Then, copy the API URL and paste it into a text editor.
3. Obtain the domain name of the virtual cluster from the URL. For example, if the URL that you copied in the previous step is https://9nfr2b8c.cde-9mbk9hlb.apps.cdp-ds-1-1/dex/api/v1, the domain name of the virtual cluster is `9nfr2b8c.cde-9mbk9hlb.apps.cdp-ds-1-1`.
Initialize the cluster and generate a self-signed certificate.
On the `cdp-ds-1-1` node, go to the directory where the `cdp-cde-util.sh` initialization script is stored. Run the following command: ./cdp-cde-utils.sh init-virtual-cluster -h <domain_name> -a. Replace `<domain_name>` in the command with the domain name that you obtained in the previous step. For example, for the domain name that is obtained in the previous step, the initialization command is as follows:
```
./cdp-cde-utils.sh init-virtual-cluster -h 9nfr2b8c.cde-9mbk9hlb.apps.cdp-ds-1-1 -a
```
After the initialization is complete, you can access the pages for creating and submitting Spark jobs and the Airflow UI from the console. However, you cannot submit jobs to the virtual Spark cluster yet. Before the user `alex` can submit jobs to the virtual cluster, you must perform security authentication for `alex`.
1. Log on to the `cdp-ds-1-1` node and run the kinit alex command. Enter the password that you set when you created the user. The first time you run the `kinit` command, you are prompted to change the password. Follow the on-screen instructions.
2. On the `cdp-ds-1-1` node, run the ipa-getkeytab -p alex -k alex.keytab command to obtain the keytab.
3. On the `cdp-ds-1-1` node, create a file named `alex.principal` that contains `alex@ALICLOUDPVC.COM`.
4. Run the following command: ./cdp-cde-utils.sh init-user-in-virtual-cluster -h <domain_name> -u admin -p alex.principal -k alex.keytab.

Step 5. Submit and schedule jobs using the Data Engineering console

Go to the Spark job management interface.
On the Spark job management interface, create a job. This example uses the Pi calculation from the Spark examples.
Click Create and Run. The submitted Spark job appears on the Job Runs tab.