DataWorks with EMR Serverless StarRocks

Background

Prerequisites

• You have activated DataWorks and created a workspace. For more information, see Activate DataWorks.

• You have purchased a resource group and completed workspace binding and network configuration. For more information, see Resource group management.

• You have created an EMR Serverless StarRocks instance. For more information, see Quick start for all-in-one instances.

  Note

  After you create the StarRocks instance, you can view its information in the EMR console. You can also connect to the instance using EMR StarRocks Manager to view information such as databases and tables.

• You have added the IP address of the DataWorks resource group to the whitelist of the EMR Serverless StarRocks instance.

  You can configure the whitelist for an EMR Serverless StarRocks instance by using the following methods:

  On the Instance Details page of the EMR Serverless StarRocks instance, click Internal Whitelist to the right of Security Group ID to configure the internal network whitelist. In the FE Details section, click Public Whitelist to the right of Public Endpoint to configure the public whitelist.

Create data source

To use EMR Serverless StarRocks in DataWorks, you must first create a StarRocks data source that connects to your existing EMR Serverless StarRocks database. This allows you to use EMR Serverless StarRocks in various DataWorks components.

For more information, see StarRocks data source. The entry point and key configuration parameters are described in the following steps.

1. 1. Log on to the DataWorks console. In the target region, click More > Management Center in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Management Center.

   2. On the Workspace Management page, click Data Sources in the left-side navigation pane to open the data source page.

2. Click Add data source. Configure the key parameters as described in the following section. You can leave other parameters at their default values.

   Choose the appropriate method to create the data source based on the network connectivity between your StarRocks instance and the DataWorks resource group. For more information about network connectivity solutions, see Overview of network connectivity solutions.

   Internal network

   Parameter	Description
   Configuration Mode	Select ApsaraDB for RDS.
   Alibaba Cloud Account	If the EMR Serverless StarRocks instance and DataWorks belong to the same account, select Current Alibaba Cloud Account. If the EMR Serverless StarRocks instance belongs to another Alibaba Cloud account, select Another Alibaba Cloud Account. After you select Another Alibaba Cloud Account, you must also configure the UID of Alibaba Cloud Account and the RAM Role. For more information about configuring the RAM role of the other account, see Cross-account authorization (RDS, Hive, or Kafka).
   Region	Select the region where your EMR Serverless StarRocks instance resides.
   Instance	Select the target Serverless StarRocks instance.
   Database Name	The name of the database that you want to connect to. You can find this name in Metadata Management after you connect to the instance by using EMR StarRocks Manager.
   Username/Password	The username and password of the instance. When you create a StarRocks instance, a default admin user is created. The password is the one that you specified during instance creation.
   Connection Configuration	Test the connectivity between the data source and the selected resource group. A status of Connectable indicates that the connection is successful.

   Public network

   Parameter	Description
   Configuration Mode	Select User-created Data Store with Public IP Addresses.
   Host Address/IP Address	The Public network address of the FE in the EMR Serverless StarRocks instance. The public endpoint is in the fe-c-<instance-id>.starrocks.aliyuncs.com format, and the HTTP port is 8030. You can click Public Whitelist to configure an access whitelist, or click Disable Public Access to deactivate public access.
   Port	The query port of the FE in the EMR Serverless StarRocks instance. The default value is 9030.
   Load URL	The addresses of StarRocks FE nodes used for StreamLoad. Separate multiple addresses with commas. The address format is FE public endpoint:FE HTTP port.
   Database Name	The name of the database that you want to connect to. You can find this name in Metadata Management after you connect to the instance by using EMR StarRocks Manager.
   Username/Password	The username and password of the instance. When you create a StarRocks instance, a default admin user is created. The password is the one that you specified during instance creation.
   Connection Configuration	Test the connectivity between the data source and the selected resource group. A status of Connectable indicates that the connection is successful.

Data Integration

DataWorks supports synchronizing data from various data sources, such as MySQL, Hive, Kafka, OSS, and HDFS, to EMR Serverless StarRocks tables. The following example describes the key steps to batch synchronize data from a MySQL data source to an EMR Serverless StarRocks table.

1. Log on to the DataWorks console. In the target region, click Data Development and O&M > Data Development in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.

2. Create a batch synchronization node. Set the data source to MySQL and the destination to the StarRocks data source.

   In the left-side navigation tree, expand Data Integration, and then choose New Node > Batch Synchronization. In the Network and Resource Configuration step, select a resource group and click Test Connectivity for both the source and destination to verify that both are Connected.

3. After you select the resource group, test the connectivity to the source and destination data sources.

4. Set a scheduling cycle, and then submit and deploy the node to run periodically.

   After you debug the node, click Scheduling in the right-side pane to configure scheduling parameters such as the scheduling cycle and rerun policy, and set the resource group for the task. After the configuration is complete, click Submission and Deploy.

Data Development and scheduling

For EMR Serverless StarRocks tasks that require periodic scheduling, you can create a StarRocks node in Data Studio. In this node, you can write EMR Serverless StarRocks SQL tasks, use the connected StarRocks data source, and set a scheduling cycle for periodic execution. The key steps are as follows:

1. Log on to the DataWorks console. In the target region, click Data Development and O&M > Data Development in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.

2. In Data Studio, create a StarRocks node, select the connected StarRocks data source, and then write your SQL task.

   Right-click the database node in your business flow and choose New Node > StarRocks. In the SQL editor that opens, select the corresponding data source and write your SQL statement. Example: SELECT * FROM testdb.testtb;.

3. To debug an EMR Serverless StarRocks SQL task in Data Studio, select the SQL statements to debug, click the Run button, and then select a resource group for the test run.

4. Set a scheduling cycle, and then submit and deploy the node to run periodically.

   After you debug the node, click Scheduling in the right-side pane to configure scheduling parameters such as the scheduling cycle and rerun policy, and set the resource group for the task. After the configuration is complete, click Submission and Deploy.

Data Analysis

You can use the Data Analysis component of DataWorks to quickly analyze data in EMR Serverless StarRocks tables. The key steps are as follows:

1. Go to the SQL Query page.

   Log on to the DataWorks console. In the target region, click Data Analysis and Service > DataAnalysis in the left-side navigation pane. Click Go to DataAnalysis to open the Data Analysis page. In the left-side navigation pane, click SQL Query.

2. Click the icon in the left-side pane, click More > System Management to go to the System Management page, and then set the query resource group for StarRocks to the one used by your task.

   In the SQL Query > Resource Group section, the StarRocks engine uses the Public Scheduling Resource Group by default. Change it to your target resource group and click Save.

3. Return to the SQL Query page. In the upper-right corner, switch the engine type to StarRocks and select your data source. You can now write and run queries to perform data analysis on EMR Serverless StarRocks.

   In the configuration panel on the right, you must also select the Workspace and then click OK.

DataService Studio

DataService Studio supports generating APIs from StarRocks data sources.

1. Log on to the DataWorks console. In the target region, click Data Analysis and Service > DataService Studio in the left-side navigation pane. Select a workspace from the drop-down list and click Go to DataService Studio.

2. Generate an API and configure its parameters.

   DataService Studio supports both wizard mode and script mode. Script mode can automatically generate API request and response parameters based on an SQL query. The following steps demonstrate how to use wizard mode.

   In the left-side navigation tree, expand the API node under the target business flow, right-click and choose New API > Generate API to go to the wizard mode configuration page.

   Set the data source type to StarRocks, select the StarRocks data source that you created, select the corresponding table, and then configure the API request parameters, response parameters, and other settings as prompted on the page.

3. In the right-side pane, click Service Resource Group and set the resource group to an exclusive resource group for data services.

   In the Exclusive Service Resource Group drop-down list, select the target resource group, such as test. Set API Gateway Instance Type to Shared Instance, Timeout to 30000 ms, and Maximum Number of Data Rows per Request to 10000.

4. After you successfully test the API, submit and publish it.

Data Map

The Data Map component supports metadata collection, search, and viewing table details for StarRocks data.