Create and use CDH resources and functions-DataWorks(DataWorks)-阿里云帮助中心

Prerequisites

A CDH cluster is registered with DataWorks. All resource and function operations require CDH compute resources.
Your resource files are ready to upload from your local machine.

Access resource management

Go to the Workspaces page in the DataWorks console. In the top navigation bar, select a desired region. Find the desired workspace and choose Shortcuts > Data Studio in the Actions column.
In the left-side navigation pane, click the Resource Management icon to go to the Resource Management page.
On the Resource Management page, click the icon to create a resource or function. Alternatively, you can first create a directory to organize your resources by clicking Create Directory. Then, right-click the target directory, select Create, and choose the type of resource or function that you want to create.

Create and use resources

Resources

In Data Studio, you can upload local resources to a CDH cluster through DataWorks. The following table lists the supported resource types.

Resource type	Description	Supported upload methods
		Local	OSS
CDH Jar	A compiled Java JAR package for running Java programs. The file extension is `.jar`.
CDH File	Any file type uploaded as a CDH File resource. Its use depends on the compute engine.

Limitations

The following limitations apply to resource uploads:

Resource size:
Resource publishing: If you use a standard mode workspace, you must publish the resource to the production environment before you can use it.

Note
Data source configurations may differ between the development and production environments. Before you query tables or use resources, verify the data source configuration for the current environment.
Resource management: In DataWorks, you can view and manage only the resources that are uploaded through the DataWorks UI.

Create a resource

Upload CDH resources from your local machine. After creating a resource, you can reference it in data development or register it as a custom function.

On the Resource Management page, create a resource, which opens the Create Resource and Function dialog box. Configure the resource Type, storage Path, and resource Name.

After you create the resource entry, upload a local file. The following table describes the key upload parameters:

Parameter	Description
Storage Path	Default path: `/user/admin/lib`. Note If Kerberos authentication is enabled, you must first grant the current user write permissions to this directory.
Data Sources	Select an existing CDH data source.
Resource Group	Select a Serverless resource group that can connect to the CDH cluster.

In the top toolbar, Save and then Publish the resource. You can use only published resources in data development.

Use a resource

After creating a resource, you can reference it in data development. In the left-side navigation pane, click Resource Management, find the target resource or function, right-click it, and select Insert Resource Path. This inserts a code snippet in the format ##@resource_reference{"Resource Name"} into your editor.

Note

For example, in a CDH Hive node, the reference might look like ##@resource_reference{"example"}. The format may vary by node type. Refer to the actual UI for the correct format.

You can also create a function from a resource and use it in your development nodes.

Create and use functions

Functions

Data Studio allows you to register resources as CDH functions. In data development or SQL queries, you can use both Hive built-in functions and custom functions that you create.

Create a function

On the Resource Management page, create a function, which opens the Create Resource and Function dialog box. Configure the function Type, storage Path, and function Name.

Click Confirm to create the function. Then, configure the function details based on its type.

Before configuring a CDH function, ensure that you have registered the CDH cluster as a compute resource in DataWorks and uploaded the required CDH resource. The following table describes the key parameters for a CDH function.

Parameter	Description
Function type	Select a function type: MATH (mathematical), AGGREGATE (aggregation), STRING (string manipulation), DATE (date), ANALYTIC (analytic), or OTHER (other).
Data Sources	Select an existing CDH data source.
Class Name	The class name for the user-defined function (UDF), in the format `ResourceName.ClassName`. The resource name can be a Java package name or a file resource name. When creating a custom function in DataWorks, you can use either JAR or File type CDH resources. If the resource type is JAR, the Class Name format is `PackageName.ActualClassName`. You can get this value from `IntelliJ IDEA` by using the `Copy Reference` command. For example, if the package name is `com.aliyun.cdh.examples.udf` and the actual class name is `UDAFExample`, set the Class Name parameter to `com.aliyun.cdh.examples.udf.UDAFExample`. Note Do not include the `.jar` suffix in the resource name. The resource must be published before you can use it.
Resource List	For a CDH function, only visual mode is supported, which requires selecting a CDH Jar or CDH File resource.
Command Format	A usage example for the UDF.

In the top toolbar, Save and then Publish the function. You can use only published functions in data development.

Use a function

After creating and publishing a function, you can reference it in data development or SQL queries.

When you edit a data development node, click Resource Management in the left-side navigation pane. Find the target function, right-click it, and select Insert Function.

The function name, such as example_function(), is automatically inserted into the editor.
When you edit a SQL query, you can use the created function directly in your SQL statement.

SELECT example_function(column_name) FROM table;

Manage resources and functions

After you upload resources or create functions through the Data Studio visual interface, you can manage them on the resource management page by clicking the target resource or function.

View version history: Click the version button on the right side of the resource or function editor page to view and compare saved or submitted function versions and see changes between different versions.

Note
For version comparison, you must select at least two versions.
Delete a resource or function: Right-click the target resource or function and click the Delete button to delete it.

To delete the resource or function in the production environment, you must deploy the task and deploy the deletion to the production environment. After the deployment succeeds, the resource or function is synchronously deleted from the production environment.