Develop and deploy extensions: Function Compute

Limitations

• Only users of DataWorks Enterprise Edition can use the Extensions module.

• The Extensions module is available in the following regions: China (Beijing), China (Hangzhou), China (Shanghai), China (Zhangjiakou), China (Shenzhen), China (Chengdu), US (Silicon Valley), US (Virginia), Germany (Frankfurt), Japan (Tokyo), China (Hong Kong), and Singapore.

Notes

• Only the Open Platform administrator, tenant administrator, Alibaba Cloud accounts, and RAM users to which the AliyunDataWorksFullAccess policy is attached have read and write permissions on the developer backend. For more information about permission management, see Permission control for global services and RAM policies for service and console permissions.

• Version limitations: If your DataWorks Enterprise Edition subscription expires, all extensions become inactive and can no longer be triggered for event checks. Any checks that have been triggered but have not yet reached a final state are automatically passed.

• Node limitations: When a composite node that contains inner nodes, such as a Platform for AI node, a do-while node, or a for-each node, triggers a check, all inner nodes must pass the check before subsequent operations can proceed.

• Triggering: You can associate multiple extensions with the same extension point event. This means a single event can trigger multiple extensions.

• Event limitations: Extensions deployed using Function Compute currently support pre-events for data download, pre-events for asset publishing and unpublishing, and pre-events for data upload.

Step 1: Configure extension dependencies

When you develop your extension, add the following dependencies to the pom.xml file.

<dependency>
 <groupId>com.aliyun</groupId>
 <artifactId>dataworks_public20200518</artifactId>
 <version>5.6.0</version>
</dependency>

<!-- https://mvnrepository.com/artifact/com.aliyun.fc.runtime/fc-java-core -->
<dependency>
    <groupId>com.aliyun.fc.runtime</groupId>
    <artifactId>fc-java-core</artifactId>
    <version>1.4.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.aliyun.fc.runtime/fc-java-event -->
<dependency>
    <groupId>com.aliyun.fc.runtime</groupId>
    <artifactId>fc-java-event</artifactId>
    <version>1.2.0</version>
</dependency>

<build>
        <plugins>
              <plugin>
                  <groupId>org.apache.maven.plugins</groupId>
                  <artifactId>maven-shade-plugin</artifactId>
                  <version>3.2.1</version>
                  <executions>
                    <execution>
                      <phase>package</phase>
                      <goals>
                        <goal>shade</goal>
                      </goals>
                      <configuration>
                        <filters>
                          <filter>
                            <artifact>*:*</artifact>
                            <excludes>
                              <exclude>META-INF/*.SF</exclude>
                              <exclude>META-INF/*.DSA</exclude>
                              <exclude>META-INF/*.RSA</exclude>
                            </excludes>
                          </filter>
                        </filters>
                      </configuration>
                    </execution>
                  </executions>
              </plugin>
        </plugins>
</build>

Step 2: Develop the extension code

To develop an extension in Function Compute, implement the PojoRequestHandler interface's handleRequest method. This method receives ExtensionRequest objects and the Context, then returns the processing result using the ExtensionResponse class.

1. Consider the following points when developing the code.

   Parse message content

   For information about the format of event messages pushed by DataWorks, see Developer reference: Event list and message format. In the message format, messageBody contains the specific content of the message. During development, you can use the messageBody.eventCode field to identify the message type and the messageId field to get the message details.

   Processing logic

   Write the message processing logic based on your business requirements. The following methods can improve efficiency:

   • Use Advanced feature: Configure parameters for an extension. For example, you can use the extension.project.disabled parameter to disable the extension for a specific workspace.

   • When you process extension points related to the DataStudio module, call the GetIDEEventDetail operation and use the MessageId to obtain a data snapshot from when the extension point event was triggered.

   Note

   The MessageId parameter corresponds to the id field in the event message. For more information, see Appendix: Format of messages sent from DataWorks to EventBridge.

   Processing result

   When you package the ExtensionResponse result, specify a CheckResult. DataWorks reads the final CheckResult to determine whether the process fails.

   • OK: The extension check passed for this extension point event.

   • FAIL: The extension check failed for this extension point event. You must investigate and resolve the error to ensure that subsequent processes run as expected.

   • WARN: The extension check passed for this extension point event, but a warning was generated.

   You can download the sample code file fc_dataworks_demo01-1.0-SNAPSHOT.jar and upload it to Function Compute for verification. The following code provides more details:

   Code details

   App.java

   Implements the PojoRequestHandler interface and its handleRequest method to receive ExtensionRequest objects and the Context, define the processing logic, and return the result as an ExtensionResponse object.

   This code sample prohibits manually uploading data to ADS tables.

   package com.aliyun.example;
   import com.alibaba.fastjson.JSON;
   import com.alibaba.fastjson.JSONObject;
   import com.aliyun.dataworks.ExtensionRequest;
   import com.aliyun.dataworks.ExtensionResponse;
   import com.aliyun.fc.runtime.Context;
   import com.aliyun.fc.runtime.PojoRequestHandler;
   public class App implements PojoRequestHandler<ExtensionRequest, ExtensionResponse> {
       public ExtensionResponse handleRequest(ExtensionRequest extensionRequest, Context context) {
           // Print the request content for debugging.
            System.out.println(JSON.toJSONString(extensionRequest));
           // Create a response object.
           ExtensionResponse extensionResponse = new ExtensionResponse();
           // Check if eventType is 'upload-data-to-table'.
           if ("upload-data-to-table".equals(extensionRequest.getEventType())) {
               try {
                   // Convert messageBody to a string and then parse it into a JSONObject.
                   String messageBodyStr = JSON.toJSONString(extensionRequest.getMessageBody());
                   JSONObject messageBody = JSON.parseObject(messageBodyStr);
                   String tableGuid = messageBody.getString("tableGuid");
                   // Check if tableGuid contains 'ads'.
                   if (tableGuid != null && tableGuid.contains("ads")) {
                       extensionResponse.setCheckResult("FAIL");
                   } else {
                       extensionResponse.setCheckResult("OK");
                   }
               } catch (Exception e) {
                   extensionResponse.setCheckResult("FAIL");
                   extensionResponse.setErrorMessage("Error processing request: " + e.getMessage());
                   return extensionResponse;
               }
           } else {
               extensionResponse.setCheckResult("FAIL");
           }
           // Set an error message for feedback.
           extensionResponse.setErrorMessage("This is a test!");
           // Return the response object.
           return extensionResponse;
       }
   }
   

   ExtensionRequest.java

   Defines the request structure for the extension, encapsulating event messages sent by DataWorks.

   public class ExtensionRequest {
       private Object messageBody;
       private String messageId;
       private String extensionBizId;
       private String extensionBizName;
       private String eventType;
       private String eventCategoryType;
       private Boolean blockBusiness;
       public ExtensionRequest() {
       }
       public Object getMessageBody() {
           return this.messageBody;
       }
       public void setMessageBody(Object messageBody) {
           this.messageBody = messageBody;
       }
       public String getMessageId() {
           return this.messageId;
       }
       public void setMessageId(String messageId) {
           this.messageId = messageId;
       }
       public String getExtensionBizId() {
           return this.extensionBizId;
       }
       public void setExtensionBizId(String extensionBizId) {
           this.extensionBizId = extensionBizId;
       }
       public String getExtensionBizName() {
           return this.extensionBizName;
       }
       public void setExtensionBizName(String extensionBizName) {
           this.extensionBizName = extensionBizName;
       }
       public String getEventType() {
           return this.eventType;
       }
       public void setEventType(String eventType) {
           this.eventType = eventType;
       }
       public String getEventCategoryType() {
           return this.eventCategoryType;
       }
       public void setEventCategoryType(String eventCategoryType) {
           this.eventCategoryType = eventCategoryType;
       }
       public Boolean getBlockBusiness() {
           return this.blockBusiness;
       }
       public void setBlockBusiness(Boolean blockBusiness) {
           this.blockBusiness = blockBusiness;
       }
   }

   ExtensionResponse.java

   Defines the response structure for the extension, encapsulating the processing result.

   public class ExtensionResponse {
       private String checkResult;
       private String errorMessage;
       public ExtensionResponse() {
       }
       public String getCheckResult() {
           return this.checkResult;
       }
       public void setCheckResult(String checkResult) {
           this.checkResult = checkResult;
       }
       public String getErrorMessage() {
           return this.errorMessage;
       }
       public void setErrorMessage(String errorMessage) {
           this.errorMessage = errorMessage;
       }
   }

2. After developing the code, package the program into a runnable .jar or .zip file for Function Compute.

   Open a command-line window in your code editor, navigate to the root directory, and run the mvn clean package command to package the code.

   • If the compilation fails, adjust the code based on the error messages in the output.

   • If the compilation is successful, the compiled JAR file is located in the target directory of the project. The file is named based on the artifactId and version fields in the pom.xml file, for example, java-example-1.0-SNAPSHOT.jar.

   Note

   On macOS and Linux, make sure the code files have read and execute permissions before packaging.

Step 1: Deploy the extension

1. Log on to the Function Compute console 2.0 and go to the Tasks page.

2. Click Create Function to create a service and the required function, and then deploy the extension. Specific event messages from the developed extension are sent directly to this service. For more information, see Create a service and Create a function. In the Create Function panel, select Create with built-in runtimes, select Process event requests for the handler type, select Upload code via JAR package for the code upload method, and then upload the prepared code package.

   • Select a runtime environment based on your code. The sample package provided in Step 1, fc_dataworks_demo01-1.0-SNAPSHOT.jar, corresponds to the Java 8 environment. You must download fc_dataworks_demo01-1.0-SNAPSHOT.jar to your local computer and upload it as the code package. Configure other parameters based on your requirements.

   • For more information about function operations, see Manage functions.

3. Modify the function handler.

   You can configure the handler in the Function Compute console. For Java functions, the handler format is [Package name].[Class name]::[Method name]. For example:

   • Package name: example.

   • Class name: HelloFC.

   • Method name: handleRequest.

   The handler can be configured as example.HelloFC::handleRequest.

   Note

   The default handler in Function Compute is example.App::handleRequest. You must modify the handler based on your actual code. For more information about handlers, see Handlers.

Step 2: Test the function

After you create the function, go to the function details page, click the Code tab, and click Test Function. If the message Execution succeeded is returned, you can proceed to register the extension in DataWorks.

Step 1: Register the extension

After deploying the extension in Function Compute, register it in DataWorks.

1. Log on to the DataWorks console. In the target region, click More > Open Platform in the left-side navigation pane. Click Go to Open Platform to open the Developer Backend page.

2. Register the extension.

   1. In the left-side navigation pane, click Extensions to go to the Extensions page.

   2. Click List of Extensions > Register Extension, select Deploy Based on Function Compute, and configure the extension information.

      You can configure the parameters based on your requirements. The following table describes the parameters.

   Parameters

   Parameter	Description
   Extension Name	A custom name to identify the extension.
   Function Compute Service and Function	Select the Function Compute service and function for the extension. Event messages are sent to this service.
   Processed Extension Points	Only messages triggered by pre-events for data download, pre-events for asset publishing and unpublishing, and pre-events for data upload can be processed. Note After you make a selection, the Event and Applicable Service fields are automatically populated. You do not need to configure them manually.
   Owner	The extension owner, whom users can contact if they encounter issues.
   Workspace for Testing	Select the workspace for testing the extension. The extension runs in the test workspace without being published. Before publishing, perform end-to-end testing in the test workspace. Trigger an event to verify that DataWorks sends a message through EventBridge and that the extension receives, reviews, and provides a callback for the message. Note If you select a tenant-level extension point event for Processed Extension Points, you do not need to configure Workspace for Testing.
   Extension Details Address	Enter the URL of a page that describes the extension's details, helping users understand and use the extension. You can create a details page during development and provide its URL here, allowing users to view the complete validation process when a check is triggered, such as the check flow and reason for a block.
   Extension Documentation Address	Enter the URL of the extension's help documentation. You can create a documentation page during development and provide its URL here, helping users understand the extension's validation logic and properties.
   Parameters for Extension	Use parameters to improve extension development and application efficiency. Add the required parameters here during code development. You can use built-in parameters for common use cases or define your own. You can add multiple parameters, with one parameter per line in the key=value format. For more information about how to use parameters, see Advanced application: Configure extension parameters.
   Options for Extension	Enter the configuration options available to users for personalized control of the extension in different workspaces. Define these options as a JSON string. For example, you can use an option to allow users to control the maximum SQL length. For the JSON format, see Advanced application: Configure extension options.

3. Complete the registration.

   Click OK. After the registration is complete, you can view the extension in the List of Extensions.

Step 2: Publish the extension

After you develop, deploy, and register the extension in DataWorks, you must test, approve, and publish it. Once published, administrators other than the extension owner can enable the extension in Management Center. For more information, see Apply an extension.