Develop an extension with a self-managed service

Prerequisites

You have enabled event message subscription. Deploying an extension using a self-managed service relies on the message distribution capability of EventBridge. Ensure that you have specified an event bus in EventBridge to receive open event messages from DataWorks, and that messages from this event bus are routed to your service that is deployed on-premises or in the cloud.

Limitations

• Only users of DataWorks Enterprise Edition can use the Extensions module.

• The Extensions module is available in the following regions: China (Beijing), China (Hangzhou), China (Shanghai), China (Zhangjiakou), China (Shenzhen), China (Chengdu), US (Silicon Valley), US (Virginia), Germany (Frankfurt), Japan (Tokyo), China (Hong Kong), and Singapore.

Usage notes

• Only the Open Platform administrator, tenant administrator, Alibaba Cloud accounts, and RAM users to which the AliyunDataWorksFullAccess policy is attached have read and write permissions on the developer backend. For more information about permission management, see Permission control for global services and Manage product-level and console access with RAM policies.

• Version limitations: If your DataWorks Enterprise Edition subscription expires, all extensions become inactive and can no longer be triggered for event checks. Any checks that have been triggered but have not yet reached a final state are automatically passed.

• Node limitations: When a composite node that contains inner nodes, such as a Platform for AI node, a do-while node, or a for-each node, triggers a check, all inner nodes must pass the check before subsequent operations can proceed.

• Triggering: You can associate multiple extensions with the same extension point event. This means a single event can trigger multiple extensions.

Step 1: Configure dependencies

When you develop an extension, add the following dependencies to the pom.xml file. EventBridge supports various types of endpoints to process and consume events. In addition to the following dependencies, you may need to add other dependencies based on the event target that you configure in EventBridge.

<dependency>
 <groupId>com.aliyun</groupId>
 <artifactId>dataworks_public20200518</artifactId>
 <version>5.6.0</version>
</dependency>

<build>
        <plugins>
              <plugin>
                  <groupId>org.apache.maven.plugins</groupId>
                  <artifactId>maven-shade-plugin</artifactId>
                  <version>3.2.1</version>
                  <executions>
                    <execution>
                      <phase>package</phase>
                      <goals>
                        <goal>shade</goal>
                      </goals>
                      <configuration>
                        <filters>
                          <filter>
                            <artifact>*:*</artifact>
                            <excludes>
                              <exclude>META-INF/*.SF</exclude>
                              <exclude>META-INF/*.DSA</exclude>
                              <exclude>META-INF/*.RSA</exclude>
                            </excludes>
                          </filter>
                        </filters>
                      </configuration>
                    </execution>
                  </executions>
              </plugin>
        </plugins>
</build>

Step 2: Develop the extension code

The event bus in EventBridge routes messages to your deployed service. This service receives the DataWorks messages from the event bus, processes them, and then calls a specific DataWorks API to return the processing result.

1. Develop the service code.

   Parse message content

   For information about the format of event messages that DataWorks sends, see Appendix: Format of messages sent from DataWorks to EventBridge. In a message, the data field contains the specific message content. During development, you can identify the message type using the data.eventCode field and retrieve message details using the id field.

   Note

   OpenEvent uses EventBridge to distribute DataWorks event messages. Before you develop an extension, you must subscribe to DataWorks messages in EventBridge. For more information, see Enable event message subscription.

   Write the processing logic

   You must process the messages from the event bus based on your business requirements. You can use the following methods to improve development efficiency and application performance:

   • Use Advanced feature: Configure parameters for an extension. For example, you can use the extension.project.disabled parameter to disable the extension for a specific workspace.

   • When you process extension points related to the DataStudio module, call the GetIDEEventDetail operation and use the MessageId to obtain a data snapshot from when the extension point event was triggered.

   Note

   The MessageId parameter corresponds to the id field in the event message. For more information, see Appendix: Format of messages sent from DataWorks to EventBridge.

   Return result to DataWorks

   Your service must return the processing result to DataWorks by calling an OpenAPI operation. The specific API operation that you call depends on the module in which the extension point event originated.

   • Extension point events in DataStudio: Call the UpdateIDEEventResult operation to return the processing result.

   • Extension point events in Operation Center: Call the UpdateWorkbenchEventResult operation to return the processing result.

   • Extension point events in other modules: Call the CallbackExtension operation to return the processing result.

   The callback API request must include the following parameters: the extension to which the service applies (ExtensionCode), the event message (MessageId), and the processing result (CheckResult).

   • CheckResult: The processing result. Valid values:

     • OK: The extension check passed for this extension point event.

     • FAIL: The extension check failed for this extension point event. You must investigate and resolve the error to ensure that subsequent processes run as expected.

     • WARN: The extension check passed for this extension point event, but a warning was generated.

   • ExtensionCode: After you register the extension, you can obtain this code from the extension list page in DataWorks.

   • MessageId: Corresponds to the id field in the message. For more information, see Appendix: Format of messages sent from DataWorks to EventBridge.

2. After you finish developing the code, package the application into an executable .jar file for deployment.

Step 3: Deploy the extension

After you develop and debug the extension code, you must deploy the packaged code as an application service on an ECS instance or a server from another provider.

Step 1: Register the extension

To develop and use an extension, you must first register it in DataWorks to obtain the ExtensionCode that is required for subsequent development. Perform the following steps:

1. Go to the Developer Backend tab.

   Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose More > Open Platform. On the page that appears, click Go to Open Platform. The Developer Backend tab appears.

2. Register the extension.

   1. In the left-side navigation pane, click Extensions to open the extension page.

   2. Click List of Extensions > Register Extension, select Deploy Based on Self-managed Service, and configure the extension information.

      Parameter description

      Parameter	Description
      Extension Name	A custom name to identify the extension.
      Processed Extension Points	Select the extension point events that this extension handles. For a list of supported extension points, see List of supported extension point events. You can select the extension events based on your business requirements. Note After you select an extension point, the Event and Applicable Service fields are automatically populated. Extension point event limit: Extension point events are either tenant-level or workspace-level. You can select only one type of event when you register an extension. For more information about event levels supported by DataWorks, see Developer reference: Event list and message format. Extensions deployed using Function Compute currently support only the data download pre-event.
      Owner	The owner of the extension. Users can contact the owner if they encounter issues.
      Workspace for Testing	Select a workspace to test the extension. The extension is active in the test workspace before it is published. Before you publish the extension, you can perform end-to-end tests in the test workspace. You can trigger an event to test whether DataWorks can send messages using EventBridge and whether the extension can receive, check, and send callback messages. Note If you select a tenant-level extension point event for Processed Extension Points, you do not need to configure Workspace for Testing.
      Extension Details Address	Enter the URL of a page that provides details about the extension. This helps users better understand and use the extension. You can develop a page that displays the details of the extension and configure the URL of the page for this parameter. This way, when the extension triggers a check, users can click the link to view the complete check process, including the check path and the reason for the block.
      Extension Documentation Address	Enter the URL of the help document for the extension. This document is for extension users. You can develop a help document page for the extension and configure the URL of the page for this parameter. This allows users to learn about the check logic and properties of the extension.
      Parameters for Extension	DataWorks allows you to use parameters to improve the development and application efficiency of extensions. When you develop the extension code, you can add the parameters that you want to apply to this section. You can use the built-in parameters that DataWorks provides for typical scenarios or define custom parameters. You can add multiple parameters. Add one parameter per line in the key=value format. Note For example, you can use the built-in extension parameter extension.project.disabled to disable the extension for a specific workspace. For more information about how to use parameters, see Advanced feature: Configure parameters for an extension.
      Options for Extension	Enter the configuration options for extension users. This allows for custom control over the extension in different workspaces. Extension developers must define options as a JSON string on this page. For example, you can configure options to allow extension users to control the SQL length. For information about the JSON format, see Advanced feature: Configure options for an extension.

3. Complete the extension registration.

   Click OK to complete the registration.

   Note

   After the registration is complete, you can view the registered extension on the List of Extensions page.

Step 2: Publish the extension

After you develop, deploy, and register the extension in DataWorks, you must test, approve, and publish it. Once published, administrators other than the extension owner can enable the extension in Management Center. For more information, see Apply an extension.

{
    "datacontenttype": "application/json;charset=utf-8", // The format of the content in the data field. The datacontenttype parameter supports only the application/json format.
    "aliyunaccountid": "1111",// The ID of the Alibaba Cloud account.
    "aliyunpublishtime": "2024-07-10T07:25:34.915Z",// The time when EventBridge received the event.
    "data": {
              
             "tenantId": 28378****10656,// The ID of the tenant. In DataWorks, each Alibaba Cloud account has a unique tenant ID. To find your ID, go to the DataStudio page and click the user information icon in the upper-right corner.
             "eventCode": "xxxx"//
            
    },
    "aliyunoriginalaccountid": "11111",
    "specversion": "1.0",
    "aliyuneventbusname": "default",// The name of the event bus in EventBridge that is used to receive event messages from DataWorks.
    "id": "45ef4dewdwe1-7c35-447a-bd93-fab****",// The unique ID of the event.
    "source": "acs.dataworks",// The event source. `acs.dataworks` indicates that DataWorks pushed the message.
    "time": "2024-07-10T15:25:34.897Z",// The time when the event was generated.
    "aliyunregionid": "cn-shanghai",// The region where the event was received.
    "type": "dataworks:ResourcesUpload:UploadDataToTable"// The event type. You can use this parameter in the EventBridge console to filter messages from DataWorks. The value of this parameter varies by event.
}