How to create and deploy packages for cross-workspace deployment-DataWorks(DataWorks)-阿里云帮助中心

DataWorks allows you to add multiple objects, such as nodes, functions, resources, and components, to a deployment package and deploy them in a single batch. This topic describes how to create and deploy a deployment package for cross-workspace deployment.

Use case

Cross-workspace deployment is primarily used to deploy objects such as nodes, functions, resources, and components from a workspace in basic mode to another workspace within the same region and under the same Alibaba Cloud account. You can also use this feature to isolate the development and production environments for workspaces in basic mode. For more information, see Appendix: How to isolate development from production in basic mode.

Note

To ensure that tasks run correctly in the destination workspace after deployment, read Publishing change logic carefully before you proceed. Make sure that you fully understand the mapping logic between the source and destination workspaces.

Limitations

Permissions

Create a deployment environment: Only members with the Workspace Administrator role can create and configure a deployment environment.
Deploy tasks to the production environment of the source workspace: Members with the Operator, Deployer, or Workspace Administrator role can deploy tasks to the source workspace's production environment in Data Studio.
Create a deployment package: Members with the Development role can create a deployment package in Deployment Center.
Deploy a deployment package: Members with the Operator, Deployer, or Workspace Administrator role in both the source and destination workspaces can deploy a deployment package.

For more information about adding members to a workspace and granting permissions, see Manage permissions on workspace-level services.

Environment limitations

The following table describes the environment requirements for deploying objects from a source workspace to a destination workspace.

Source workspace		Destination workspace
Workspace mode	Data Studio version	Basic mode New Data Studio	Basic mode Legacy Data Studio	Standard mode New Data Studio	Standard mode Legacy Data Studio
Basic mode	New Data Studio	Supported	Not supported	Supported	Not supported
Basic mode	Legacy Data Studio	Not supported	Supported	Not supported	Supported
Standard mode	-	Not supported	Not supported	Not supported	Not supported

Other limitations

Cross-workspace Deployment is used to deploy objects across different workspaces within the same Alibaba Cloud account and the same region. Cross-region deployment is not supported. If you need to deploy objects across regions, use Deploy Center.

Deployment process

Procedure

All operations for cross-workspace deployment are performed in the source workspace.

Step 1: Create a deployment environment

When you deploy nodes from the source workspace to the destination workspace, the destination is a completely new deployment environment. Therefore, you must map information such as engine instances, resource groups, and dependencies from the source to the destination. After the mapping is configured, the system automatically replaces the environment used by the deployed nodes with the mapped environment information when you deploy a deployment package, without requiring additional configuration in the destination workspace.

Note

Only members with the Workspace Administrator role can create and configure a deployment environment.

1. Go to Deployment Center

Log on to the DataWorks console. In the target region, click More > Management Center in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Management Center.
In the upper-left corner of the page, click the icon and choose All Products > More > Deployment Center to go to Deployment Center.

2. Configure the deployment environment

On the Deployment Center page, click Deployment Environment Management in the left-side navigation pane.

Click Create Deployment Environment and configure the environment information.

Configure the basic environment information.

Parameter	Description
Environment Name	The name of the new deployment environment.
Deployment Type	In this scenario, select Cross-workspace Deployment.
Destination workspace	Select the destination workspace for cross-workspace deployment.
Deployment Object Owner	Specifies the owner of each node in the deployment package after it is deployed to the destination workspace. Valid values: Default: DataWorks preferentially sets the node owner in the source workspace as the primary owner. If the owner is not a member of the destination workspace, the system sets the deployment package executor (the user who deploys the deployment package) as the node owner. Deployment Package Creator: DataWorks sets the user who created the deployment package as the node owner. Deployment Package Executor: DataWorks sets the user who performs the deployment operation (deploys the target deployment package) as the node owner.

Configure Computing Resource Mappings.

Configure the mapping between compute resources in the source workspace and compute resources in the destination workspace. After the mapping is configured, the system deploys the nodes, resources, functions, components, and other related content from the source compute resource to the mapped destination compute resource when you deploy tasks.

Note

At least one Computing Resource in Source Workspace must be retained in the Computing Resource Mappings section, and a mapping to the destination must be configured for it.
Determine whether you need to map source compute resources in the destination workspace based on your actual business requirements.

Parameter	Description
Computing Resource in Source Workspace	Displays all compute resources associated with the source workspace.
Computing Resource Type	The type of compute resource associated with the source workspace.
Computing Resource in Destination Workspace	Select the destination compute resource to which the source workspace compute resource is mapped. You can go to the Deployment Center > Deployment Environment Management page of the destination workspace and click View Workspace Environment Information in the upper-right corner to view compute resource information.
Ignore Mapping	After this feature is enabled, the destination workspace does not map the corresponding source workspace compute resource, and the nodes, resources, functions, components, and other content under the source compute resource will not be packaged for deployment. When the nodes, resources, functions, components, and other content under an engine instance in the source workspace do not need to be deployed, you can enable Ignore Mapping for the engine instance.

Configure Resource Group Mappings.

Configure the mapping between resource groups in the source workspace and resource groups in the destination workspace. After the mapping is configured, the system uses the mapped destination resource group instead of the source resource group to run tasks during deployment.

Note

DataWorks supports mapping only between resource groups of the same type.
When you map Data Integration resource groups, make sure that the allowlist, network connectivity, and other configurations of the destination Data Integration resource group are consistent with those of the source Data Integration resource group. Otherwise, Data Integration tasks may not run properly. For more information about network connectivity configuration, see Overview.

Parameter	Description
Resource Group in Source Workspace	Displays all resource groups associated with the source workspace.
Resource Group Type	The type of resource group associated with the source workspace.
Resource Group in Destination Workspace	Select the destination resource group to which the source workspace resource group is mapped. You can go to the Deployment Center > Deployment Environment Management page of the destination workspace and click View Workspace Environment Information in the upper-right corner to view resource group information.
Ignore Mapping	After this feature is enabled, the destination workspace does not map the corresponding source workspace resource group, and the source workspace resource group is still used when tasks are deployed. Note Default resource groups and shared resource groups do not support the Ignore Mapping feature. When the source workspace has more resource groups than the destination workspace, you can enable Ignore Mapping for the extra resource groups. After this feature is enabled, the source resource group is still used to run tasks during deployment.

Configure Dependency Mappings.

This section displays the names of all upstream projects that have cross-project dependencies in the source workspace. You can configure the target project to which each project is mapped. After the mapping is configured, the system sets the target project as the upstream dependency of the deployed nodes in the destination workspace based on the mapping, generating new upstream and downstream dependency relationships.

Note

The Dependency Mappings section displays dependencies that are automatically updated on a T+1 basis by default. To view real-time dependencies, click Refresh Dependencies..
If you configure a cross-project dependency mapping to a new project, DataWorks modifies the node dependency based on the mapping. Make sure that the mapped workspace supports adding dependencies properly. For common scenarios where dependencies cannot be added, see Node commit error: Dependent ancestor output name not found.
After dependency mappings are configured for MaxCompute engine tasks (such as ODPS SQL and ODPS Spark), the system also updates the task code accordingly.

Parameter	Description
Source Project Name	Displays the names of all projects with cross-project dependencies in the source workspace.
Property	The project category.
Destination Project Name	Select the target project to which the source project is mapped.
Ignore Mapping	After this feature is enabled, the system ignores the dependency mapping for this project during deployment and continues to use the source project as the cross-project dependency project. The upstream and downstream dependency relationships of the deployed nodes remain unchanged.

Configure Data Source Mappings.

Configure the mapping between data sources in the source workspace and data sources in the destination workspace. After the mapping is configured, the data sources used in the deployed tasks are replaced with the mapped destination data sources during deployment.

Parameter	Description
Source Name	Displays all data sources in the source workspace.
Data Source Type	The data source type.
Destination Name	Select the destination data source to which the source data source is mapped.
Ignore Mapping	After this feature is enabled, the system ignores this data source during deployment and does not map it to the destination workspace.

Configure Parameter Mappings.

Synchronize the parameter configurations from the source workspace to the destination workspace. During deployment, the system references the parameters configured here (including workflow parameters and scheduling parameters) to the destination workspace. You can define parameter names and values by using the visual interface or expressions. For more information about parameter configuration, see Scheduling parameter formats.

Note
Make sure that the parameter names are unique in each deployment. Avoid assigning the same parameter to different tasks.
Click Create to create the deployment environment.

Step 2: Deploy tasks to the production environment of the source workspace

Taking a workspace in basic mode that uses Data Studio (new version) as an example, this step requires you to deploy tasks to the production environment of the current workspace (source workspace). After deployment, the system pushes the built deployment package to Nodes to Deploy in Deployment Center for subsequent cross-workspace deployment.

Note

Only members with the Operator, Deployer, or Workspace Administrator role can perform deployment operations.
When using a workspace in basic mode with DataStudio (legacy), find the nodes, functions, and other objects to deploy in Data Studio, and click Commit () on the top toolbar.

In Data Studio, find the nodes, functions, and other objects to deploy, and click Deploy on the top toolbar to start the deployment process.
In the deployment process, click Start Production Deployment to deploy the objects to the production environment of the source workspace.

The deployment process consists of three stages: Package Build, Production Checker, and Deploy to Production. After each stage is completed, the status is displayed as Completed. You can click View Details to view the execution information.

Step 3: Create a deployment package

Note

Members with the Development role can create a deployment package in Deployment Center. If you have the Workspace Administrator role, you do not need to manually create a deployment package. After you select the objects to deploy, you can directly deploy them in a batch, and the system automatically generates a deployment package.

Use a member with the Development role to go to Deployment Center of the source workspace.

You can go to any DataWorks module, click the icon in the upper-left corner of the page, and then choose All Products > More > Deployment Center to go to Deployment Center.
In the left-side navigation pane, click Nodes to Deploy to view the generated objects to deploy. In the Actions column, click Add to Deployment Package for the desired objects. After you add objects, the number of objects to deploy is displayed on the View Deployment Package button.
Note
- You can also select multiple objects to deploy and click Batch Add to Deployment Package at the bottom of the page.
- When using new Data Studio, if you clicked Terminate Deployment in the deployment process in Data Studio, the corresponding objects to deploy are no longer displayed here.
After adding objects to the deployment package, click View Deployment Package to verify whether the objects in the deployment package meet your expectations. You can click Remove to remove unwanted objects from the deployment package.

Note
Currently, only objects from legacy Data Studio can be viewed. Objects from new Data Studio cannot be viewed yet.
On the Pending Deployment page, click Deployment Package Creation and follow the on-screen instructions to create the deployment package.

Note
If you have the Workspace Administrator role, the Deployment Package Creation button is not displayed. Instead, the Deploy All button is displayed, allowing you to directly deploy the objects.

Step 4: Deploy the deployment package

Note

The operator in this step must have the Operator, Deployer, or Workspace Administrator role in both the source and destination workspaces.

Use a member with the Operator, Deployer, or Workspace Administrator role to go to Deployment Center of the source workspace.

You can go to any DataWorks module, click the icon in the upper-left corner of the page, and then choose All Products > More > Deployment Center to go to Deployment Center.
In the left-side navigation pane, click Deployment Packages to view the generated deployment packages.
In the Actions column, click Deploy for the target deployment package and follow the on-screen instructions to complete the deployment.
Note
- During cross-workspace deployment, members with the Operator, Deployer, or Workspace Administrator role can edit the resource group used by the deployed tasks in the destination workspace.
- For nodes that generate instances on the next day (T+1), the deployment takes effect as follows:
  - If the deployment is performed before 22:00, the deployment takes effect the next day.
  - If the deployment is performed after 22:00, the deployment takes effect on the third day.
- For nodes that generate instances immediately, the deployment takes effect as follows:
  - Instances whose scheduled time is more than 10 minutes after the deployment time are normally generated or updated.
  - Instances whose scheduled time is within 10 minutes of the deployment time, or instances deployed after 22:00, take effect on the third day.

Step 5: View the deployment result

After the deployment package is created and deployed, you can view the deployment result on the Deployment Packages page of the current workspace.

On the Deployment Packages page, verify that the Deployment Status column displays Deployed, which indicates that the cross-workspace deployment is complete.
After the deployment succeeds, you can go to Operation Center of the destination workspace to view the deployed tasks.