Git code synchronization and merge

Step 1: Prepare resources and network

Code synchronization tasks run on a serverless resource group and require network access to your Git repository and OSS.

1. Prepare a serverless resource group and configure the network.

   • Prepare a DataWorks serverless resource group and associate it with the target workspace.

   • Ensure that the resource group can access the SSH port of the Git server. Only port 22 is supported.

     • Public Git repository (for example, Alibaba Cloud DevOps CodeUp or GitHub.com): Configure an Internet NAT Gateway and an EIP for the resource group's VPC to enable public network access.

     • Private Git repository: Ensure that the resource group and the Git server can communicate over the private network (VPC). For more information, see Network connectivity solutions.

2. Prepare an OSS data source. The code synchronization feature uses OSS for temporary storage.

   • Create an OSS bucket in the same region as the DataWorks workspace.

   • On the Data Source Management page of the DataWorks workspace, create an OSS data source that points to the bucket.

     Note

     Make sure that the access credentials used by the OSS data source have the following permissions: oss:GetObject, oss:ListObjects, oss:PutObject, and oss:DeleteObject.

Step 2: Prepare the Git environment

Create a dedicated account on the Git platform, configure SSH keys, and set up branch protection rules.

1. Create a dedicated Git account and generate SSH keys.

   1. Create an account on the Git platform (for example, Alibaba Cloud DevOps CodeUp or GitLab) that is dedicated to automatic code commits (for example, dataworks_pusher), and configure a valid email address.

   2. Generate an SSH key pair in RSA algorithm mode for the account. Do not set a passphrase during the key generation process.

      • Purpose: Generate the SSH public key and private key for authentication.

      • Command:

        # Replace "your_user_email@example.com" with the account email configured in the previous step
        ssh-keygen -t rsa -C "your_user_email@example.com"
        
        # When prompted for a passphrase, press Enter to skip
        # Enter passphrase (empty for no passphrase): [Enter]
        # Enter same passphrase again: [Enter]

      • Result: After the command is executed, the id_rsa (private key) and id_rsa.pub (public key) files are generated in the ~/.ssh/ directory. The contents of these two files will be used in subsequent steps.

        # Copy and save the public and private key contents, which will be needed in later steps
        cat ~/.ssh/id_rsa.pub
        cat ~/.ssh/id_rsa

2. Add the public key on the Git platform.

   Log on to the Git platform and add the public key generated in the previous step (the contents of the id_rsa.pub file) on the SSH key management page of the dataworks_pusher account.

3. Create a code repository and configure branch protection rules.

   1. Create a Git repository dedicated to DataWorks code management, for example, DataWorks_code.

   2. Set up branch protection rules in the repository to prevent direct modifications to branches that are automatically managed by DataWorks. This operation must be performed by a repository administrator.

      Important

      Incorrectly configured branch protection rules may cause branches that are automatically managed by DataWorks to be accidentally modified, which can break synchronization consistency.

      • Branch name pattern: dataworks_*

      • Allowed to push: Select specified members and add the dataworks_pusher account.

      • Allowed to merge: Select No one.

Step 3: Configure and initialize code synchronization

Step 4: Verify code synchronization

1. Create a Shell node for data development and name it shell_test.

2. Enter the following code in the code editor and click the Save button on the toolbar.

   echo 'Code push test.'

3. Log on to Git, go to the target repository, select the save branch, and find the shell_test folder. You can see that the folder contains three files: shell_test.sh (code file), shell_test.spec.json (scheduling configuration file), and dataworks.properties (variable file). Click shell_test.sh to view the shell_test node and its code content, which indicates that the synchronization is successful.

4. To verify the synchronization status of the other two branches, deploy nodes to the corresponding environments. For more information about how to deploy nodes, see Deploy nodes.

Reverse merge

DataWorks supports merging the latest changes from a remote Git branch back into the platform.

This capability works with both the main synchronization branch and feature branches created from it. Developers can work on independent branches, perform code reviews, and then integrate tested code back into DataWorks for efficient team collaboration.

1. Merge entry

   In DataStudio > Code Management, expand the Code Merge section.

2. Merge preview

   In the Branch input box, enter the name of the branch to merge, such as feature_cn-shanghai_branch. Click merge preview. The system compares the additions, modifications, and deletions between the source branch and the default DataWorks save branch.

   • No conflicts: The interface displays a diff of the changes, clearly listing the nodes to be added, modified, or deleted along with the code change details.

   • Conflicts detected: The interface displays the conflicts along with the specific conflicting content. In this case, you need to go back to your local environment, resolve the conflicts manually in Git, and then initiate the merge again.

3. Confirm merge

   1. After verifying that the preview is correct, click Confirm merge.

   2. The system starts the merge task, and you can monitor the merge progress in real time.

   3. All historical merge records are displayed in the code merge history section below. You can view the owner, merge status, and branch details at any time.

Cross-workspace merge

Git synchronization supports cross-workspace and cross-region cloning of DataWorks projects, letting you use a standardized code set as a template that can be reused across business scenarios. For example, you can deploy a common user analysis model to independent workspaces of different business lines, each running on its own compute resources and data sources.

1. Initialization setup

   1. Create two projects: git_cross_project_1 and git_cross_project_2. Cross-region creation is supported.

   2. Follow Step 1 to configure the resource group and OSS data source in each workspace. The two data sources can use the same configuration. Configure the network connectivity as well.

   3. Follow Step 2 to prepare one Git repository as the shared code management repository for both workspaces.

   4. Follow Step 3 to configure Git synchronization for both workspaces (use the same SSH address and private key for both), and make sure the initialization succeeds. At this point, branches for both workspaces, 270256 (project1) and 270257 (project2), appear in the Git repository.

2. Cross-project merge policy

   1. When merging code branches across projects, only node code, basic properties, and scheduling configurations are merged. Runtime configurations are not merged into the target workspace.

   2. Because the compute resources, resource groups, and data sources used by the two workspaces may differ, you need to configure a resource mapping file merge_mapping.

      1. Open the terminal on your local computer, clone the save branch of the source workspace to your local machine, and switch to the save branch of the source workspace project1, such as dataworks_cn_shenzhen_270256_save.

         # Clone the remote Git repository to your local machine
         git clone git@your_git_server_domain:64dc86a16800a4a57137536/cross_project_shenzhen.git
         
         # Switch to the save branch of project1
         git checkout <your branch name>

      2. Create the merge_mapping mapping file.

         # cross_project_shenzhen is the repository name
         cd cross_project_shenzhen 
         
         # Create the directory
         mkdir -p DATAWORKS_SYSTEM_CONFIG/merge_mapping
         
         # Create the mapping file, e.g., cn_shenzhen_270256_to_cn_shenzhen_270257.properties
         vi DATAWORKS_SYSTEM_CONFIG/merge_mapping/<region>_<projectId>_to_<region>_<projectId>.properties

         Configure the file content based on your actual setup. Modify the actual values on both sides of =. Multiple values are supported for each type.

         # Data source
         # spec.datasource.name.<project1 data source name>=<project2 data source name>
         spec.datasource.name.mysql_01=mysql_02
         
         # Resource group
         # spec.runtimeResource.resourceGroup.<project1 resource group ID>=<project2 resource group ID>
         spec.runtimeResource.resourceGroup.group_524257424564736=Serverless_res_group_524257424564736_764027070300961
         
         # Node output name prefix
         # spec.output-prefix.<project1 name>=<project2 name>
         spec.output-prefix.git_cross_project_1=git_cross_project_2
         
         # Project prefix for tables in MaxCompute SQL
         # script.project-identifier.<project1 name>=<project2 name>
         script.project-identifier.git_cross_project_1=git_cross_project_2
         
         # Image
         # spec.script.runtime.container.imageId.<image ID used in project1>=<image ID used in project2>
         spec.script.runtime.container.imageId.Default=System_python311_ubuntu2204_20251201
         
         # RAM role
         # spec.script.runtime.linkedRoleArn.<RAM role ARN used in project1>=<RAM role ARN used in project2>
         spec.script.runtime.linkedRoleArn.acs:ram::1107550004253538:role/aliyundataworksaccessingenirole=acs:ram::1107550004253538:role/aliyundataworksaccessingossrole

         Push the code to the Git repository.

         # Stage current directory changes in git
         git add .
         # Commit the mapping file
         git commit -m "add mapping files"
         # Push the branch content to the remote Git repository
         git push

   3. Merge code across projects

      1. Go to DataStudio of the target workspace git_cross_project_2. Click Code Management in the left-side navigation pane, and find the Code Merge tab.

      2. In the branch input box, enter the name of the save branch of the source workspace, such as dataworks_cn_shenzhen_270256_save.

      3. Click merge preview. After confirming that the preview is correct, merge the code from workspace git_cross_project_1 into workspace git_cross_project_2. For details about the merge operation, see Merge from Git to DataWorks.