EMR Workflow to DataWorks
This topic describes how to migrate scheduling workflows from EMR Workflow to DataWorks using the LHM scheduling migration tool. The process involves three steps: exporting EMR Workflow tasks, converting the scheduling tasks, and importing the tasks into DataWorks.
1. Export EMR Workflow scheduling workflows
The scheduling tool obtains workflow, resource, and data source information using the EMR Workflow page API. The following section describes the export process.
1. Prerequisites
Prepare a JDK 17 runtime environment and ensure network connectivity between the runtime environment and EMR Workflow. Then, download the scheduling migration tool to a local directory and decompress it.
To test the network connectivity, verify that you can connect to the EMR Workflow page URL.
Log on to the EMR Workflow console to obtain the URL. Then, verify the network connectivity in the environment where the tool is deployed.
ping emr-workflow-cn-hangzhou.data.aliyun.com2. Configure connection information
In the `conf` folder of the project directory, create an export configuration file in JSON format, such as `read.json`.
Before you use the file, delete the comments in the JSON code.
{
"schedule_datasource": {
"name": "YourEMR", // Give your EMR Workflow a name.
"type": "EmrWorkflow",
"properties": {
"project": "MyWorkspace", // Workspace name
"projectId": "1201575", // Workspace ID
"cookie": "session.user=1663676968115556_1201575; sessionId=b1b46589-9050-4501-8306-a165f88aa9b6; cna=EqsbH4WKzkcBASQBsYAqwPbC; _abfpc=27b472bfc7f89623e8ed9d284f86de7a5942990e_2.0; t=b42b2d23a05efcfc3904fd27809e8876; tongyi_guest_ticket=u6xQ53c2yDcQuUoyW8m8_P$Euc2tK1vAJrbd0Pp8RqQlFXqgAxQlQbm5GMjKeLhhMmLVgmaWvv690; tongyi_sso_ticket=apRX0qcdzyjC5$ojKmLhMMmeswgxpWvG_dV8knnYvIBjexQnxm08QA6sXpfbGzzg_8yn1gsBH1oO0; bs_n_region=cn-hangzhou; yunpk=1663676968115556; login_aliyunid_pk=1663676968115556; login_current_pk=205706621271387868; help_csrf=1QU0Vd9goAw21fKSbkhMwV0BsXQrSRzyYlNzgRJH%2Bm1zjtkNMXBven%2BTyFm2hkxdBeFQC%2B9a%2B%2FUPNAeOxxzEYES1Ybz4HdCfbYmp3WHK1bDF1bb6loa86ycMrrV0dBU6sEOTH1LoEBE15m%2Fv%2BzdV4A%3D%3D; cr_token=80407c38-06df-44b6-b383-b7464af3f24f; bd=s0ouCmI%3D; sca=2fe2ea4a; ecsCustomBuyVersion=new; copilot-session-id-dw-copilot=be98f3db-e5dd-458d-bd56-53bb43c15b2d; bs_n_lang=zh_CN; activeRegionId=cn-hangzhou; notice_behavior=implied,eu; channel=r%2BOBdZKdhDzlinkO0n6spUNMqtdofQvlsXqge2s9iQMPfaUh1kgmvsuAjBSBefwJsQGEHPZJROwUH7DmRkyDtA%3D%3D; _samesite_flag_=true; cookie2=167d0d6e4a710c25359bc7ec926b0f40; _tb_token_=e3b766e8e037e; cnaui=%2522hesheng%2520%2540%25201663676968115556%2522; aui=%2522hesheng%2520%2540%25201663676968115556%2522; language=zh_CN; ck2=1dfffdf1279716911724407ae9119282; an=hesheng; lg=true; sg=g28; atpsida=6dd15eb4619a28ae3d50f053_1741692372_2; currentRegionId=cn-hangzhou; partitioned_cookie_flag=doubleRemove; aliyun_site=CN; aliyun_lang=zh; login_aliyunid_csrf=_csrf_tk_1216439968355914; login_aliyunid=\"hesheng @ 1663676968115556\"; login_aliyunid_ticket=3RctYK12wchS7wBJzHKJeoTE.1118Eu1zAQBswru9dwX3p626Hkye6D1zqmF4iKW2rjiRruDxeFL6TkGmxD94zf6dACfqLfEmPxA1FcHTRSpbBazr4oWhCMPAWPhXubTua1j6aeR2c5PfrgpP5UEJcKDQWHKbMqcTYYMeca46BiaSeZUx6uw7n3YvbyzhFytW88Ssyks5mzB.1QDta7ZWM7nyQUHpV7rKtSCR4bWExYZ7CEpGotZErJz844UFnjNEc4pRohJrZ7mYS; login_aliyunid_sc=3R5H3e3HY2c8gwLZuY5GmS7K.1113hM7ys3oXbRaRYNucf2xgCHC4tuX7FWZ2gYR18qCgHCDqFDK5JHMUcKeBLZckMwoh5W.2mWNaj35xPge9RXAhrfkJ4XtVqVFhv4QRSUVinmgzVoCwXMVBTfHwBYfhCWbqsFbgJ; dw_bff=205706621271387868.%E5%92%8C%E5%A3%B0.580581087550304.1742181195731.1663676968115556..26842.OFFICIAL.18.1742267595731.hesheng.5.2.1742262992.zh_CN.f2a68809ec371fbcb96ebf58a9d7a0ea3fe8053a; c_token=16241d6d00c3d29488917ae3f47b612c; tfstk=gZIj2cgqa1IyJzllnSeyFJZxZot1c1ZUXA9OKOnqBnKvCh6h1jR27sROP6fsb-XToRZ139TwWSxVf_6CIIh20FP6565P01-Yi1_1KOW4InnqfF6CGZl2DPDcr6WK0izDbctDjhFUTkrUntxMXnOdIPRceTvTUKQldkQkjhFUa4o9oktGtXmuTGB8FdJZDKC96QeWQpttDFnxybdkBhK9WdH-2pvv6fdxk86JZdK961Kt6qcWQr9Oh-CUznygYCXvNcnO2lYXOt3ZXcIWhEsOHQ61Cg9XlBLzdgE1A9BAxdsgpVO1KZC2E1ZQN3IfGiLWflik09_RGU_b10Tly9j9PaVI-9XAGwLdDuwyPiXeDQsgIcAAuOIw5iFKQIIcggThjjNy_wWPDU1La01M5ZQpDGNQ2gzmYBTOhVMW-cpWT8wSSVVcc6qLHbYAtEpkHky7FbgMkLv5M8wSGFYvEKF_F8GSS; isg=BNfX5pEkfPamJ_k2pCb_1X4nZk0hHKt-NoZ1UCkEaaYPWPWaMe_mz2VwvvjGsIP2",
"region": "cn-hangzhou" // Region
},
"operaterType": "AUTO"
},
"conf": {
}
}2.1. Workspace name and ID
Open the EMR Workbench - Workflow console to obtain the workspace name and ID.
2.2. Cookie
The tool obtains scheduling information through frontend APIs. Therefore, you must obtain the cookie information. Open the developer tools in your browser and perform an operation on the page to trigger an API call. This lets you retrieve the cookie from the Headers. Note that cookies expire periodically, so you must update the cookie before you export the tasks.
2.3. Region
The region is an enumerated value. You can obtain it from the EMR Workflow URL.
3. Download the EMR Workflow definition file
Use the export feature on the EMR Workflow page to download the workflow definition to a local directory. The definition is a JSON file.
4. Run the scheduling discovery tool
The scheduling discovery tool parses and inventories EMR Workflow and its task information. It generates the following file:
A package parsed by the discovery tool that standardizes the data structure of the original information. This package is called the ReaderOutput package.
The ReaderOutput package is the final result of the scheduling export.
Run the discovery tool from the command line. The command is as follows:
sh ./bin/run.sh read \
-c ./conf/<your_configuration_file>.json \
-f ./data/0_OriginalPackage/<EMR_Workflow_definition_file>.json \
-o ./data/1_ReaderOutput/<source_discovery_export_package>.zip \
-t emr-workflow-readerThe `-c` parameter specifies the path of the configuration file. The `-f` parameter specifies the storage path of the EMR Workflow definition file (see Section 3). The `-o` parameter specifies the storage path of the ReaderOutput package. The `-t` parameter specifies the name of the discovery plugin.
For example, to export Project A:
sh ./bin/run.sh read \
-c ./conf/projectA_read.json \
-f ./data/0_OriginalPackage/47000000000.json \
-o ./data/1_ReaderOutput/projectA_ReaderOutput.zip \
-t emr-workflow-reader5. View the export results
Open the generated `ReaderOutput.zip` package in the `./data/1_ReaderOutput/` directory to preview the export results.
The statistical report provides a summary of the basic information about workflows, nodes, resources, functions, and data sources in EMR Workflow.
The `data/project` folder contains the standardized data structure of the EMR Workflow scheduling information.
The statistical report provides two special features:
1. You can change some properties of workflows and nodes in the report. The editable fields are displayed in blue font. In the next stage, scheduling conversion, the tool applies the property changes from the table during initialization.
2. The report lets you skip certain workflows during conversion by deleting their corresponding rows in the workflow child table. This feature is the workflow blacklist. Note: If workflows have dependencies on each other, the related workflows must be converted in the same batch. Do not use the blacklist to separate them, as this will cause exceptions.
For more information, see Use the overview report in the migration tool to modify scheduling properties.
2. Convert EMR Workflow workflows to DataWorks format
1. Prerequisites
The discovery tool has run successfully, the EMR Workflow scheduling information has been exported, and the `ReaderOutput.zip` package has been generated.
(Optional but recommended) Open the exported discovery package and view the statistical report to verify that the full scope of the migration was exported.
2. Configuration items for conversion
2.1. Conversion configuration template
Before you use the file, delete the comments in the JSON code.
{
"conf": {},
"self": {
"if.use.default.convert": false,
"if.use.migrationx.before": false,
"if.use.dataworks.newidea": true,
"conf": [
{
"nodes": "all",
"rule": {
"settings": {
"workflow.converter.sql.submitAs": {
"HIVE": "ODPS_SQL",
"PRESTO": "ODPS_SQL",
"STARROCKS": "ODPS_SQL"
},
"workflow.converter.impalaShell.submitAs": "ODPS_SQL",
"workflow.converter.remoteShell.submitAs": "SSH",
"workflow.converter.shellNodeType": "DIDE_SHELL",
"workflow.converter.spark.submitAs": {
"JAVA": "ODPS_SPARK",
"PYTHON": "ODPS_SPARK",
"SQL": "ODPS_SQL"
},
"workflow.converter.target.engine.type": "MaxCompute",
"workflow.converter.target.unknownNodeTypeAs": "DIDE_SHELL",
"workflow.converter.mrNodeType": "ODPS_MR",
"workflow.converter.commandSqlAs": "DIDE_SHELL",
"workflow.converter.connection.mapping": {
"HiveSource": "hesheng_maxcompute_test4",
"PrestoSource": "hesheng_maxcompute_test4",
"SrSource": "hesheng_maxcompute_test4",
"SshSource": "sshecs1"
}
}
}
}
]
},
"schedule_datasource": {},
"target_schedule_datasource": {}
}2.2. Owner mapping
EMR Workflow records the owner of each workflow. The owner is critical information for team development. The tool supports mapping EMR Workflow users to DataWorks users to mark the corresponding owners for workflows and nodes.
EMR Workflow users are not exposed on the frontend. You can open the developer tools on the project management page and retrieve the user ID from the `get-user-info` request. In the following figure, the user ID is 430.
The ID of the workflow owner can be viewed in the `projects` request.
In a DataWorks workspace, you can add users as members. You can obtain the user ID from the upper-right corner.
You can also retrieve the ID from the **Owner** drop-down list on the Data Development page.
2.3. Node conversion rules
2.3.1. Scope of rules
You can set the scope for node conversion rules. For example, to convert all nodes using a uniform rule, you can set `nodes` to `all` and fill in the **Settings**. Typically, you only need to configure one `all` rule group.
{
"conf": {},
"self": {
"conf": [
{
"nodes": "all", // The scope of the rule group is ALL. All nodes are converted based on this rule.
"rule": {
"settings": {
// Settings
}
}
]
}
}If certain nodes require separate conversion rules, you can specify the scope of the rule by entering the task ID or name in the nodes field. To set a rule for multiple nodes, separate their IDs or names with commas. We recommend using IDs instead of names to specify the scope and prevent misconfiguration. You can also use a regular expression to match node names. Additionally, we strongly recommend setting up a "normal" rule group to define a default conversion rule for the remaining nodes.
{
"conf": {},
"self": {
"conf": [
{
"nodes": "node1Name, node2Id", // The scope of the rule group is node1 and node2.
"rule": {
"settings": {
// Settings 1
}
},
{
"nodes": "node3Name, node4Id", // The scope of the rule group is node3 and node4.
"rule": {
"settings": {
// Settings 2
}
},
{
"nodes": "regexExpression", // You can use a regular expression to filter node names.
"rule": {
"settings": {
// Settings 3
}
},
{
"nodes": "normal", // The conversion rule for the remaining nodes.
"rule": {
"settings": {
// Settings 4
}
}
]
}
}2.3.2. Conversion rules
This tool supports the following EMR Workflow node types:
SHELL, SQL, PYTHON, DATAX, SQOOP, SEATUNNEL, HIVE CLI, SPARK, IMPALA SHELL, REMOTE SHELL, MR, PROCEDURE, HTTP, CONDITIONS, SWITCH, DEPENDENT, and SUB PROCESS.
You can configure DataWorks mapping rules for the following types:
· SHELL (workflow.converter.shellNodeType):
You can convert them to nodes such as DIDE_SHELL, EMR_SHELL, or VIRTUAL.
· SQL (workflow.converter.sql.submitAs):
Source database types: SSH, DORIS, STARROCKS, PRESTO, and HIVE.
You can convert them to various SQL nodes or database nodes.
· PROCEDURE (workflow.converter.dolphinscheduler.sqlNodeTypeMapping):
You can convert them to various SQL nodes or database nodes.
· PYTHON (workflow.converter.pyNodeType):
You can convert them to nodes such as PYTHON, PYODPS, PYODPS3, or EMR_SHELL.
· HIVECLI (workflow.converter.dolphinscheduler.sqlNodeTypeMapping/HIVE):
You can convert them to nodes such as EMR_HIVE or ODPS_SQL.
· SPARK (workflow.converter.sparkSubmitAs):
You can convert SparkJava and SparkPython nodes to ODPS_SPARK or EMR_SPARK nodes.
You can convert SparkSql nodes to ODPS_SQL or EMR_SPARK_SQL nodes.
· IMPALASHELL (workflow.converter.impalaShell.submitAs):
You can convert them to nodes such as ODPS_SQL.
· REMOTESHELL (workflow.converter.remoteShell.submitAs):
You can convert them to SSH nodes.
· MR (workflow.converter.mrNodeType):
You can convert them to ODPS_MR or EMR_MR nodes.
The enumeration class for DataWorks node types is as follows:
Node types that follow fixed conversion rules:
· DATAX: Converted to DI nodes. These nodes can be configured in either a custom template mode (JSON Script mode) or a regular, form-based frontend mode.
The tool converts reader plugin configurations for the following data sources: MYSQL → mysql, POSTGRESQL → postgresql, ORACLE → oracle, SQLSERVER → sqlserver, ODPS → odps, OSS → oss, HIVE → hdfs, HDFS → hdfs, CLICKHOUSE → clickhouse, and MONGODB → mongodb.
You can use this tool to convert writer plugin configurations for the following data sources: MYSQL to mysql, POSTGRESQL to postgresql, ORACLE to oracle, SQLSERVER to sqlserver, ODPS to odps, OSS to oss, HIVE to hdfs, HDFS to hdfs, CLICKHOUSE to clickhouse, and MONGODB to mongodb.
· SQOOP: Nodes are converted to DI nodes.
The tool supports converting reader plugin configurations for the following data sources: MySQL, Hive, and HDFS.
The tool supports converting writer plugin configurations for MySQL, Hive, and HDFS data sources.
· SEATUNNEL: Has been converted to DI nodes.
Script conversion is not yet supported. The conversion is limited to node and scheduling information.
· HTTP: HTTP nodes are converted to DIDE_SHELL (general-purpose Shell) nodes. The migration tool automatically concatenates the request parameters into a curl command.
· SWITCH: Converted to CONTROLLER_BRANCH (branch) nodes. The functionality remains unchanged after the migration.
· SUB_PROCESS: These are converted to SUB_PROCESS nodes, and their functionality is unchanged after the migration. When importing to DataWorks, the migration tool enables the "Can be referenced" switch for the referenced workflow. As a result, the referenced workflow can be started only by a call from a SUB_PROCESS node and cannot be scheduled to run independently.
· DEPENDENT: DEPENDENT nodes are converted to VIRTUAL nodes, and their dependencies are converted to node lineage. For example, a dependency on Workflow A creates a lineage from the last node of Workflow A to the DEPENDENT node. A dependency on Node A creates a lineage from Node A to the DEPENDENT node. The following figures show examples.
· CONDITIONS: This node contains two layers of logic implemented with a two-level structure of CONTROLLER_JOIN (merge) nodes. For example, consider the case in the figure below. The CONDITIONS node has two upstream nodes, A and B, and two downstream nodes, C and D. The logical expression is ((!A&B)|(A&!B)|(!A&!B)). If the expression is true, the flow proceeds to C. Otherwise, the flow proceeds to D. The upper level generates three merge nodes to calculate the results of !A&B, A&!B, and !A&!B respectively. The lower level generates two nodes. One triggers the execution of downstream node C if ((!A&B)|(A&!B)|(!A&!B)) is true. The other triggers the execution of downstream node D if (!(!A&B)&!(A&!B)&!(!A&!B)) is true. This setup replicates the behavior of the CONDITIONS node.
3. Run the scheduling conversion tool
To run the conversion tool from the command line, the command is as follows:
sh ./bin/run.sh convert \
-c ./conf/<your_configuration_file>.json \
-f ./data/1_ReaderOutput/<source_discovery_export_package>.zip \
-o ./data/2_ConverterOutput/<conversion_result_output_package>.zip \
-t emrworkflow-dw-converterThe -c parameter specifies the path of the configuration file, -f specifies the storage path for the ReaderOutput package, -o specifies the storage path for the ConverterOutput package, and -t specifies the conversion plugin.
The following example shows how to convert EMR Workflow Project A:
sh ./bin/run.sh convert \
-c ./conf/projectA_convert.json \
-f ./data/1_ReaderOutput/projectA_ReaderOutput.zip \
-o ./data/2_ConverterOutput/projectA_ConverterOutput.zip \
-t emrworkflow-dw-converterThe conversion tool displays process information during the conversion. You can check this information for errors. When the conversion is complete, the command line displays statistics for successful and failed conversions. The overall conversion process is not affected if some nodes fail to convert. You can manually modify any failed nodes after they are migrated to DataWorks.
4. View the conversion results
You can open the generated ConverterOutput.zip package in the ./data/2_ConverterOutput/ directory to preview the export results.
The statistical report provides a summary of converted workflows, nodes, resources, functions, and data sources.
The data/project folder is the main package for scheduling migrations after conversion.
The statistical report has two special features:
1. You can change some properties of workflows and nodes in the report. The editable fields are displayed in blue font. The tool applies these property changes when you import the report into DataWorks.
2. The report lets you skip specific workflows when importing into DataWorks by deleting their corresponding rows in the workflow child table. This is known as the workflow blacklist. Note: If workflows have dependencies on each other, they must be imported in the same batch. Do not separate them using the blacklist, because this will cause exceptions.
For more information, see Modifying scheduling properties using the migration tool's overview report.
3. Import workflows into DataWorks
The heterogeneous conversion feature of the LHM migration tool transforms scheduling elements from the source into the DataWorks scheduling format. The tool provides a unified entry point to upload and import workflows from various migration scenarios into DataWorks.
The import tool supports multiple write operations, automatically creating new workflows or overwriting existing ones.
1. Prerequisites
1.1. Successful conversion
The conversion tool has successfully converted the source scheduling information into DataWorks scheduling information and generated the ConverterOutput.zip package.
(Recommended) Open the conversion output package and view the statistical report to check whether the migration scope was successfully converted.
1.2. DataWorks configuration
Perform the following steps in DataWorks:
1. Create a workspace.
2. Create an AccessKey pair (AK and SK) with administrator permissions for the workspace. (We strongly recommend creating an AccessKey pair that is bound to your account to help troubleshoot write issues.)
3. In the workspace, you can create data sources, attach computing resources, and create resource groups.
4. In the workspace, upload resource files and create user-defined functions (UDFs).
1.3. Network connectivity check
You can verify connectivity to the DataWorks endpoint.
The following is a list of endpoints:
ping dataworks.aliyuncs.com2. Configuration items for import
In the project's conf folder, create an export configuration file in JSON format, such as writer.json.
Before using the file, delete the comments from the JSON.
{
"schedule_datasource": {
"name": "YourDataWorks", // Give your DataWorks data source a name.
"type": "DataWorks",
"properties": {
"endpoint": "dataworks.cn-hangzhou.aliyuncs.com", // Endpoint
"project_id": "YourProjectId", // Workspace ID
"project_name": "YourProject", // Workspace name
"ak": "************", // AK
"sk": "************", // SK
},
"operaterType": "MANUAL"
},
"conf": {
"di.resource.group.identifier": "Serverless_res_group_***_***", // Scheduling resource group
"resource.group.identifier": "Serverless_res_group_***_***", // Data Integration resource group
"dataworks.node.type.xls": "/Software/bwm-client/conf/CodeProgramType.xls", // Path to the DataWorks node type table
"qps.limit": 5 // QPS limit for sending API requests to DataWorks
}
}2.1. Endpoint
Select an endpoint based on your DataWorks workspace region. For more information, see:
2.2. Workspace ID and name
Go to the workspace product page in the DataWorks console. You can obtain the workspace ID and name from the Basic Information section on the right.
2.3. Create and grant permissions to an AccessKey pair
Create an AccessKey pair on the user page that has administrator read and write permissions for the target DataWorks workspace.
Permission management involves two areas. If an account is a RAM user, you must first grant it permissions to perform operations in DataWorks.
You can find the Access Policy page at: https://ram.console.aliyun.com/policies
Then, you can assign workspace permissions to the account in the DataWorks workspace.
Note: You can set a network access control policy for the AccessKey. Ensure that the IP address of the machine where the migration tool is running is granted access.
2.4. Resource groups
On the DataWorks workspace product page, go to the Resource Group page in the left-side menu bar. Attach a resource group and obtain its ID.
A general-purpose resource group can be used for both node scheduling and Data Integration. In the configuration items, you can set both the scheduling resource group (resource.group.identifier) and the Data Integration resource group (di.resource.group.identifier) to the same general-purpose resource group.
2.5. QPS settings
The tool imports data by calling DataWorks APIs. The queries per second (QPS) and daily call limits for OpenAPI read and write operations vary based on the DataWorks edition. For more information, see Limits.
Set "qps.limit" to 5 for DataWorks Basic Edition, Standard Edition, and Professional Edition, and to 20 for Enterprise Edition.
Note: Do not run multiple import tools simultaneously.
2.6. DataWorks node type ID settings
In DataWorks, some node types are assigned different type IDs based on the region. The specific type ID is determined by the DataWorks Data Development interface. This behavior primarily affects database nodes. For more information, see Database nodes.
For example, the NodeTypeId of a MySQL node is 1000039 in the China (Hangzhou) region and 1000041 in the China (Shenzhen) region.
To accommodate these regional differences in DataWorks, you can configure the node TypeId table used by the tool.
You can import the table using a configuration item in the import tool:
"conf": {
"dataworks.node.type.xls": "/Software/bwm-client/conf/CodeProgramType.xls" // Path to the DataWorks node type table
}To obtain the node type ID in the DataWorks Data Development interface, you can create a new workflow, add a new node to it, and click Save to view the workflow's spec.
If the node type is configured incorrectly, the following error is reported when you publish the workflow.
3. Run the DataWorks import tool
You can run the conversion tool from the command line. The command is as follows:
sh ./bin/run.sh write \
-c ./conf/<your_configuration_file>.json \
-f ./data/2_ConverterOutput/<conversion_result_output_package>.zip \
-o ./data/4_WriterOutput/<import_result_storage_package>.zip \
-t dw-newide-writerThe -c, -f, -o, and -t parameters specify the path of the configuration file, the storage path of the ConverterOutput package, the storage path of the WriterOutput package, and the name of the submission plugin, respectively.
For example, to import Project A into DataWorks:
sh ./bin/run.sh write \
-c ./conf/projectA_write.json \
-f ./data/2_ConverterOutput/projectA_ConverterOutput.zip \
-o ./data/4_WriterOutput/projectA_WriterOutput.zip \
-t dw-newide-writerThe import tool displays process information during runtime. You can check this output for errors. After the import is complete, the command line displays statistics for successful and failed imports. The overall import process is not affected if some nodes fail to import. You can manually modify any failed nodes in DataWorks.
4. View the import results
After the import is complete, you can view the results in DataWorks. You can also monitor the workflows as they are imported. To stop the import, run the jps command to find BwmClientApp and then use the kill -9 command to terminate the process.
5. Q&A
5.1. The source system is under continuous development. How can I submit these increments and changes to DataWorks?
The migration tool operates in overwrite mode. You can rerun the export, conversion, and import processes to submit incremental changes from the source to DataWorks. The tool matches workflows by their full path to determine whether to create or update them. Therefore, to migrate changes, do not move the workflows.
5.2. The source system is under continuous development, and I am also modifying and managing workflows in DataWorks. Will an incremental migration overwrite the changes made in DataWorks?
Yes, the migration tool operates in overwrite mode. We recommend that you complete the migration before making any modifications in DataWorks. Alternatively, you can migrate in batches. After a batch of workflows is migrated, you can start modifying them in DataWorks once you confirm that they will not be overwritten again. Different batches do not affect each other.
5.3. Importing the entire package takes too long. Can I import only a part of it?
Yes. You can manually trim the package before importing it. In the data/project/workflow folder, keep the workflows that you want to import and delete the rest. Then, recompress the folder into a ZIP package and run the import tool. Note that workflows with mutual dependencies must be imported together. Otherwise, the node lineage between them will be lost.