DataWorks real-time synchronization tasks support elastic scaling. You can scale the resource group for an integration task based on a predefined schedule to optimize resource usage and reduce costs. This topic describes how to configure elastic scaling.
How it works
To improve resource utilization in Data Integration and leverage the elasticity of the cloud, DataWorks supports task-level dynamic scaling based on Serverless resource groups. By adjusting the resource configuration for different cycles and execution times, you can ensure that integration tasks receive more computing units (CUs) during peak periods and run with minimal CUs during off-peak periods. This helps you avoid resource waste and reduce costs.
Usage notes
-
Within a single execution plan, the start time of a new resource configuration must be at least 30 minutes after the start time of the previous one. This allows the previous scaling operation to complete. If the interval is less than 30 minutes, you cannot save the new configuration.
-
If you use a subscription resource group and elastic scaling exceeds the Data Integration resource quota, the system triggers a rollback. The system then reverts the synchronization task's resources to their size from the last successful run.
-
A scaling operation temporarily interrupts the job for up to 3 minutes.
Procedure
-
Go to Data Integration.
Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose . On the page that appears, select the desired workspace from the drop-down list and click Go to Data Integration.
-
On the Data Integration page, click the
icon to go to the Synchronization Task page. In the Task List module, you can view your Data Integration tasks. -
Go to the Create/Modify Execution Plan page.
You can create or modify an execution plan for a Data Integration task in either of the following ways:
-
From the task list:
After creating a Data Integration task, you can create or modify its execution plan.
On the task list page, find the desired integration task. In the Operations column, choose to go to the Create/Modify Execution Plan page.
-
From the task details page:
While a Data Integration task is running, you can create or modify its execution plan.
On the task list page, click the Name/ID of the task to go to the Task Details page. In the , click Tune Resource to go to the Create/Modify Execution Plan page.
-
Create or modify an execution plan
-
On the Create/Modify Execution Plan page, go to the plan list. To create a new plan, click Create Plan. To modify an existing plan, you must first click Stop if the plan is running, and then click Edit in the Operations column.
-
Click Add Resource Settings to configure the scaling schedule, including its cycle, time, and the number of CUs. You can also click Edit for an existing configuration to modify it.
-
Plan Name: You must specify a name for the plan. You cannot edit or modify a plan's name after it is saved.
-
Add Resource Settings: When you create or modify an execution plan, you can add or modify the following settings.
Configuration Item
Description
Execution Frequency
Configure the frequency based on the resource requirements of the Data Integration task. Options include Once, Daily, Weekly, and Monthly.
Execution Date
If you set the execution frequency to Run Once, Weekly, or Monthly, you can specify the exact date or dates for execution.
Execution time
Set the specific time and time zone for the plan to run.
Number of CUs
Configure the number of CUs required for the Data Integration task.
Operations
-
While editing a configuration, you can Save, Cancel, or Delete your changes.
-
For a saved configuration, you can Edit or Delete it.
Within an execution plan, you can click Add Resource Settings to add multiple resource configurations.
-
-
-
After you complete the configuration, click Confirm to save the execution plan.
Apply an execution plan
After you create an execution plan, you can find it in the plan list. Click Apply in the Operations column to apply the plan to the current integration task.
A Data Integration task can have only one active execution plan at a time. To use different resource levels for different time periods, add multiple resource configurations within a single plan instead of creating multiple plans.
View execution history
On the Create/Modify Execution Plan page, you can view created plans in the Plans tab. In the Tuning History section, you can review the Execution Result, Execution Time, and Execution Information for applied plans.
More operations
After creating or modifying an execution plan, you can perform additional management operations on it from the plan list on the Create/Modify Execution Plan page.
-
Delete an execution plan: You can delete inactive plans. To delete an active plan, you must stop it first.
-
View an execution plan: You can view the resource configuration details of an active plan.
-
Stop an execution plan: You can stop an active plan. Once stopped, a plan can be edited or deleted.