Configure a batch synchronization task in script mode

Usage notes

Use the Code Editor in the following scenarios:

• The data source does not support configuration in wizard mode.

  Note

  The user interface indicates whether a data source supports wizard mode.

  For example, if you select HBase11xsql as the destination data source, a yellow warning message appears on the network and resource configuration page: "The current data source type does not support task editing in wizard mode. The task will be configured in script mode." In this case, click the Code Editor button in the toolbar to switch modes and configure the task.

• Some data source configuration parameters are available only in the Code Editor.

• You can use the Code Editor to configure some data sources that cannot be created directly in DataWorks.

Step 1: Create a batch synchronization node

Step 2: Configure the data source and resource group

You can switch from wizard mode to script mode at any step. For a complete script configuration, we recommend the following approach:

1. First, use the wizard to select the data source and resource group, and test network connectivity.

2. Then switch to script mode.

The system automatically populates this information into the generated JSON script.

Alternatively, you can switch directly and then manually configure the settings in script mode: specify the data source in the JSON code, and set the resource group and required resource size in the Advanced Settings panel on the right.

Step 3: Switch to script mode and import a template

Click the Convert to Script icon in the toolbar.

If the script has not been configured, you can click the icon in the toolbar to quickly import a script template by following the on-screen instructions.

Step 4: Edit the script to configure the synchronization task

Common script mode configurations:

{
    "type":"job",
    "version":"2.0",
    "steps":[
        {
            "stepType":"plugin_name",
            "parameter":{...},
            "name":"Reader",
            "category":"reader"
        },
        {
            "stepType":"plugin_name",
            "parameter":{...},
            "name":"Writer",
            "category":"writer"
        }
    ],
    {
        "name":"Processor",
        "stepType":null,
        "category":"processor",
        "copies":1,
        "parameter":{...}
    },
    "setting":{
        "executeMode":null,
        "errorLimit":{
            "record":""
        },
        "speed":{
            "concurrent":2,
            "throttle":false
        },
        "timeZone":"Asia/Shanghai"
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
            }
        ]
    }
}

1. Configure the basic information and column mappings for the reader and writer.

   Important

   The configurations vary for different plug-ins. The following content describes only common configurations as examples. Whether a plug-in supports a specific configuration and how the configuration is implemented depend on the plug-in. For details, see the Reader Script Demo and Writer Script Demo for each data source in the list of data sources.

   By configuring these parameters, you can:

   • Reader

     Configuration	Description
     where (configure the synchronization scope)	Some source types support data filtering. You can specify a condition (a WHERE clause, but you do not need to include the where keyword) to filter source data. During task execution, only data that meets the condition is synchronized. For more information, see Configure a filter condition. To implement incremental synchronization, you can combine the filter condition with scheduling parameters to make it dynamic. For example, by using gmt_create >= '${bizdate}', the task synchronizes only newly added data each day. You also need to assign a value to the variable defined here when configuring the schedule settings. For more information, see Configure scheduling parameters. The incremental synchronization configuration method varies for different data sources (plug-ins). If no data filter condition is configured, full data of the table is synchronized by default.
     splitPk (configure the split key for relational databases)	Specifies the column used to split source data for parallel reading. During task execution, data is split into multiple tasks based on this column for concurrent batch reads. We recommend that you use the primary key as the splitPk value because primary keys are generally evenly distributed, which helps prevent data hotspots. Currently, splitPk supports only integer-based splitting. Other types such as strings, floating-point numbers, and dates are not supported. If you specify an unsupported type, the splitPk feature is ignored and the data is synchronized through a single channel. If splitPk is not specified, including when the splitPk parameter is not provided or is left empty, the data is synchronized through a single channel. Not all plug-ins support split key configuration. The preceding information is an example only. For details, see the specific plug-in documentation. For more information, see Supported data sources and sync solutions.
     column (define source columns)	Define the columns to synchronize from the source in the column array. You can also write constants, variables, and functions as custom columns to the destination, such as '123', '${variable_name}', and 'now()'.

   • Writer

     Configuration	Description
     preSql & postSql (configure pre-sync and post-sync SQL statements)	Some data sources support executing SQL statements on the destination before synchronization (before writing) and after synchronization (after writing). Example: MySQL Writer supports preSql and postSql configuration, which allows you to execute MySQL commands before or after data is written to MySQL. For example, you can configure the MySQL truncate table command truncate table tablename in the Pre-import Preparation Statement (preSql) setting of the MySQL Writer to clear old data from the table before synchronization (before writing data to MySQL).
     writeMode (define the write mode for conflicts)	Specifies the write mode when conflicts occur, such as path or primary key conflicts. This varies by data source and writer plug-in. See the specific writer plug-in documentation for details.

2. Channel Control.

   You can configure efficiency settings in the setting section, including concurrency, synchronization speed, and dirty data handling.

   Parameter	Description
   executeMode (distributed processing)	Controls whether to enable distributed mode. distribute: Enables distributed processing. The task is split into slices distributed to multiple execution nodes for concurrent execution, allowing synchronization speed to scale horizontally with the cluster size beyond single-node limits. null: Disables distributed processing. The configured concurrency applies only to process-level concurrency on a single node, and multi-node computing cannot be used. Important If your exclusive resource group for Data Integration has only one node, distributed mode is not recommended because it cannot leverage multi-node resources. If a single node meets your speed requirements, single-node mode is recommended for simplicity. The concurrency must be 8 or higher to enable distributed processing. Some data sources support distributed mode for task execution. For details, refer to the specific plug-in documentation. Enabling distributed processing consumes more resources. If an out-of-memory (OOM) error occurs during execution, try disabling this option.
   concurrent (maximum expected concurrency)	Defines the maximum number of threads for parallel reading from the source or writing to the destination. Note Due to factors such as resource specifications, the actual concurrency may be less than or equal to the configured value. The debug resource group is billed based on actual concurrency. For more information, see Billing.
   throttle (synchronization speed)	Controls the synchronization speed. true: Enables throttling. This protects the source database by capping extraction speed to prevent overload. The minimum throttling rate is 1 MB/s. Note When throttle is set to true, you also need to set the mbps (synchronization speed) parameter. false: Disables throttling. The task delivers the maximum transfer performance that the hardware environment allows within the configured concurrency limit. Note The throughput metric is a Data Integration internal measurement and does not represent actual NIC traffic. Typically, NIC traffic is 1 to 2 times the channel traffic. The actual ratio depends on the serialization mechanism of the data storage system.
   errorLimit (error count control)	Defines the dirty data threshold and how it affects the task. Important Excessive dirty data can affect the overall synchronization speed of the task. If not configured, dirty data is allowed by default and does not affect task execution. If set to 0, no dirty data is allowed. The task fails if dirty data is generated during synchronization. If dirty data is allowed and a threshold is set: If the dirty data is within the threshold, the synchronization task ignores the dirty data (does not write it to the destination) and continues normally. If the dirty data exceeds the threshold, the synchronization task fails. Note Dirty data criteria: Dirty data is data that has no business significance, has an invalid format, or encounters issues during synchronization. If an exception occurs while writing a record to the destination, that record is considered dirty data. For example, writing VARCHAR data from the source to an INT column in the destination results in dirty data due to an invalid type conversion. You can control whether dirty data is allowed during synchronization task configuration, and set a maximum error count so that the task fails when the dirty data count exceeds the specified limit.
   timeZone (time zone setting)	Specifies the time zone for the synchronization task. This setting takes effect when time-type columns are converted on the source or destination. Data Integration reads and writes time columns based on the specified time zone. Configuration example: "timeZone":"Asia/Shanghai". This parameter can be configured only in the setting section in script mode. The destination in wizard mode (codeless UI) does not support time zone settings. The time zone value uses the standard IANA time zone format, such as Asia/Shanghai and America/New_York. If not configured, Data Integration uses the system default time zone.

   Note

   Overall synchronization speed is also affected by source data source performance, network environment, and other factors. For more information, see Tune batch synchronization tasks.

Step 5: Configure schedule settings

Step 6: Submit and deploy the task

References

• Supported data sources and synchronization solutions

• Overview of batch synchronization

• Codeless UI configuration