MaxCompute

Batch read

• MaxCompute Reader supports reading data from partitioned and non-partitioned tables, but not from virtual views or external tables.

• When you perform a batch read from a MaxCompute partitioned table, you cannot directly configure field mapping for partition key columns. To synchronize partition key values, you can add a custom field, manually enter the partition name, and then configure the field mapping.

• Use this method to synchronize data from a partition that corresponds to the scheduling time.

  For example, a partitioned table named t0 contains the id and name columns. The level-1 partition key is pt and the level-2 partition key is ds. To read data from the partition where pt=<Business date> and ds=hangzhou, you must specify the partition values as pt=${scheduling parameter} and ds=hangzhou when you configure the source. Then, you can configure the field mapping for the id and name columns.

• You can write partition key columns to a destination table by adding them as custom fields.

• MaxCompute Reader supports data filtering by using a WHERE clause.

Batch write

• MaxCompute Writer does not support the VARCHAR data type if the source data contains null values.

• If the destination table is a DeltaTable, expand Advanced Configuration and set Visible After Synchronization to Yes. Otherwise, the task reports an error if the concurrency is greater than 1.

• Synchronizing data from a source to a MaxCompute external table is not supported.

• If a column in the destination table is not mapped to a source column, Data Integration sets its value to null after synchronization, even if a default value was specified when the table was created.

Real-time write

• Real-time data synchronization tasks support serverless resource groups.

• Real-time data synchronization tasks do not support tables without a primary key.

• Synchronizing data from a source to a MaxCompute external table is not supported.

• When data is synchronized in real time to the default MaxCompute data source (usually odps_first), a temporary AccessKey is used by default. This temporary AccessKey automatically expires after seven days, causing the task to fail. When the platform detects that a task failed due to the temporary AccessKey, it automatically restarts the task. If you have configured this type of monitoring alert for the task, you will receive an alert.

• On the same day you configure a one-click real-time synchronization task to MaxCompute, you can query only the historical full data. You can query incremental data in MaxCompute only after it is merged the next day.

• A one-click real-time synchronization task to MaxCompute generates a full data partition each day. To prevent excessive storage consumption, the automatically created MaxCompute table has a default 30-day lifecycle. If this duration is unsuitable, you can modify the lifecycle by clicking the MaxCompute table name during task configuration.

• Data Integration uses the data synchronization channels of the MaxCompute engine for data uploads and downloads. For information about the service level agreement (SLA) for these channels, see Data transfer service (upload) scenarios and tools. Evaluate your data synchronization methods based on the SLA of these channels.

• When using one-click real-time synchronization to MaxCompute in instance mode, the exclusive resource group for Data Integration requires at least 8 vCPUs and 16 GiB of memory.

• Data Integration supports only user-created MaxCompute data sources in the same region as the current workspace. Although a connection test for a data source in a different region may succeed, the synchronization task will fail during table creation and report an "engine does not exist" error.

• When MaxCompute is the destination for whole-database synchronization, it supports only the whole-database full and incremental task and the real-time incremental stream mode for regular tables. If the destination table is a DeltaTable, both real-time synchronization and the whole-database full and incremental task are supported.

  Note

  When you use a user-created MaxCompute data source, you must still associate a MaxCompute engine with your DataWorks project. Otherwise, MaxCompute SQL nodes cannot be created, which prevents the 'done' marker node for full synchronization from being created.

MaxCompute 1.0 data types

MaxCompute 2.0 and Hive-compatible data types

Data type conversion

The following table describes the data type conversion rules for MaxCompute Reader.

Connect to MaxCompute and enable project-level configurations

• Log on to the MaxCompute client. For more information, see Connect using the local client (odpscmd).

• Enable the required project-level configurations in MaxCompute. These operations require project owner permissions. For more information about MaxCompute permissions, see Role planning.

Enable ACID semantics

To enable ACID semantics, run the following command on the client as a project owner. For more information about MaxCompute ACID semantics, see ACID semantics.

setproject odps.sql.acid.table.enable=true;

(Optional) Enable V2.0 data types

To use the TIMESTAMP data type, run the following command on the client as a project owner to enable MaxCompute V2.0 data types.

setproject odps.sql.type.system.odps2=true;

(Optional) Account authorization

When you bind a MaxCompute compute engine to a workspace, a MaxCompute data source is automatically created in DataWorks. You can use this data source for data synchronization within the current workspace. To synchronize data from this MaxCompute data source in another workspace, ensure the access account for the data source in the other workspace has the required permissions to access this MaxCompute compute engine. For information about cross-account authorization, see Cross-account authorization (MaxCompute and Hologres).

Single-table batch synchronization task

• See Configure batch synchronization in the codeless UI and Use the code editor.

• For a full list of parameters and a sample script for the code editor, see Appendix: Sample script and parameters.

Single-table real-time synchronization task

See Configure a single-table real-time synchronization task.

Full-database synchronization task

See Configure a full-database batch synchronization task, Configure a full-database real-time synchronization task, and Configure a full-database full and incremental synchronization task.

Configure a batch synchronization task by using the code editor

If you want to configure a batch synchronization task by using the code editor, you must configure the related parameters in the script based on the unified script format requirements. For more information, see Use the Code Editor. The following information describes the parameters that you must configure for data sources when you configure a batch synchronization task by using the code editor.

Reader script demo

{
    "type":"job",
    "version":"2.0",
    "steps":[
        {
            "stepType":"odps",// The plugin name.
            "parameter":{
                "partition":[],// The partitions to read data from.
                "isCompress":false,// Specifies whether to compress data.
                "datasource":"",// The data source.
                "column":[// The columns in the source table.
                    "id"
                ],
                "where": "",// The WHERE clause for data filtering.
                "enableWhere":false,// Specifies whether to use a WHERE clause for data filtering.
                "table":""// The table name.
            },
            "name":"Reader",
            "category":"reader"
        },
        { 
            "stepType":"stream",
            "parameter":{
            },
            "name":"Writer",
            "category":"writer"
        }
    ],
    "setting":{
        "errorLimit":{
            "record":"0"// The error record count.
        },
        "speed":{
            "throttle":true,// Specifies whether to enable throttling. If this parameter is set to true, the mbps parameter takes effect.
            "concurrent":1, // The job concurrency.
            "mbps":"12"// The throttling rate. 1 mbps = 1 MB/s.
        }
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
            }
        ]
    }
}

To specify a MaxCompute Tunnel endpoint, you can manually configure the data source in script mode. Replace the "datasource":"", line in the preceding sample with your data source parameters. For example:

"accessId":"*******************",
"accessKey":"*******************",
"endpoint":"http://service.eu-central-1.maxcompute.aliyun-inc.com/api",
"odpsServer":"http://service.eu-central-1.maxcompute.aliyun-inc.com/api", 
"tunnelServer":"http://dt.eu-central-1.maxcompute.aliyun.com", 
"project":"*****", 

Reader script parameters

Writer script demo

The following code is a sample configuration.

{
    "type":"job",
    "version":"2.0",// The version number.
    "steps":[
        {
            "stepType":"stream",
            "parameter":{},
            "name":"Reader",
            "category":"reader"
        },
        {
            "stepType":"odps",// The plugin name.
            "parameter":{
                "partition":"",// The destination partition.
                "truncate":true,// The cleanup rule.
                "isCompress":false,// Specifies whether to compress data.
                "datasource":"odps_first",// The data source name.
            "column": [// The destination column names.
                "id",
                "name",
                "age",
                "sex",
                "salary",
                "interest"
                ],
                "table":""// The destination table name.
            },
            "name":"Writer",
            "category":"writer"
        }
    ],
    "setting":{
        "errorLimit":{
            "record":"0"// The error record count, which is the maximum number of dirty data records allowed.
        },
        "speed":{
            "throttle":true,// Specifies whether to enable throttling. If this parameter is set to true, the mbps parameter takes effect.
            "concurrent":1, // The job concurrency.
            "mbps":"12"// The throttling rate. 1 mbps = 1 MB/s.
        }
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
            }
        ]
    }
}

To specify a MaxCompute Tunnel endpoint, you can manually configure the data source in script mode. Replace the "datasource":"", line in the preceding sample with your data source parameters. The following code provides an example.

"accessId":"<yourAccessKeyId>",
 "accessKey":"<yourAccessKeySecret>",
 "endpoint":"http://service.eu-central-1.maxcompute.aliyun-inc.com/api",
 "odpsServer":"http://service.eu-central-1.maxcompute.aliyun-inc.com/api", 
"tunnelServer":"http://dt.eu-central-1.maxcompute.aliyun.com", 
"project":"**********", 

Writer script parameters