Import data by using DataWorks

Procedure

1. You have created a MaxCompute project and created a table to store the imported data.

2. You have created a DataWorks workspace and associated a MaxCompute compute resource.

3. Import data.

   Import local files

   1. Log in to the DataWorks console and select a region in the upper-left corner.

   2. In the navigation pane on the left, choose Data Integration > Data Upload and Download.

   3. In the left navigation bar, click the upload icon , and click Data Upload.

   4. Follow the on-screen instructions to upload your data.

      You can import CSV, XLS, XLSX, and JSON files into MaxCompute by using Local File or OSS. Limits:

      • Local File: The maximum size for a CSV file is 5 GB. For other file types, the maximum size is 100 MB.

      • OSS: You can upload data only from a bucket that is in the same region as your MaxCompute project.

   Upload data.

   For older workspaces, upload local CSV or custom text files through the Upload data feature.

   Import from other data sources

   1. Log in to the DataWorks console and select a region in the upper-left corner.

   2. In the navigation pane on the left, choose Data Development and O&M > Data Development.

   3. Select Workspace, and click Go to Data Studio.

   4. On the left side of DataStudio, click , and select Create Node > Data Integration > Batch Synchronization or Real-time synchronization.

      • Batch synchronization node: Sets the data destination to MaxCompute and the data source to other data sources.

      • Real-time synchronization node: Sets the output to MaxCompute and the input to other data sources.

      Configure a synchronization node in wizard mode, Configure a synchronization node in script mode, and Configure a real-time synchronization task (obsolete).

   5. Return to the DataWorks console.

      In the navigation pane on the left, choose Data Integration > Data Integration.

   6. In the left-side navigation pane, select Sync Task, and click Create Synchronization Task. In the Create Sync Task dialog box that appears, configure the data source information.

      Configure a real-time synchronization task for an entire database.

Synchronization capabilities

Data Integration synchronizes data from various sources, such as RDS databases, to MaxCompute. Capabilities vary by synchronization scenario.

• Batch synchronization uses Reader and Writer plugins to read from and write to data sources.

  • Each batch synchronization node can import data from one or more source tables into a single MaxCompute table.

• Real-time synchronization combines multiple input and output data sources into a synchronization link for real-time incremental synchronization of a single table or an entire database.

• Synchronization solutions cover scenarios such as full-database batch synchronization and full-and-incremental real-time synchronization.

The following table lists the capabilities supported when MaxCompute is used as the destination.

For more information about MaxCompute synchronization capabilities, see MaxCompute data source.

Billing

Data Integration requires resource groups for data integration and scheduling. Shared and exclusive resource groups are available. Data transfers over the public network may incur internet data transfer fees.

• Data Integration resource groups: Billing of subscription exclusive resource groups for Data Integration and Pay-as-you-go shared resource groups for Data Integration (for debugging).

• Internet data transfer: Internet data transfer billing.

• Scheduling resource groups: Billing of subscription exclusive resource groups for scheduling and Billing of shared resource groups for scheduling.

Best practices