Migrate data from MaxCompute to OSS

Procedure

1. Activate services

   Ensure that MaxCompute, DataWorks, and OSS are in the same region.

   1. Activate MaxCompute.

   2. Activate DataWorks and bind a MaxCompute compute resource.

2. Create a table in DataWorks

   1. Log in to the DataWorks console and select a region in the upper-left corner.

   2. In the navigation pane on the left, choose Workspace.

   3. On the Workspaces page, in the Actions column of the target workspace, choose Shortcuts > DataStudio.

   4. On the DataStudio page, create a MaxCompute SQL node. Run the following statement to create a table named transs.

      CREATE TABLE transs
      (name    STRING,
      id    STRING,
      gender    STRING);

3. Import data into the transs table

   1. Download the sample CSV file demo_data.csv to your computer.

   2. In the navigation pane on the left, choose Data Integration > Data Upload and Download.

   3. Click Go to Data Upload and Download.

   4. In the left navigation bar, click the upload icon , and click Data Upload.

      Upload the data file to the MaxCompute table named transs.

4. Prepare the destination in OSS

   1. Log in to the OSS console.

   2. In the left-side navigation pane, click Buckets.

   3. On the Buckets page, click Create Bucket.

   4. Create an empty file named qwee.csv on your computer and upload it to the bucket. This file will be populated with the data from MaxCompute. Ensure that the field names in qwee.csv are identical to the field names in the transs table.

      For more information about how to upload a file, see Upload files.

5. Add a MaxCompute data source in DataWorks

   1. Log in to the DataWorks console and select a region in the upper-left corner.

   2. In the navigation pane on the left, choose Workspace.

   3. On the Workspaces page, click the name of the target workspace.

   4. On the Workspace Details page, click Data Sources in the navigation pane on the left.

   5. On the Data Source tab, click Add Connection, and then select MaxCompute.

   6. On the Add MaxCompute Data Source page, fill in the Basic Information.

      For more information, see Configure a MaxCompute data source.

6. Add an OSS data source in DataWorks

   For more information, see Configure an OSS data source.

7. Configure a MaxCompute (ODPS) Reader and an OSS Writer

   1. Log in to the DataWorks console and select a region in the upper-left corner.

   2. In the navigation pane on the left, choose Data Development and O&M > Data Development.

   3. Select Workspace, and click Go to Data Studio.

   4. On the left side of DataStudio, click , and select Create Node > Data Integration > Batch Synchronization.

      • For the data source, select MaxCompute (ODPS).

      • For the data destination, select OSS.

   5. On the node configuration page, configure the following parameters.

      • Data SourceAdd Method: Select the MaxCompute data source that you added.

      • Data SourceDestination: Select the OSS data source that you added.

      • SourceTable: Select the MaxCompute table that you created.

      • Destination-Text type: Select CSV.

      • Destination-File Name: The destination file name in OSS. You can include a path, such as a.txt or aaa/bbb/cc.csv.

      • Runtime Resource: Select the dedicated resource group that you created.

      • Leave the other parameters at their default values.

      Alternatively, click the icon above the configuration area to switch to script mode and configure the node using a script. The following code provides an example:

      {
          "order":{
              "hops":[
                  {
                      "from":"Reader",
                      "to":"Writer"
                  }
              ]
          },
          "setting":{
              "errorLimit":{
                  "record":"0"
              },
              "speed":{
                  "concurrent":1,
                  "dmu":1,
                  "throttle":false
              }
          },
          "steps":[
              {
                  "category":"reader",
                  "name":"Reader",
                  "parameter":{
                      "column":[
                          "name",
                          "id",
                          "gender"
                      ],
                      "datasource":"odps_first",
                      "partition":[],
                      "table":"Transs"
                  },
                  "stepType":"odps"
              },
              {
                  "category":"writer",
                  "name":"Writer",
                  "parameter":{
                      "datasource":"Trans",
                      "dateFormat":"yyyy-MM-dd HH:mm:ss",
                      "encoding":"UTF-8",
                      "fieldDelimiter":",",
                      "fileFormat":"csv",
                      "nullFormat":"null",
                      "object":"qwee.csv",
                      "writeMode":"truncate"
                  },
                  "stepType":"oss"
              }
          ],
          "type":"job",
          "version":"2.0"
      }                           

   6. Click Run.

8. Verify the data migration. After the task completes, go to the OSS console and check that the CSV file contains the data migrated from MaxCompute. For more information, see Download files.