Preparations

Step 1. Create a destination MaxCompute project

A project is the basic organizational unit in MaxCompute.

• Before you migrate data, you must select a destination MaxCompute project for the source data.

• During the migration, the MaxCompute project runs various migration jobs.

Therefore, you must create a MaxCompute project before migrating data. If you have an existing project, you can skip this step. To create a project in the MaxCompute console, follow these steps:

1. Log in to the MaxCompute console and select a region in the upper-left corner.

2. In the left-side navigation pane, choose Manage Configurations > Projects.

3. On the Internal Project tab, click Create Project.

4. In the Create Project dialog box, configure the settings and click OK.

   For more information about the parameters, see Create a MaxCompute project.

Step 2. Set up a VPC network connection

If you already have a VPC network connection in the current region, you can skip this step.

1. Log in to the MaxCompute console and select a region in the upper-left corner.

2. In the left-side navigation pane, choose Manage Configurations > Network Connection .

3. On the Network Connection page, click Add Network Connection.

4. In the Add Network Connection dialog box, configure the parameters as prompted and click OK. When you add a network connection for the first time, you must first grant authorization to allow the MaxCompute platform proxy to request network interface cards. Otherwise, the connection cannot be created.

   The following table describes the parameters.

   Parameter	Required	Description
   Connection Name:	Required	A custom name for the connection. The name must meet the following requirements: Start with a letter. Contain only letters, underscores (_), and digits. Be 1 to 63 characters in length.
   Type:	Required	The default value is Passthrough. A Passthrough connection is a VPC connection.
   Region:	Required	The system automatically populates this parameter based on the region that you select in the upper-left corner. For more information, see Supported regions.
   VPC Selected:	Required	A virtual private cloud (VPC) is an isolated virtual network. It provides a secure and configurable private network space, similar to a traditional data center. The ID of the VPC. To create a new VPC, see Create or delete a VPC. To obtain the ID: Log in to the Virtual Private Cloud console. In the navigation pane on the left, choose VPC. In the top-left corner, select a region. On the VPC page, find the Instance ID/Name of the VPC. If you connect to an HBase or Hadoop cluster, you can find this information in the network connection details on the corresponding console.
   Switch:	Required	A vSwitch is used to create a subnet. Different vSwitches in the same VPC can communicate with each other over the internal network. You can deploy cloud resources across vSwitches in different zones to protect your application from failures in a single zone. The ID of the vSwitch that is attached to the VPC. If no vSwitches are available, see Create or delete a vSwitch. To obtain the ID: Log in to the Virtual Private Cloud console. In the navigation pane on the left, choose vSwitch. In the top-left corner, select a region. On the vSwitch page, find the Instance ID/Name of the vSwitch. If you connect to an HBase or Hadoop cluster, you can find this information in the network connection details on the corresponding console.
   Security group:	Required	A security group acts as a virtual firewall for your cloud resources. By managing security groups and their rules, you can implement fine-grained network isolation and access control. The ID of the security group. To create a security group, see Create a security group.

5. Migration jobs need to access your data source. Ensure that the VPC can access the internet or the private connection to your data source.

   • If your data source is on the public internet, use a NAT Gateway to provide internet access. For more information, see Use a NAT Gateway for outbound internet traffic.

   • If your data source is in an on-premises data center, connect it to Alibaba Cloud by using an Express Connect circuit. For more information, see Express Connect.

Step 3. Create a service-linked role

Before you use MMS for the first time, you must create a service-linked role. MMS uses the AliyunServiceRoleForMaxComputeMMS role to access your resources in MaxCompute and other Alibaba Cloud services.

1. If you are using your Alibaba Cloud account, you can create the service-linked role directly.

   If you are using a RAM user, you must first grant it the AliyunRAMFullAccess policy. For more information, see Manage permissions of a RAM role.

2. You can create the service-linked role in the MaxCompute console or the Resource Access Management (RAM) console.

   MaxCompute console

   1. Log in to the MaxCompute console and select a region in the upper-left corner.

   2. In the left-side navigation pane, choose Data Transfer > Migration Service.

   3. On the Data Source tab, click Add Data Source.

   4. To create the service-linked role, click OK in the MaxCompute Service-linked Role dialog box. If this dialog box does not appear, the role already exists.

   RAM console

   1. Log in to the RAM console.

   2. In the left navigation bar, select Identities > Roles.

   3. On the Roles page, click Create Role.

   4. In the upper-right corner of the Create Role page, click Create Service Linked Role.

   5. On the Create Service Linked Role page, under Select Service, select AliyunServiceRoleForMaxComputeMMS, and click Create Service Linked Role.

Step 4. Configure permissions

1. Grant the service-linked role permissions on MaxCompute data

   MaxCompute console

   1. Log in to the MaxCompute console and select a region in the upper-left corner.

   2. In the left-side navigation pane, choose Manage Configurations > Projects.

   3. On the Projects page, click Manage in the Actions column for the target project.

   4. Configure data operation permissions

      Admin role

      1. On the Project Settings page, click the Role Permissions tab.

      2. Find the admin role and click Manage Members in the Actions column.

      3. In the Manage Members dialog box, enter RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS in the Add Member Manually field. Replace <account_id> with your Alibaba Cloud account ID.

      4. Click OK.

      Dedicated role for MMS

      Create a dedicated project-level role in MaxCompute and grant it only the data operation permissions required for MMS.

      1. On the Project Settings page, click the Role Permissions tab.

      2. Click Create Project-level Role, configure the parameters as prompted, and click Confirm.

         The following table describes the parameters.

         Parameter	Required	Description
         Role Name	Required	The name must be 1 to 64 characters in length, and can contain letters and digits. The name must start with a letter.
         Role Type	Required	Admin: A management role. You can grant management permissions by using a policy, but you cannot grant resource permissions. You cannot grant management permissions by using an ACL. Resource: A resource-type role. You can grant resource permissions by using a policy or an ACL, but you cannot grant management permissions.
         Authorization Method	Required	Access Control List (ACL): A whitelist-based authorization mechanism that allows users or roles to perform specific actions on specific objects. Policy: A mechanism that supports both allow and deny rules, suitable for large enterprises and complex scenarios that require fine-grained control.

         Grants a user permissions to perform operations on projects, tables, models, resources, functions, or instances. The following table lists the permissions that you can grant.

         Object	Permissions
         Table	Describe, Select, Update, Alter, Drop, ShowHistory, Download
         Resource	Read, Write, Download, Delete
         Function	Read, Write, Download, Execute, Delete
         Package	Read
         Project	Read, Write, List, CreateTable, CreateInstance

   5. Return to the Role Permissions tab, find the newly created project-level role, and click Manage Members in the Actions column. Select the desired Alibaba Cloud account or RAM user, and click OK to grant the permissions. If you cannot find the account by searching, you can add it in the Add Member Manually section.

   SQL

   1. Connect to MaxCompute by using the MaxCompute client (odpscmd) and add the service-linked role as a user to your project.

      USE <target_projectname>;
      
      -- Add the service-linked role for the user.
      Add USER  `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;

   2. Grant permissions to the user.

      MaxCompute supports access control at different granularities. You can grant permissions based on the operational scope of users in your project.

      • Coarse-grained authorization: Grant the admin role directly to the user. This avoids repeatedly granting permissions to the same user for each table.

        GRANT admin TO USER `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;

      • Fine-grained authorization: Grant permissions at the project, table, or instance level.

        Project-level permissions

        -- Grant project-level data operation permissions.
        GRANT <actions> 
          ON project <project_name> TO USER `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;
        -- Supported actions
        Read,Write,List,CreateTable,CreateInstance,CreateFunction,CreateResource,All
        -- Example 1
        GRANT Read,Write,List,CreateTable,CreateInstance,CreateFunction,CreateResource
          ON project <project_name> TO USER `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;
        -- Example 2
        GRANT ALL 
          ON project <project_name> TO USER `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;

        Table-level permissions

        -- Grant table-level data operation permissions.
        GRANT <actions> 
          ON table <table_name> [(<column_list>)] TO USER `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;
        -- Supported actions
        Describe,Select,Alter,Update,Drop,ShowHistory,All
        -- Example 1
        GRANT Describe,Select,Alter,Update,Drop,ShowHistory
          ON table <table_name> [(<column_list>)] TO USER `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;
        -- Example 2
        GRANT All
          ON table <table_name> [(<column_list>)] TO USER `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;

        Instance-level permissions

        -- Grant instance-level data operation permissions. 
        GRANT <actions> 
          ON instance <instance_id> [(<column_list>)] TO USER `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;
        -- Supported actions
        Read,Write,All 
        
        -- Example 1
        GRANT Read,Write 
          ON instance <instance_id> [(<column_list>)] TO USER `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;
        
        -- Example 2
        GRANT All 
          ON instance <instance_id> [(<column_list>)] TO USER `RAM$<account_id>:role/AliyunServiceRoleForMaxComputeMMS`;
        

   Parameter descriptions:

   • target_projectname: Required. The name of the destination project.

   • account_id: Required. Your Alibaba Cloud account ID.

   • privileges: Required. The name of the permission to grant. You can specify multiple actions in a single grant statement. Separate multiple action names with a comma (,).

2. Configure permissions for a RAM user

   This section describes how to grant a RAM user the permissions to perform MMS operations, such as managing data sources, metadata, and migration jobs. If you perform data migration by using your Alibaba Cloud account, you can skip this section.

   • If the RAM user already has the AliyunMaxComputeFullAccess policy, you can skip this section.

   • To grant the RAM user permissions only for MMS operations, follow these steps:

     1. Log in to the RAM console.

     2. In the left navigation bar, select Manage Permissions > Policies.

     3. On the Policies page, click Create Policy.

     4. On the Create Policy page, select JSON Editor. Enter the following policy:

        All MMS operations

        {
          "Version": "1",
          "Statement": [
            {
              "Effect": "Allow",
              "Action": [
                "odps:ListMmsDataSources",
                "odps:CreateMmsDataSource",
                "ram:GetRole",
                "odps:GetMmsDataSource",
                "odps:UpdateMmsDataSource",
                "odps:DeleteMmsDataSource",
                "odps:CreateMmsFetchMetadataJob",
                "odps:GetMmsFetchMetadataJob",
                "odps:ListMmsFetchMetadataJobLogs",
                "odps:ListMmsDbs",
                "odps:GetMmsDb",
                "odps:ListMmsTables",
                "odps:GetMmsTable",
                "odps:ListMmsPartitions",
                "odps:GetMmsPartition",
                "odps:ListMmsJobs",
                "odps:GetMmsJob",
                "odps:CreateMmsJob",
                "odps:DeleteMmsJob",
                "odps:StartMmsJob",
                "odps:StopMmsJob",
                "odps:RetryMmsJob",
                "odps:ListMmsTasks",
                "odps:GetMmsTask",
                "odps:ListMmsTaskLogs",
                "odps:StopMmsTask",
                "odps:StartMmsTask",
                "odps:RetryMmsTask",
                "odps:GetMmsAsyncTask",
                "odps:GetMmsProgress",
                "odps:GetMmsSpeed",
                "odps:CreateMmsAuthFile",
                "odps:ListMmsAgents",
                "odps:ListMmsTimers",
                "odps:GetMmsTimer",
                "odps:UpdateMmsTimer",
                "odps:ListMmsTimerLogs",
                "odps:CreateMmsTimer",
                "odps:UpdateMmsTables",
                "odps:UpdateMmsTable",
                "odps:UpdateMmsDb",
                "odps:ListNetworkLinks"
              ],
              "Resource": "*"
            }
          ]
        }

        MMS source data management

        {
          "Version": "1",
          "Statement": [
            {
              "Effect": "Allow",
              "Action": [
                "odps:ListMmsDataSources",
                "odps:CreateMmsDataSource",
                "ram:GetRole",
                "odps:GetMmsDataSource",
                "odps:UpdateMmsDataSource",
                "odps:DeleteMmsDataSource",
                "odps:CreateMmsFetchMetadataJob",
                "odps:GetMmsFetchMetadataJob",
                "odps:ListMmsFetchMetadataJobLogs",
                "odps:ListMmsDbs",
                "odps:GetMmsDb",
                "odps:ListMmsTables",
                "odps:GetMmsTable",
                "odps:ListMmsPartitions",
                "odps:GetMmsPartition",
                "odps:GetMmsAsyncTask",
                "odps:GetMmsProgress",
                "odps:GetMmsSpeed",
                "odps:CreateMmsAuthFile",
                "odps:ListMmsAgents",
                "odps:UpdateMmsTables",
                "odps:UpdateMmsTable",
                "odps:UpdateMmsDb"
              ],
              "Resource": "*"
            }
          ]
        }

        MMS migration job management

        {
          "Version": "1",
          "Statement": [
            {
              "Effect": "Allow",
              "Action": [
                "odps:ListMmsDataSources",
                "odps:GetMmsDataSource",
                "odps:CreateMmsFetchMetadataJob",
                "odps:GetMmsFetchMetadataJob",
                "odps:ListMmsFetchMetadataJobLogs",
                "odps:ListMmsDbs",
                "odps:GetMmsDb",
                "odps:ListMmsTables",
                "odps:GetMmsTable",
                "odps:ListMmsPartitions",
                "odps:GetMmsPartition",
                "odps:ListMmsJobs",
                "odps:GetMmsJob",
                "odps:CreateMmsJob",
                "odps:DeleteMmsJob",
                "odps:StartMmsJob",
                "odps:StopMmsJob",
                "odps:RetryMmsJob",
                "odps:ListMmsTasks",
                "odps:GetMmsTask",
                "odps:ListMmsTaskLogs",
                "odps:StopMmsTask",
                "odps:StartMmsTask",
                "odps:RetryMmsTask",
                "odps:GetMmsAsyncTask",
                "odps:GetMmsProgress",
                "odps:GetMmsSpeed",        
                "odps:ListMmsTimers",
                "odps:GetMmsTimer",
                "odps:UpdateMmsTimer",
                "odps:ListMmsTimerLogs",
                "odps:CreateMmsTimer"
              ],
              "Resource": "*"
            }
          ]
        }

     5. Click OK. In the Create Policy dialog box, enter a custom Policy Name. You can also add a description and tags.

     6. Attach the custom MMS policy to the RAM user. For more information, see Manage permissions of a RAM user.

Step 5. Set up a quota

Data migration jobs consume computing resources. You must associate a computing quota with the MaxCompute project that runs the migration jobs.

1. Log in to the MaxCompute console and select a region in the upper-left corner.

2. In the left-side navigation pane, choose Manage Configurations > Quotas .

3. On the Quotas page, click New Quota.

   On the purchase page, select Data Transmission Service as the specification type.

   We recommend that you purchase a subscription computing quota to reduce migration costs.

   If you already have a Data Transmission Service quota, you can skip this step.

4. Assign a computing quota to the MaxCompute project that runs the migration jobs.

   • Use the default computing quota:

     By default, a project is assigned a computing quota for its jobs. No additional configuration is required. You can associate a new project with this default quota during creation.

   • Switch the computing quota:

     1. In the left-side navigation pane, choose Manage Configurations > Projects.

     2. On the Projects page, click Manage in the Actions column for the target project.

     3. On the Project Settings page, click the Parameter Configuration tab.

     4. In the Basic Properties section, click Edit to modify the configuration as needed.

     5. In the Authorized Tunnel Quota drop-down list, select the computing quota for the project and click Submit.

5. When you add a data source, select the project to which you assigned the computing quota as the Project to run migration jobs.

   1. In the left-side navigation pane, choose Data Transfer > Migration Service.

   2. On the Migration Service page, click the Data Source tab.

   3. On the Data Source tab, click Add Data Source.

   4. From the MaxCompute project to run migration jobs in drop-down list, select the project to which you assigned the computing quota.