Access Alibaba Cloud VPC

Updated at:

By default, MaxCompute Spark cannot access services in a virtual private cloud (VPC). To access services in a VPC, such as specific IP addresses, domain names, ApsaraDB RDS, HBase clusters, or Hadoop clusters, you must establish network connectivity between MaxCompute and the VPC.

Limitations

  • Account limits

    The MaxCompute project that runs your Spark job and the target VPC must belong to the same Alibaba Cloud account. In other words, the UID of the Alibaba Cloud account that owns the project must match the UID of the account that owns the VPC. Otherwise, the job fails with the following error: "You are not allowed to use this vpc - vpc owner and project owner must be the same person".

  • This procedure describes how to connect to a single VPC. If your job needs to access multiple VPCs simultaneously, you can link the connected VPC to additional VPCs using a leased line and an Elastic Network Interface (ENI). For more information, contact Alibaba Cloud VPC technical support.

Procedure

For more information, see VPC access solution (leased-line direct connection).

Step 1: Prepare account and project

Before you create a network connection between MaxCompute and the target service, ensure that you meet the following requirements:

  1. Create a MaxCompute project. For data lakehouse use cases, we recommend setting the project's data type edition to Hive-compatible.

  2. If you need to access a target service in a VPC, ensure that the same Alibaba Cloud account is used for the following: the VPC, the MaxCompute project, and the administration of the target service.

Step 2: Grant permissions

  • Grant permissions to the operating user:

    • The user who grants the authorization must be the project owner (Project Owner) or a user with the tenant-level Super_Administrator or Admin role. For more information, see Role planning.

    • For more information about the authorization process, see List of object permissions in a tenant.

  • Grant permissions to MaxCompute: Authorize MaxCompute to create an elastic network interface (ENI) in your VPC. This allows MaxCompute to connect to your VPC. To grant this permission, click Authorize while you are logged on to your Alibaba Cloud account.

Step 3: Add security group rules

In the VPC instance, create a dedicated security group to control MaxCompute's access to resources within the VPC.

We recommend that you create a new Basic Security Group and avoid using existing or other types of security groups. MaxCompute creates an elastic network interface (ENI) in your VPC to access your services.

  • Configure the outbound rules of the security group to control the destination addresses that MaxCompute jobs (via the ENI) can access. If you have no special requirements, you can keep the default outbound rules.

  • Because traffic entering the ENI is return traffic, you must allow all inbound traffic.

  1. Log in to the Virtual Private Cloud console.

  2. In the navigation pane on the left, choose VPC. In the top-left corner, select a region.

  3. On the VPC page, click the Instance ID/Name of the target VPC.

  4. On the VPC details page, click the Resource Management tab.

  5. On the Resource Management tab, in the VPC Resources section, hover over the number next to Security Group and click Add.

    • For Security group type, select Basic Security Group.

      A Basic Security Group allows outbound traffic by default. An Advanced Security Group denies outbound traffic by default, which prevents access to any services in the VPC.

    • For Network, select the same VPC that the target service uses.

    For more information about how to create a security group, see Create a security group.

  6. Configure the security group's rules to support the MaxCompute network.

    1. In the Actions column of the target security group, click Manage rules.

    2. On the Access rules page, select the Inbound tab. Click Edit in the Actions column for the target rule. Configure the rule to allow all inbound traffic with the following settings:

      • Set Action to Allow.

      • Set Priority to 1.

      • Set Protocol type to All.

      • For Source, enter the CIDR block of the VPC or the vSwitch where the target cloud service is located.

      • Destination defaults to ALL(-1/-1).

    For more information about the configuration, see Security group use cases.

  7. If the service does not support security group authorization, you can add the IP address of the ENI created by MaxCompute to a whitelist. Because the ENI IP address can change, we recommend that you add the CIDR block of the vSwitch for the VPC instance to the whitelist instead. To obtain the ENI IP address, log on to the ECS console and click Elastic network interfaces in the left-side navigation pane.

Note

During the network connection creation process, MaxCompute automatically creates two ENIs based on bandwidth requirements at no charge. These ENIs reside in this security group.

Step 4: Create a network connection between MaxCompute and the target VPC

An Alibaba Cloud account or a RAM user with the tenant-level Super_Administrator or Admin role can create a connection between MaxCompute and a VPC in the MaxCompute console. For more information about roles, see MaxCompute tenant-level roles. To create a connection, follow these steps:

  1. Log in to the MaxCompute console and select a region in the upper-left corner.

  2. In the left-side navigation pane, choose Manage Configurations > Network Connection .

  3. On the Network Connection page, click Add Network Connection.

  4. In the Add Network Connection dialog box, configure the parameters as prompted and click OK. When you add a network connection for the first time, you must first grant authorization to allow the MaxCompute platform proxy to request network interface cards. Otherwise, the connection cannot be created.

  5. In the Add Network Connection dialog box, configure the parameters as prompted and click OK. When you add a network connection for the first time, you must first grant authorization to allow the MaxCompute platform proxy to request network interface cards. Otherwise, the connection cannot be created.

    The following table describes the parameters.

    Parameter

    Required

    Description

    Connection Name:

    Required

    A custom name for the connection. The name must meet the following requirements:

    • Start with a letter.

    • Contain only letters, underscores (_), and digits.

    • Be 1 to 63 characters in length.

    Type:

    Required

    The default value is Passthrough.

    A Passthrough connection is a VPC connection.

    Region:

    Required

    The system automatically populates this parameter based on the region that you select in the upper-left corner. For more information, see Supported regions.

    VPC Selected:

    Required

    A virtual private cloud (VPC) is an isolated virtual network. It provides a secure and configurable private network space, similar to a traditional data center.
    • The ID of the VPC. To create a new VPC, see Create or delete a VPC.

    • To obtain the ID:

      1. Log in to the Virtual Private Cloud console.

      2. In the navigation pane on the left, choose VPC. In the top-left corner, select a region.

      3. On the VPC page, find the Instance ID/Name of the VPC.

      If you connect to an HBase or Hadoop cluster, you can find this information in the network connection details on the corresponding console.

    Switch:

    Required

    A vSwitch is used to create a subnet. Different vSwitches in the same VPC can communicate with each other over the internal network. You can deploy cloud resources across vSwitches in different zones to protect your application from failures in a single zone.
    • The ID of the vSwitch that is attached to the VPC. If no vSwitches are available, see Create or delete a vSwitch.

    • To obtain the ID:

      1. Log in to the Virtual Private Cloud console.

      2. In the navigation pane on the left, choose vSwitch. In the top-left corner, select a region.

      3. On the vSwitch page, find the Instance ID/Name of the vSwitch.

      If you connect to an HBase or Hadoop cluster, you can find this information in the network connection details on the corresponding console.

    Security group:

    Required

    A security group acts as a virtual firewall for your cloud resources. By managing security groups and their rules, you can implement fine-grained network isolation and access control.

    The ID of the security group. To create a security group, see Create a security group.

Step 5: Configure your job

After you set up the leased-line network connection, add the following configuration items to your spark-defaults.conf file or DataWorks configuration to allow Spark to access the VPC:

spark.hadoop.odps.cupid.eni.enable = true
# Format: region:vpcid. Replace vpcid with the target VPC ID used during leased-line setup.
spark.hadoop.odps.cupid.eni.info = regionid:vpc-**********
# Example: cn-hangzhou:vpc-bp1wth********04ug3s