How to access Apsara File Storage HDFS across VPCs by using Cloud Enterprise Network (CEN)-Apsara File Storage for HDFS(HDFS)-阿里云帮助中心

This topic describes how to use Cloud Enterprise Network (CEN) to access an Apsara File Storage for HDFS file system across different regions and Virtual Private Clouds (VPCs) within the same Alibaba Cloud account.

Prerequisites

Create a file system. For more information, see Create a file system.
Add a mount point. For more information, see Add a mount target.
Configure a Hadoop environment on the ECS instance that you will use to access Apsara File Storage for HDFS. For more information, see Configure Hadoop and Deploy dependencies.

Background information

Cloud Enterprise Network (CEN) is a highly available network service that runs on Alibaba Cloud's private global network. It uses a Transit Router (TR) to establish private network connections between VPCs across different regions, and between VPCs and on-premises data centers. This helps you build a flexible, reliable, and large-scale enterprise cloud network.

By default, to mount a file system on an ECS instance, the mount point of the file system and the ECS instance must be in the same VPC. If they are in different VPCs, you must use CEN to enable communication between the VPCs before you can mount the file system.

This example describes how to connect VPC1 and VPC3 in the same region and VPC2 in a different region to the same CEN instance to enable network communication. The following table outlines the network plan for these three VPCs.

Note

When planning your network, ensure the CIDR blocks of the VPCs do not overlap and that the security group rules allow resources in the VPCs to access each other.
You can also use the multi-mount feature of Apsara File Storage for HDFS to allow a single file system instance to be accessed from multiple VPCs in the same region.

Property	VPC1	VPC2	VPC3
CIDR block	VPC CIDR block: 192.168.0.0/16 vSwitch CIDR block: 192.168.0.0/24	VPC CIDR block: 10.0.0.0/16 vSwitch CIDR block: 10.0.0.0/24	VPC CIDR block: 172.16.0.0/16 vSwitch CIDR block: 172.16.0.0/24
Region	China (Hangzhou)	China (Beijing)	China (Hangzhou)
ECS instance IP address	192.168.0.239	10.0.0.121	172.16.0.201

Create a CEN instance

Log on to the CEN console.
On the Instances page, click Create CEN Instance.

In the Create CEN Instance dialog box, configure the parameters and click OK.

Parameter	Description
Name	Enter a name for the CEN instance.
Description	Enter a description for the CEN instance.
Resource Group	Select a resource group for the CEN instance. If you do not select a resource group, the CEN instance is added to the default resource group. You can manage the Resource Groups for CEN instances and other cloud resources in the Resource Management console. For more information, see What is Resource Management?
Tag	Add tags to the CEN instance. Tag key: cannot be an empty string. The key can be up to 64 characters in length and cannot start with `aliyun` or `acs:`. It cannot contain `http://` or `https://`. Tag value: can be an empty string. The value can be up to 128 characters in length and cannot start with `aliyun` or `acs:`. It cannot contain `http://` or `https://`. You can add multiple tags to a CEN instance. For more information about tags, see Manage tags.

Connect VPC instances

Connect VPC1, VPC2, and VPC3 to the Transit Router (TR) instance in the China (Hangzhou) region. Once connected, the VPCs automatically learn route entries from each other, which enables network communication.

On the CEN Instances page, click the ID of the CEN instance.
On the Basic Information tab, in the VPC section, click the icon.
On the Connect Network Instance page, set the following parameters and click Create.
- Instance Type: Select Virtual Private Cloud (VPC).
- Region: Select the region where the network instance that you want to connect is located.
- Transit Router: The system automatically creates a TR instance in the selected region.
- Resource Owner ID: Select the type of account to which the network instance belongs. In this example, Current Account is selected.
- Network Instance: Select the network instance that you want to connect.
Repeat Step 3 to connect VPC2 and VPC3 to their respective TR instances.

After the connections are established, VPC1, VPC2, and VPC3 can automatically learn route entries from each other. Communication between VPC1 and VPC3 works as expected. For cross-region communication between VPC2 and VPC1, and between VPC2 and VPC3, CEN provides a default bandwidth of 1 Kbps for IPv4. This bandwidth is for connectivity testing only and is insufficient for regular network traffic.

Purchase a bandwidth package

To enable regular communication between the cross-region VPCs, you must purchase a bandwidth package and create a cross-region connection.

On the CEN Instances page, click the ID of the CEN instance that you created in the Create a CEN instance section.
On the details page of the CEN instance, navigate to the Basic Information > Bandwidth Package Management tab and click Purchase Bandwidth Package (Subscription).
On the purchase page, select a Product Type.
- Non-Cross-border: A bandwidth package for connections where both endpoints are either within the Chinese mainland or outside the Chinese mainland. For example, connections between two regions in the Chinese mainland, or between a region in Asia Pacific and a region in North America.
- Cross-border: A bandwidth package for connections between the Chinese mainland and regions outside the Chinese mainland. For example, a connection between a region in the Chinese mainland and a region in North America.
In this example, Non-Cross-border is selected.

Set the following parameters, click Buy Now, and complete the payment.

Parameter	Description
Cloud Enterprise Network	Select the CEN instance for which you want to purchase a bandwidth package. After you complete the payment, the bandwidth package is automatically associated with this CEN instance. In this example, the CEN instance created in the Create a CEN instance section is selected.
Area A	Select the area where the network instances are located. In this example, Chinese mainland is selected. Note You cannot change the connected areas after the bandwidth package is created. For information about the areas and regions that support bandwidth packages, see Use bandwidth packages.
Area B	Select the area where the network instances are located. In this example, Chinese mainland is selected.
Billing Method	The billing method for the bandwidth package. The default value is Pay-by-bandwidth. For more information about billing, see Billing.
Bandwidth	Select a bandwidth value for the bandwidth package based on your business requirements. Unit: Mbps.
Bandwidth Package Name	Enter a name for the bandwidth package.
Subscription Duration	Select the subscription duration. Select Auto-renewal to enable auto-renewal for the bandwidth package.
Resource Group	Select a resource group for the bandwidth package. This parameter is available only when you purchase a non-cross-border bandwidth package.

Create a cross-region connection

On the CEN Instances page, click the ID of the CEN instance that you created in the Create a CEN instance section.
On the Basic Information > Bandwidth Package Management tab, click Set Cross-region Bandwidth.

On the Connect Network Instance page, set the following parameters for the cross-region connection and click Create.

Parameter	Description
Instance Type	Select Cross-region Connection.
Region	The region that you want to connect. In this example, China (Hangzhou) is selected.
Transit Router	The ID of the Transit Router in the region selected above. This field is automatically populated.
Peer Region	The peer region that you want to connect. In this example, China (Beijing) is selected.
Transit Router	The ID of the Transit Router in the peer region selected above. This field is automatically populated.
Bandwidth Package Instance	Select a bandwidth package instance that is associated with the CEN instance.
Bandwidth	Enter the bandwidth for the cross-region connection, in Mbps.

Access Apsara File Storage for HDFS across VPCs

After you complete the preceding steps, VPC1, VPC2, and VPC3 can communicate with each other. The following steps describe how to test the connectivity from an ECS instance to the Apsara File Storage HDFS file system.

Log on to the ECS instance in VPC2. For more information, see ECS remote connection guide.
Verify the connectivity between the ECS instance and Apsara File Storage for HDFS.
```
ping f-xxxxxx.cn-xxxx.dfs.aliyuncs.com
```
In the command, f-xxxxxx.cn-xxxx.dfs.aliyuncs.com is the mount point domain name of the Apsara File Storage for HDFS file system.
Verify that the ECS instance can access the Apsara File Storage for HDFS mount point on port 10290.
```
telnet f-xxxxxx.cn-xxxx.dfs.aliyuncs.com 10290
```

Verify that the ECS instance can access files on Apsara File Storage for HDFS through the Apsara File Storage for HDFS mount point.

hadoop fs -ls  dfs://f-xxxxxx.cn-xxxx.dfs.aliyuncs.com:10290/
hadoop fs -mkdir  dfs://f-xxxxxx.cn-xxxx.dfs.aliyuncs.com:10290/test_dir
hadoop fs -touchz  dfs://f-xxxxxx.cn-xxxx.dfs.aliyuncs.com:10290/test_dir/words
echo "hello world" | hadoop fs -appendToFile - dfs://f-xxxxxx.cn-xxxx.dfs.aliyuncs.com/test_dir/words
hadoop fs -cat  dfs://f-xxxxxx.cn-xxxx.dfs.aliyuncs.com/test_dir/words

The preceding commands verify access from the ECS instance in VPC2 to the Apsara File Storage for HDFS file system. You can perform similar tests for connectivity between the ECS instance in VPC3 and the file system.