How to use an HDFS file system on a compute node-Apsara File Storage for HDFS(HDFS)-阿里云帮助中心

This topic describes how to mount and uninstall an Apsara File Storage for HDFS file system.

Prerequisites

The Java Development Kit (JDK) 1.8 or later is installed on your ECS instance.
You have created an Apsara File Storage for HDFS file system and added a mount target. For more information, see Create a file system and Add a mount target.
A Hadoop client is installed. Hadoop 2.7.2 or later is recommended. To download Hadoop, see Hadoop Releases.

Mounting instructions

If the mount target uses the virtual private cloud (VPC) network type, you can mount the file system only on an ECS instance that resides in the same VPC. Additionally, the authorized address in the permission group attached to the mount target must include the IP address of the ECS instance.

Mount the file system

Connect to the ECS instance. For more information, see Connect to an ECS instance.
Configure the core-site.xml file.
1. Run the following command to open the core-site.xml file.
```
vim ${HADOOP_HOME}/etc/hadoop/core-site.xml
```
  If you have not configured the ${HADOOP_HOME} environment variable, run the following command to set it to your Hadoop installation directory.
```
export HADOOP_HOME=/your/hadoop/installation/directory
```
2. In the core-site.xml file, add the following configuration.
```
<property>
     <name>fs.defaultFS</name>
     <value>dfs://f-xxxxxxxx.cn-xxxxx.dfs.aliyuncs.com:10290</value>
</property>
<property>
     <name>fs.dfs.impl</name>
     <value>com.alibaba.dfs.DistributedFileSystem</value>
</property>
<property>
     <name>fs.AbstractFileSystem.dfs.impl</name>
     <value>com.alibaba.dfs.DFS</value>
</property>
```
  In the preceding configuration, replace f-xxxxxxxx.cn-xxxxx.dfs.aliyuncs.com with the mount target address of your Apsara File Storage for HDFS file system.
3. Optional: Adjust the core-site.xml configuration to optimize cluster performance. For more information, see Best practices for performance optimization.
4. Synchronize the core-site.xml file across all nodes.
Deploy the Apsara File Storage for HDFS Java software development kit (SDK).
1. Download the latest Apsara File Storage for HDFS Java SDK.
2. Add the downloaded Apsara File Storage for HDFS Java SDK to the CLASSPATH of the Hadoop Distributed File System (HDFS) component.
```
cp aliyun-sdk-dfs-x.y.z.jar  ${HADOOP_HOME}/share/hadoop/hdfs
```
3. Synchronize the Apsara File Storage for HDFS Java SDK across all nodes.

Run the following command to verify that the file system is mounted.

Run the command.
```
${HADOOP_HOME}/bin/hadoop fs -df
```

Sample output

If the output is similar to the following, the Apsara File Storage for HDFS file system is mounted.

Filesystem                                                          Size  Used       Available  Use%
dfs://f-e389e5c7e****.cn-hangzhou.dfs.aliyuncs.com:10290  10995116277760     0  10995116277760    0%
22/09/20 14:19:04 INFO dfs.DistributedFileSystem: FileSystem close com.alibaba.dfs.DistributedFileSystem@51ee79a3
22/09/20 14:19:04 INFO dfs.DfsClient: DfsClient close Client-6676663918471637276
22/09/20 14:19:04 INFO common.LeaseRenewer: close all files being written
22/09/20 14:19:04 INFO common.LeaseRenewer: LeaseRenwerForClient-6676663918471637276 closed

Uninstalling a file system

If you no longer need to use the Apsara File Storage for HDFS file system, you must uninstall it. To do so, refer to Mount file systems, delete all deployed Apsara File Storage for HDFS Java SDKs, and delete all Apsara File Storage for HDFS-related configurations from all core-site.xml files.

If you want to release the Apsara File Storage for HDFS file system instance, see Delete a file system.

Warning

After you release a file system instance, the data stored in the file system is deleted and cannot be recovered. Before you release the instance, make sure that you no longer need the data.