SDK examples

Updated at:

This topic provides Java code examples that show how to use the Hadoop FileSystem API to perform common operations on an Apsara File Storage for HDFS file system. You can use these examples to develop your application.

Prerequisites

  • You have activated the Apsara File Storage for HDFS service and created a file system instance and a mount target. For more information, see Quick Start for Apsara File Storage for HDFS.

  • You have installed Java Development Kit (JDK) 1.8 or a later version.

  • You have installed the Apsara File Storage for HDFS file system SDK. For more information, see Install the file system SDK.

Background information

Apsara File Storage for HDFS is compatible with the Hadoop FileSystem API through its file system SDK. For more information, see Hadoop FileSystem API.

Note

The Apsara File Storage for HDFS file system SDK does not yet support all Hadoop FileSystem APIs. For more information, see Limits.

Install Maven dependencies

Your application must include the following dependencies in its Maven project to operate on an Apsara File Storage for HDFS file system.

Note

If you have already deployed the Apsara File Storage for HDFS file system SDK to your application's dependency environment as described in Install the file system SDK, do not package the following dependencies into your application.

    <dependencies>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <!-- Use Hadoop 2.7.2 or a later version. -->
            <version>2.7.2</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/com.aliyun.dfs/aliyun-sdk-dfs -->
        <dependency>
            <groupId>com.aliyun.dfs</groupId>
            <artifactId>aliyun-sdk-dfs</artifactId>
            <!-- Use the latest SDK version. -->
            <version>1.0.5</version>
        </dependency>
    </dependencies>

Application initialization

Before you can access an Apsara File Storage for HDFS file system using the Hadoop FileSystem API, you must load the Apsara File Storage for HDFS configurations and then instantiate the FileSystem.

The following is a sample initialization code.

  • The core-site.xml file is configured. You can use the following code for initialization.

         private void init() throws IOException {
           conf = new Configuration();
           // Read configurations from the specified core-site.xml file.
           conf.addResource(new Path(PATH_TO_CORE_SITE_XML));
           fileSystem = FileSystem.get(conf);
         }
  • If the core-site.xml file is not configured, use the following code to load the Apsara File Storage for HDFS configurations for initialization.

         private void init() throws IOException {
           conf = new Configuration();
           // Replace DfsMountpointDomainName with the mount target domain name of your Apsara File Storage for HDFS file system. For example, xxxx.cn-hangzhou.dfs.aliyuncs.com.
           conf.set("fs.defaultFS", "dfs://DfsMountpointDomainName:10290");
           conf.set("fs.dfs.impl", "com.alibaba.dfs.DistributedFileSystem");
           conf.set("fs.AbstractFileSystem.dfs.impl", "com.alibaba.dfs.DFS");
           fileSystem = FileSystem.get(conf);
         }

Create a directory

Call the `exists` method of the `FileSystem` instance to check if the directory already exists:

  • If the directory exists, the method returns.

  • If the directory does not exist, call the `mkdirs` method of the `FileSystem` instance to create the directory.

The following code provides an example of creating a directory.

    /**
     * Create a directory.
     *
     * @param dirPath
     * @return
     * @throws IOException
     */
    private boolean createDir(final Path dirPath) throws IOException {
        if (!fileSystem.exists(dirPath)) {
            return fileSystem.mkdirs(dirPath);
        }
        return true;
    }

Move or rename

In Apsara File Storage for HDFS, moving and renaming are the same operation. Call the `rename` method of the `FileSystem` instance to move or rename a file or directory in the Apsara File Storage for HDFS file system.

The following is a code sample for moving or renaming.

    /**
     * Move or rename.
     *
     * @param srcPath
     * @param destPath
     * @return
     * @throws IOException
     */
    private boolean moveOrRename(final Path srcPath, final Path destPath) throws IOException {
        return fileSystem.rename(srcPath, destPath);
    }

Delete a file or directory

Call the exists method of the FileSystem instance to check if the file or directory exists:

  • If the path exists, call the `delete` method of the `FileSystem` instance to delete the path.

  • If the path does not exist, the method returns `false`.

The following code provides an example of deleting a file or directory.

    /**
     * Delete a file or directory.
     *
     * @param targetPath
     * @return
     * @throws IOException
     */
    private boolean deletePath(final Path targetPath, boolean recursive) throws IOException {
        if (!fileSystem.exists(targetPath)) {
            return false;
        }
        return fileSystem.delete(targetPath, recursive);
    }

Upload a file

Call the `copyFromLocalFile` method of the `FileSystem` instance to upload a file from the local file system to Apsara File Storage for HDFS.

The following code provides an example of uploading a file.

    /**
     * Upload a file.
     *
     * @param localFilePath
     * @param destFilePath
     * @throws IOException
     */
    private void uploadFile(final Path localFilePath, final Path destFilePath, boolean overwrite) throws IOException {
        fileSystem.copyFromLocalFile(false, overwrite, localFilePath, destFilePath);
    }

Download a file

Call the `copyToLocalFile` method of the `FileSystem` instance to download a file from Apsara File Storage for HDFS to the local file system.

The following code provides an example of downloading a file.

    /**
     * Download a file.
     *
     * @param srcFilePath
     * @param localFilePath
     * @throws IOException
     */
    private void downloadFile(final Path srcFilePath, final Path localFilePath) throws IOException {
        fileSystem.copyToLocalFile(srcFilePath, localFilePath);
    }

Get file or directory information

Call the `listStatus` method of the `FileSystem` instance to retrieve information about a file or directory.

The following code provides an example of retrieving file or directory information.

    /**
     * Get file or directory information.
     *
     * @param targetPath
     * @throws IOException
     */
    private void printStatus(final Path targetPath) throws IOException {
        FileStatus[] fileStatuses = fileSystem.listStatus(targetPath);
        System.out.println("# hadoop fs -ls " + targetPath.toString());
        for (FileStatus fileStatus : fileStatuses) {
            System.out.print(fileStatus.getPermission() + "\t");
            System.out.print(fileStatus.getOwner() + "\t");
            System.out.print(fileStatus.getGroup() + "\t");
            System.out.print(fileStatus.getAccessTime() + "\t");
            System.out.print(fileStatus.getPath() + "\n");
        }
        System.out.println("");
    }

Create and write to a file

Call the `create` method of the `FileSystem` instance to obtain an output stream for writing to a file. Then, write to this output stream.

The following code provides an example of creating and writing to a file.

    /**
     * Create and write to a file.
     *
     * @param filePath
     * @param content
     * @throws IOException
     */
    private void writeFile(final Path filePath, final String content) throws IOException {
        try (FSDataOutputStream out = fileSystem.create(filePath)) {
            out.write(content.getBytes());
            out.hsync();
        }
    }

Read a file

To read a file is to retrieve its content from Apsara File Storage for HDFS. You can do this by calling the open method of a FileSystem instance to obtain an input stream, and then using that stream to read the content from Apsara File Storage for HDFS.

The following code provides an example of reading a file.

    /**
     * Read a file.
     *
     * @param filePath
     * @return
     * @throws IOException
     */
    private StringBuffer readFile(final Path filePath) throws IOException {
        StringBuffer strBuffer = new StringBuffer();
        try (FSDataInputStream in = fileSystem.open(filePath);
             BufferedReader reader = new BufferedReader(new InputStreamReader(in))) {
            String tempOneLine;
            while ((tempOneLine = reader.readLine()) != null) {
                strBuffer.append(tempOneLine);
            }
            return strBuffer;
        }
    }
}