Apsara File Storage for HDFS
Apsara File Storage for HDFS is a file storage service for compute resources, such as Alibaba Cloud ECS instances and container services. You can manage and access data in Apsara File Storage for HDFS in the same way as a Hadoop Distributed File System (HDFS). You do not need to modify your existing big data analytics applications to use a distributed file system that features unlimited capacity, performance scaling, a single namespace, shared access, high reliability, and high availability.
Scenarios
Apsara File Storage for HDFS is ideal for big data analytics and machine learning scenarios that require high throughput. It provides high-throughput, low-latency access, eliminating the need to migrate data to local compute resources.
After you store data in Apsara File Storage for HDFS, compute resources such as ECS instances can access the data directly. You can deploy Hadoop or other machine learning applications on multiple compute resources. These applications can then access data through the Hadoop FileSystem interface to perform offline or online computing. You can also write the computing results directly to Apsara File Storage for HDFS for permanent retention.
Performance
The main performance metric for Apsara File Storage for HDFS is throughput capacity. The maximum throughput is limited by the bandwidth of the ECS instance. For example, if an ECS instance has a bandwidth of 1.5 Gbps, the maximum throughput can reach 187.5 MB/s. Throughput capacity is related to the purchased storage capacity. For more information, see Cluster throughput performance test.
Data durability and service availability
Similar to File Storage NAS, Apsara File Storage for HDFS stores multiple data replicas on the backend across different devices and fault domains. This design provides 99.999999999% (eleven 9s) of data reliability and effectively reduces the risk of data loss.
Scalability and elasticity
Apsara File Storage for HDFS provides high throughput, high input/output operations per second (IOPS), and low-latency storage performance for application payloads. Its performance scales linearly with capacity. This allows the service to meet the increasing demands for capacity and storage performance as your business grows.
Security
Apsara File Storage for HDFS provides multiple security mechanisms to protect your file system data. These mechanisms include standard file system access control, permission group-based access control, and access control for Resource Access Management (RAM) users.
Interfaces
The Apsara File Storage for HDFS software development kit (SDK) includes a file system SDK and a management system SDK. During the current public preview, only the file system SDK is available. You can perform management operations in the console. The Apsara File Storage for HDFS SDK implements the Hadoop FileSystem interface to provide a Hadoop-compatible file system. It is packaged as a single JAR file named aliyun-sdk-dfs-x.y.z.jar. Using this SDK, Apache Hadoop data analytics applications, such as MapReduce, Hive, Spark, and Flink, can use Apsara File Storage for HDFS as the default file system (defaultFS) without code modification or recompilation. This provides functional and performance advantages over the original HDFS.
If you prefer to use a graphical web application, you can use the management console to manage your HDFS file systems.
Cost model
The billing for Apsara File Storage for HDFS is based on storage capacity and provisioned throughput. By default, the service is billed hourly based on your actual usage (pay-as-you-go). You can also purchase subscription resource plans to prepay for a specific resource quota and duration to receive greater discounts. For more information, see Pricing.