Deploy a 3FS cluster based on an ACK cluster
3FS is a high-performance distributed storage system independently developed by DeepSeek. It is designed and optimized for AI large model training, big data analytics, and high-performance computing (HPC) scenarios. 3FS integrates modern SSD storage with RDMA networking to overcome the performance bottlenecks of traditional networks. This topic describes how to deploy 3FS on Alibaba Cloud's self-developed elastic RDMA network by using Container Service for Kubernetes (ACK), a managed Kubernetes service, for rapid deployment.
Deployment solution
Deploy a 3FS cluster on elastic ephemeral disks over Alibaba Cloud eRDMA, using an ACK managed cluster for multi-node orchestration.
Alibaba Cloud does not provide technical support for 3FS or guarantee its performance. For issues, contact us on GitHub or join DingTalk group 35532895.
eRDMA is an elastic Remote Direct Memory Access (RDMA) network developed by Alibaba Cloud for the cloud. eRDMA reuses virtual private clouds (VPCs) as the underlying link and uses a congestion control (CC) algorithm that is developed by Alibaba Cloud. eRDMA features high throughput and low latency based on RDMA supports. Compared with RDMA, eRDMA implements large-scale RDMA networking within seconds. eRDMA supports traditional HPC applications, AI applications, and Transmission Control Protocol/Internet Protocol (TCP/IP) applications.
See eRDMA.
Elastic ephemeral disks provide high-performance, cost-effective temporary storage for ECS instances with custom capacities. See Elastic ephemeral disks.
Container Service for Kubernetes (ACK) provides fully managed, highly available Kubernetes clusters that simplify containerized application deployment and management.
See What is ACK?
Key 3FS components
3FS consists of the following key components:
Metadata Service (Meta): A stateless service that handles file system metadata requests. It uses FoundationDB to perform atomic operations.
Storage Service (Storage): Runs on storage nodes. Data is stored in blocks on high-performance NVMe SSDs. It uses the Chain Replication with Apportioned Queries (CRAQ) protocol to manage replicas and provides write-all-read-any semantics.
Cluster manager (Mgmtd): Manages the cluster's configuration information and storage node status. It is responsible for electing a primary node and synchronizing updates to other components.
Client: 3FS provides two types of clients that balance compatibility and efficiency, making it an ideal storage interface for AI and big data workloads. One type uses Filesystem in Userspace (FUSE) for standardized access, and the other uses USRBIO for high-performance transport.
FUSE client: Mounts the storage cluster as a local directory through the user-space file system interface. It provides POSIX-compatible file operation interfaces such as
read,write, andmkdir, without requiring application code modifications.USRBIO high-performance client: Achieves microsecond-level latency and ultra-high throughput through a user-space I/O stack and RDMA, meeting the demands of AI and HPC.
All components communicate over an RDMA network, bypassing the kernel protocol stack to significantly reduce CPU load and network latency.
Prerequisites
Prepare an ECS instance with Internet access as the client for creating and managing the ACK cluster. If you already have an ECS instance with Alibaba Cloud CLI configured, skip to Set up an ACK managed cluster by using Alibaba Cloud CLI.
All operations in this topic are performed on the client instance.
Prepare a client instance
Install and configure Alibaba Cloud CLI for the client instance
Set up an ACK managed cluster by using Alibaba Cloud CLI
Set up an ACK managed cluster with Terway ENI passthrough, eRDMA-capable ECS instances, ESSDs, containerd runtime with image acceleration, and advanced security groups.
This section uses the automatic VPC creation method, which simultaneously creates related resources including ECS instances, VPCs, vSwitches, security groups, CLB instances, public NAT gateways, and EIPs. You may be charged for these resources. See Billing of ACK managed and dedicated clusters.
This section provides example operations only. In production, plan the cluster scale, network requirements, VPC configurations (VPC and vSwitches), and cluster network configurations (CNI, container CIDR block, and service CIDR block) in advance. See ACK managed cluster network planning.
The following command uses a subset of parameters. See Create a cluster for all parameters.
To create an ACK cluster in an existing VPC, specify resources such as vSwitches and security groups.
Cluster initialization may take some time.
Connect to the client instance and create the cluster.
The following tables describe the parameters.
The command returns the cluster ID.
{ "cluster_id": "cb12xxxd0c", "instanceId": "cb1xxxd0c", "request_id": "6B0xxx815C7A", "task_id": "T-680d" }View cluster information.
aliyun cs GET /clusters/cluster_idReplace cluster_id with the cluster ID obtained in the previous step.
[root@iZ xxx xZ ~]# aliyun cs GET /clusters/cb1xxx d0c { "cluster_id": "cb12d4cxxx d0c", "cluster_spec": "ack.pro.small", "cluster_type": "ManagedKubernetes", "created": "2xxx-xxx-xxx-xxx00", "current_version": "1.32.1-aliyun.1", "deletion_protection": false, "disable_encryption": true, "external_loadbalancer_id": "lb-2zxxx xxx x5", "init_version": "1.32.1-aliyun.1", "ip_stack": "ipv4", "maintenance_info": { "cluster_upgrade_info": { "nodepools": null }, "is_active": false }, "maintenance_window": { "enable": false, "weekly_period": "" }, "master_url": "{\"api_server_endpoint\":\"\"xxx\"intranet_api_server_endpoint\":\"https://10.59.196.223:6443\"}"You can also view the cluster in the ACK console.
The cluster list shows the created cluster 3FS-example, cluster type ACK managed cluster, cluster specification Pro, status running, 4 nodes, Kubernetes version
1.32.1-aliyun.1.Install kubectl and connect to the cluster. See Connect to an ACK cluster using kubectl.
ImportantThis example connects to the ACK cluster over the internal network. Ensure the client instance and the ACK cluster reside in the same VPC.
If they are in different VPCs, change the VPC of the client instance. See Change the VPC for an ECS instance.
View all namespaces:
kubectl get namespace[root@iz xxx ~]# kubectl get namespace NAME STATUS AGE ack-csi-fuse Active 46m ack-erdma-controller Active 44m default Active 47m kube-node-lease Active 47m kube-public Active 47m kube-system Active 47mInstall Helm V3.
Helm manages Kubernetes application packaging and versioning. This example uses Helm to deploy the 3FS cluster. See Simplify application deployment with Helm.
wget https://get.helm.sh/helm-v3.3.0-linux-amd64.tar.gz tar -xzvf helm-v3.3.0-linux-amd64.tar.gz mv linux-amd64/helm /usr/local/bin/helm helm versionThe following output indicates that Helm V3 is installed.
[root@xxx ~]# helm version version.BuildInfo{Version:"v3.3.0"
Deploy a 3FS cluster on an ACK cluster
Alibaba Cloud has made lightweight technical adaptations to the open-source 3FS file system, packaging it as standardized container images and providing out-of-the-box Helm Charts for automated deployment in Kubernetes environments. The related customized code, deployment manifests, and continuous integration configurations have been fully open-sourced and hosted on GitHub, supporting developers to complete deployment with a single command.
This example deploys a 3FS cluster with two meta nodes, two mgmtd nodes, one FUSE client, and three storage nodes on ACK using Helm.
Connect to the client instance, obtain the code repository, and switch to the deployment directory.
git clone https://github.com/AliyunContainerService/alibabacloud-3FS.git cd alibabacloud-3FS/deploy/containerDeploy FoundationDB Operator (FDB Operator).
3FS uses FoundationDB (FDB) as its storage engine. Deploy FDB Operator before 3FS.
kubectl create ns fdb kubectl apply -n fdb -f ./fdb-operator[root@ixxxx container]# kubectl apply -n fdb -f ./fdb-operator customresourcedefinition.apiextensions.k8s.io/foundationdbbackups.apps.foundationdb.org created customresourcedefinition.apiextensions.k8s.io/foundationdbclusters.apps.foundationdb.org created customresourcedefinition.apiextensions.k8s.io/foundationdbrestores.apps.foundationdb.org created serviceaccount/fdb-kubernetes-operator-controller-manager created clusterrole.rbac.authorization.k8s.io/fdb-kubernetes-manager-clusterrole created clusterrole.rbac.authorization.k8s.io/fdb-kubernetes-operator-manager-role created rolebinding.rbac.authorization.k8s.io/fdb-kubernetes-operator-manager-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/fdb-kubernetes-operator-manager-clusterrolebinding created deployment.apps/fdb-kubernetes-operator-controller-manager createdDeploy 3FS and its required FDB cluster by using the Helm chart provided in the code repository.
helm install 3fs ./chart -n 3fs --create-namespace --timeout 10mNoteDuring deployment, elastic ephemeral disks are automatically created and attached to the ECS instances hosting storage nodes.
Deployment may take several minutes. Containers may crash and restart during initialization, which is expected.
After deployment completes:
[root@i-xxx container]# helm install 3fs ./chart -n 3fs --create-namespace --timeout 10m NAME: 3fs LAST DEPLOYED: Thu Apr 24 19:22:02 2025 NAMESPACE: 3fs STATUS: deployed REVISION: 1 TEST SUITE: NoneThe 3FS distributed file system is now deployed on ACK with eRDMA for inter-node communication.
View all pods in the 3fs namespace:
kubectl get pods -o wide -n 3fs[root@i ~]# kubectl get pods -o wide -n 3fs NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES admin-cli-3fs 1/1 Running 4 (11m ago) 23h 10.7.185.73 cn-beijing.10.7.185.56 <none> <none> fdb-meta-3fs-cluster-controller-39839 2/2 Running 0 23h 10.7.185.78 cn-beijing.10.7.185.57 <none> <none> fdb-meta-3fs-log-56992 2/2 Running 0 23h 10.7.185.87 cn-beijing.10.7.185.56 <none> <none> fdb-meta-3fs-log-71409 2/2 Running 0 23h 10.7.185.80 cn-beijing.10.7.185.57 <none> <none> fdb-meta-3fs-log-74648 2/2 Running 0 23h 10.7.185.89 cn-beijing.10.7.185.57 <none> <none> fdb-meta-3fs-log-79318 2/2 Running 0 23h 10.7.185.84 cn-beijing.10.7.185.55 <none> <none> fdb-meta-3fs-storage-13747 2/2 Running 0 23h 10.7.185.83 cn-beijing.10.7.185.56 <none> <none> fdb-meta-3fs-storage-24732 2/2 Running 0 23h 10.7.185.88 cn-beijing.10.7.185.58 <none> <none> fdb-meta-3fs-storage-35553 2/2 Running 0 23h 10.7.185.77 cn-beijing.10.7.185.55 <none> <none> fuse-3fs 1/1 Running 4 (23h ago) 23h 10.7.185.74 cn-beijing.10.7.185.58 <none> <none> meta-3fs-100 1/1 Running 4 (23h ago) 23h 10.7.185.72 cn-beijing.10.7.185.57 <none> <none> meta-3fs-101 1/1 Running 4 (23h ago) 23h 10.7.185.82 cn-beijing.10.7.185.58 <none> <none> mgmtd-3fs-1 1/1 Running 4 (23h ago) 23h 10.7.185.75 cn-beijing.10.7.185.56 <none> <none> mgmtd-3fs-2 1/1 Running 4 (23h ago) 23h 10.7.185.79 cn-beijing.10.7.185.57 <none> <none> storage-3fs-10000 1/1 Running 3 (23h ago) 23h 10.7.185.85 cn-beijing.10.7.185.55 <none> <none> storage-3fs-10001 1/1 Running 4 (23h ago) 23h 10.7.185.81 cn-beijing.10.7.185.58 <none> <none> storage-3fs-10002 1/1 Running 4 (23h ago) 23h 10.7.185.86 cn-beijing.10.7.185.57 <none> <none>Log on to the 3FS cluster management console:
kubectl attach -n 3fs admin-cli-3fs -itPress Ctrl+D to display the prompt, then enter
list-nodesto view the node information of the 3FS cluster. Press Ctrl+D to exit.[root@iz xxx container]# kubectl attach -n 3fs admin-cli-3fs -it If you don't see a command prompt, try pressing enter. / > list-nodes bash: line 1: /usr/sbin/ibdev2netdev: No such file or directory [2025-04-24T11:26:23.323018337+00:00 admin_cli: 1 IBDevice.cc:165 ERROR] Failed to run ibdev2netdev, ret code 127 [2025-04-24T11:26:23.323066545+00:00 admin_cli: 1 IBDevice.cc:215 WARNING] Failed to load ibdev2netdev, maybe running in container. [2025-04-24T11:26:23.324204434+00:00 admin_cli: 1 IBDevice.cc:380 WARNING] IBDevice erdma_0:1's netdev is unknown, maybe running in container. [2025-04-24T11:26:23.324213256+00:00 admin_cli: 1 IBDevice.cc:441 CRITICAL] IBDevice erdma_0:1 can't set zone by IP, fallback to UNKNOWN Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion 1 MGMTD PRIMARY_MGMTD mgmtd-3fs-1 1 [] N/A 0(UPTODATE) 250228-dev-1-999999-f5fd8c05 2 MGMTD HEARTBEAT_CONNECTED mgmtd-3fs-2 1 [] 2025-04-24 11:26:16 0(UPTODATE) 250228-dev-1-999999-f5fd8c05 100 META HEARTBEAT_CONNECTED meta-3fs-100 1 [] 2025-04-24 11:26:20 0(UPTODATE) 250228-dev-1-999999-f5fd8c05 101 META HEARTBEAT_CONNECTED meta-3fs-101 1 [] 2025-04-24 11:26:13 0(UPTODATE) 250228-dev-1-999999-f5fd8c05 10000 STORAGE HEARTBEAT_CONNECTED storage-3fs-10000 1 [] 2025-04-24 11:26:21 0(UPTODATE) 250228-dev-1-999999-f5fd8c05 10001 STORAGE HEARTBEAT_CONNECTED storage-3fs-10001 1 [] 2025-04-24 11:26:22 0(UPTODATE) 250228-dev-1-999999-f5fd8c05 10002 STORAGE HEARTBEAT_CONNECTED storage-3fs-10002 1 [] 2025-04-24 11:26:22 0(UPTODATE) 250228-dev-1-999999-f5fd8c05Check the FUSE client:
kubectl exec -n 3fs fuse-3fs -it -- ls /mnt/3fs[root@xxx container]# kubectl exec -n 3fs fuse-3fs -it -- ls /mnt/3fs 3fs-virt3fs-virtis a special folder for the 3FS FUSE client.The /mnt/3fs directory is also mounted on the ECS instances. You can run tools such as fio on the instances to test performance.