Deploy 3FS on an ACK cluster

Deployment solution

You can build a 3FS cluster using Alibaba Cloud's high-performance eRDMA network and elastic ephemeral disks. An Alibaba Cloud managed ACK cluster allows for rapid deployment and helps you easily set up a cluster service that spans multiple nodes.

• eRDMA is an elastic Remote Direct Memory Access (RDMA) network developed by Alibaba Cloud for the cloud. eRDMA reuses virtual private clouds (VPCs) as the underlying link and uses a congestion control (CC) algorithm that is developed by Alibaba Cloud. eRDMA features high throughput and low latency based on RDMA supports. Compared with RDMA, eRDMA implements large-scale RDMA networking within seconds. eRDMA supports traditional HPC applications, AI applications, and Transmission Control Protocol/Internet Protocol (TCP/IP) applications.

  For more information, see Elastic RDMA (eRDMA).

• An elastic ephemeral disk is a block storage device that you can create with an instance or create separately, and whose capacity you can customize. It provides temporary, high-performance, and cost-effective data storage for ECS instances. For more information, see Elastic ephemeral disks.

• Container Service for Kubernetes (ACK) is one of the first container service platforms in the world to pass the Kubernetes Certified Conformance Program. It provides high-performance services for managing containerized applications and supports the full lifecycle management of enterprise-grade Kubernetes containerized applications. This lets you easily and efficiently run Kubernetes containerized applications in the cloud.

  For more information, see Container Service for Kubernetes (ACK managed clusters).

Preparations

You must prepare an ECS instance with Internet access to use as a client for creating and managing the ACK cluster. If you already have an instance with the Alibaba Cloud command-line interface (CLI) installed and configured, you can skip this section and proceed to Set up an ACK cluster using the Alibaba Cloud CLI.

Set up an ACK cluster using the Alibaba Cloud CLI

You can use the CLI to create an ACK cluster based on Alibaba Cloud's managed Kubernetes service. This cluster uses the Terway Elastic Network Interface (ENI) network mode to connect containers directly to ENIs. It is equipped with accelerated instances that support eRDMA and enterprise SSDs (ESSDs) to optimize storage performance. The containerd runtime and a pre-configured image accelerator ensure container efficiency. Enterprise-grade security group rules are enabled to enhance network isolation.

1. Connect to the client instance and run the following command to create the cluster.

   Click to view the command to create an ACK cluster using the CLI

   aliyun cs POST /clusters --header "Content-Type=application/json" --body '{
       "name": "3FS-example",
       "cluster_type": "ManagedKubernetes",
       "kubernetes_version": "1.34.1-aliyun.1",
       "region_id": "cn-beijing",
       "snat_entry": true,
       "proxy_mode": "ipvs",
       "addons": [
           {
               "name": "terway-controlplane"
           },
           {
               "name": "terway-eniip"
           },
           {
               "name": "csi-plugin"
           },
           {
               "name": "managed-csiprovisioner"
           },
           {
               "name": "nginx-ingress-controller",
               "disabled": true
           },
           {
               "name": "ack-erdma-controller",
               "config": "{\"agent\":{\"preferDriver\":\"compat\",\"allocateAllDevices\":true}}"
           }
       ],
       "cluster_spec": "ack.pro.small",
       "charge_type": "PostPaid",
       "zone_ids": [
           "cn-beijing-i"
       ],
       "service_cidr": "192.168.0.0/16",
       "ip_stack": "ipv4",
       "is_enterprise_security_group": true,
       "nodepools": [
           {
               "nodepool_info": {
                   "name": "default-nodepool"
               },
               "scaling_group": {
                   "system_disk_category": "cloud_essd",
                   "system_disk_size": 60,
                   "system_disk_performance_level": "PL0",
                   "system_disk_encrypted": false,
                   "instance_types": [
                       "ecs.g8ise.xlarge",
                       "ecs.g8i.xlarge"
                   ],
                   "instance_charge_type": "PostPaid",
                   "platform": "AliyunLinux",
                   "image_type": "AliyunLinux3",
                   "desired_size": 4
               },
               "kubernetes_config": {
                   "pre_user_data": "bWtkaXIgLXAgL2V0Yy9jb250YWluZXJkL2NlcnQuZC9kb2NrZXIuaW8vCmNhdCA+IC9ldGMvY29udGFpbmVyZC9jZXJ0LmQvZG9ja2VyLmlvL2hvc3RzLnRvbWwgPDxFT0YKc2VydmVyID0gImh0dHBzOi8vcmVnaXN0cnktMS5kb2NrZXIuaW8iCltob3N0LiJodHRwczovL21pcnJvcnMtc3NsLmFsaXl1bmNzLmNvbS8iXQogIGNhcGFiaWxpdGllcyA9IFsicHVsbCIsICJyZXNvbHZlIl0KICBza2lwX3ZlcmlmeSA9IHRydWUKRU9G",
                   "runtime": "containerd",
                   "runtime_version": "1.6.37"
               }
           }
       ]
   }'

   The cluster is configured as follows:

   • Basic cluster configuration

     Parameter	Value	Description
     name	3FS-example	The cluster name, which must be unique.
     cluster_type	ManagedKubernetes	Managed Kubernetes cluster.
     kubernetes_version	1.34.1-aliyun.1	The Alibaba Cloud-customized Kubernetes version.
     region_id	cn-beijing	Select a region that supports elastic ephemeral disks. The China (Beijing) region is recommended.
     charge_type	PostPaid	The pay-as-you-go billing method.

   • Network configuration

     Parameter	Function	Description
     service_cidr	Set the service CIDR block.	Also known as Service CIDR, this is the IP address pool for assigning IPs to internal cluster services. This CIDR block must not overlap with the VPC or any existing cluster CIDR blocks in the VPC, and must not overlap with the Container CIDR Block.
     ip_stack	The IP stack of the cluster.	Set this parameter to support only IPv4.
     is_enterprise_security_group	Specifies whether to create an enterprise security group.	Set this parameter to true. An enterprise security group is automatically created to provide stricter access rules. For more information, see Basic security groups and enterprise security groups.
     snat_entry	Specifies whether the cluster VPC has Internet access.	Set this parameter to true. A NAT Gateway is created and SNAT rules are automatically configured. The cluster VPC will have Internet access.

   • Node pool configuration (nodepools)

     • Scaling group configuration (scaling_group)

       Parameter	Value	Description
       system_disk_category	System disk type:  cloud_essd	Alibaba Cloud ESSDs, combined with 25 GE networking and RDMA technology, provide up to 1 million random read/write IOPS per disk and single-path low latency. For more information, see ESSDs.
       system_disk_size	System disk size:  60 GB	The minimum recommended size is 40 GB.
       system_disk_performance_level	ESSD performance level:  PL0	The basic performance level. It provides a medium limit for concurrent I/O performance and relatively stable read/write latency.
       instance_types	Instance type list	Select instance types that support eRDMA and elastic ephemeral disks. We recommend that you use the . Storage-enhanced general-purpose instance family g8ise.
       platform	Operating system:  AliyunLinux
       image_type	Image type:  AliyunLinux3

     • Kubernetes runtime configuration: Configure the containerd image accelerator to point to Alibaba Cloud Container Registry.

       View the Base64-encoded script configured for pre_user_data:

       

   • Add-on configuration (addons)

     Add-on name	Feature	Description
     terway-controlplane	Network plugin control plane	ACK supports two container network plugins: Terway and Flannel. This example uses Terway, a network plugin developed by Alibaba Cloud, with Trunk ENI support. For more information, see Container network plugins.
     terway-eniip	Trunk ENI support, pod-level vSwitch and security group configuration, and fixed IP addresses
     csi-plugin	Storage plugin	The csi-plugin and csi-provisioner components help you dynamically create, attach, and detach persistent volumes. For more information, see Manage CSI components.
     managed-csiprovisioner	CSI driver controller
     nginx-ingress-controller	Ingress controller	For more information, see Nginx Ingress Controller.
     ack-erdma-controller	The Kubernetes controller for Alibaba Cloud eRDMA. It provides eRDMA NIC management, scheduling, and pod network capabilities.	preferDriver: Select the compat driver type to be compatible with the RoCE driver mode. allocateAllDevices: Allocate all eRDMA devices on a node to its pods. For more information, see Use eRDMA to accelerate container networking.

   The cluster ID is returned in the command output:

   

2. After the cluster is created, run the following command to view the cluster instance.

   aliyun cs GET /clusters/cluster_id

   Replace cluster_id with the cluster ID from the output of the previous step. The returned information shows the detailed configuration of the cluster.

   

   You can also log on to the ACK console to view the created cluster in the cluster list:

   

3. Install kubectl to manage the cluster and its applications. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

   Important

   • In this example, you connect to the ACK cluster over the internal network. Make sure that the client instance where kubectl is installed and the ACK cluster are in the same VPC and can communicate with each other.

   • If the client instance and the ACK cluster are not in the same VPC, you can change the VPC of the client instance to the VPC of the ACK cluster. For more information, see Change the VPC for an ECS instance.

   After the configuration is successful, run the following command to list all namespaces in the Kubernetes cluster.

   kubectl get namespace

   

4. Install Helm v3.

   Helm is an open source tool used to manage applications in Kubernetes clusters. It provides a unified way to package applications and supports versioning, which reduces the complexity of application distribution and deployment. In this topic, you will use Helm to deploy the 3FS cluster. For more information about Helm, see Simplify application deployment using Helm.

   wget https://get.helm.sh/helm-v3.3.0-linux-amd64.tar.gz
   tar -xzvf helm-v3.3.0-linux-amd64.tar.gz
   mv linux-amd64/helm /usr/local/bin/helm
   helm version

   The following output indicates that the installation is successful.

   

Deploy 3FS on an ACK cluster

Alibaba Cloud adapted the open source 3FS file system, packaged it into a standardized container image, and provided an out-of-the-box Helm Chart for automated deployment in a Kubernetes environment. The customized code, deployment manifests, and continuous integration configurations are open source and hosted in a GitHub repository. This allows developers to complete the deployment using a single command.

This example uses Helm to deploy a 3FS cluster on an ACK cluster. The 3FS cluster contains two meta nodes, two mgmtd nodes, one FUSE client, and three storage nodes.

1. Connect to the client instance. Run the following commands to obtain the code repository and switch to the deployment directory.

   git clone https://github.com/AliyunContainerService/alibabacloud-3FS.git
   cd alibabacloud-3FS/deploy/container

2. Run the following commands to deploy the FoundationDB Operator.

   3FS builds its storage engine on FoundationDB. Before you can deploy 3FS, you must set up an FDB cluster. You can use the Kubernetes Operator to deploy the cluster.

   kubectl create ns fdb
   kubectl apply -n fdb -f ./fdb-operator

   

3. Run the following command to use the Helm chart in the code repository to deploy 3FS and the required FDB cluster in one step.

   helm install 3fs ./chart -n 3fs --create-namespace --timeout 10m

   Note

   • During the deployment, the system automatically creates elastic ephemeral disks for the ECS instances where the storage nodes reside and attaches the disks to the instances based on the Helm pod YAML configuration.

   • The deployment may take several minutes. During initialization, some containers may crash multiple times. This is normal. You can wait for the process to complete.

   After the deployment is complete, the following output is displayed:

   

   After the deployment is complete, you will have a high-performance distributed file system that is based on 3FS and deployed using ACK. The file system uses eRDMA for communication between nodes.

4. Run the following command to view all pods in the 3fs namespace.

   kubectl get pods -o wide -n 3fs

   

5. Run the following command to access the 3FS cluster management console.

   kubectl attach -n 3fs admin-cli-3fs -it

   Press Ctrl+D to display the prompt. Then, enter list-nodes to view the information about each node in the 3FS cluster. Press Ctrl+D to exit.

   

6. Run the following command to check the FUSE client.

   kubectl exec -n 3fs fuse-3fs -it -- ls /mnt/3fs

   

   • The output includes 3fs-virt, which is a special folder for the 3FS FUSE client.

   • The /mnt/3fs directory is also mounted to the ECS instance. You can run tools, such as fio, on the instance to test performance.