DocsICP FilingConsole
Documentation
Home Page Cloud Parallel File Storage CPFS General-purpose Edition User Guide Mount and access Mount access FAQ

Mount access FAQ

Updated at:
Copy as MD
My Favorites

This topic describes common issues and solutions when mounting a CPFS file system fails on a Linux operating system by using a CPFS-POSIX client or CPFS-NFS client.

Overview

POSIX client mount point

CPFS-POSIX client mount

CPFS-NFS client mount

CPFS scale-out

What do I do if the "insufficient inventory" error is returned when I create a POSIX client mount point?

  • Symptom:

    When creating a POSIX client mount point for a file system in the Cloud Parallel File Storage (CPFS) console, the following error is reported: insufficient inventory in the specified zone.

  • Cause:

    When you create a POSIX mount point, CPFS automatically creates three pay-as-you-go ECS instances (ecs.g*.large) under your Alibaba Cloud account. These instances are used to manage the CPFS-POSIX client cluster. When you create a POSIX client mount point, make sure that your Alibaba Cloud account is in a normal state and can purchase ECS instances.

  • Solution:

    You can log on to the ECS console and go to Custom Launch to check the inventory of the required ECS instance types. This helps avoid mount point creation failures caused by insufficient inventory of the specified instance type.

What do I do if the "insufficient vSwitch IPs" error is returned when I create a POSIX client mount point?

  • Symptom:

    When you create a POSIX client mount point for a file system in the CPFS console, the following error is returned: insufficient vSwitch IPs.

  • Cause:

    The storage nodes of a CPFS file system need to use IPs allocated from the vSwitch specified by the POSIX client mount point. Each storage node occupies one IP. The maximum number of IPs occupied by storage nodes of a CPFS file system is 160.

  • Solution:

    You can log on to the VPC console to query the number of available IPs in the vSwitch under the target VPC. Make sure that the vSwitch has sufficient available IPs.

Why am I unable to create a POSIX client mount point?

If you cannot find a mount point that you just created in the CPFS console, check whether your Alibaba Cloud account has overdue payments. If your Alibaba Cloud account has overdue payments, you cannot create a CPFS file system mount point. Top up your account and try again to create a CPFS file system mount point.

How many CPFS file systems can be mounted on a single ECS instance?

A single ECS instance can mount at most one CPFS file system.

What do I do if the "unsupported OS for 'X86_64' architecture" error is returned when I mount a CPFS file system?

When you mount a CPFS file system, if the following error is returned, it indicates that the CPFS file system does not support the operating system of the compute node. Change the compute node specifications. For the operating systems supported by the CPFS client, see Limits.

[ FATAL ] You cannot add cpfs-client-001 node because it has an unsupported OS for 'X86_64' architecture.

What do I do if the "make sure kernel-devel version is consistent with kernel" error is returned when I mount a CPFS file system?

When you mount a CPFS file system, if the following error is returned, it indicates that the kernel-devel or kernel-headers package is not installed on the ECS instance or the installed package version is incompatible.

No package kernel-devel-3.10.0-957.21.3.el7.x86_64 available.

Error: Nothing to do

please make sure kernel-devel version is consistent with kernel

Run the following commands to check the package installation on the ECS instance.

rpm -qa | grep kernel-devel-`uname -r`

If the output is empty, the packages on the ECS instance are not correctly installed. Reinstall the packages on the ECS instance. For more information, see Step 1: Prepare the environment.

What do I do if the "ssh: connect to host A port 22: Connection timed out" error is returned when I mount a CPFS file system?

When you mount a CPFS file system, if the following error is returned, it indicates that the network between the ECS instance and the file system is disconnected.

====> start check ssh
try ssh root@a.b.c.d by /root/.ssh/id_rsa.pub

ssh: connect to host a.b.c.d port 22: Connection timed out

Check the following possible causes and try to fix the issue:

Possible cause

Solution

The network between the ECS instance (a.b.c.d) and the POSIX client admin node (qr-001) is disconnected

Check the network connectivity and run the mount command again.

The ECS instance (a.b.c.d) has not been added to the qr-sg security group

Check the security group configuration and try again. For more information, see Configure a security group.

The ECS instance (a.b.c.d) and the CPFS mount point are not in the same VPC

Select an ECS instance in the same VPC as the mount point to perform the mount operation.

The IP of the ECS instance (a.b.c.d) does not exist

Check the status of the ECS instance.

What do I do if the "not active on:<hostname>" error is returned when I mount a CPFS file system?

When you mount a CPFS file system, if the following error is returned, it indicates that the CPFS file system cannot be started on the ECS instance.

[ WARN ] GPFS is not active on: hostname. Consult the install toolkit logs for possible errors 
during install. The GPFS service can also be started manually by running GPFS command 
'mmstartup -N Node[,Node...]'

[ FATAL ] GPFS NOT ACTIVE

Check the following possible causes and try to fix the issue:

  1. The security group of the ECS instance is incorrectly configured or the instance has not been added to the qr-sg security group. For more information, see Configure a security group.

  2. The CPFS file system requires the ECS instance to have more than 4 GB of memory. If the ECS instance has insufficient memory, an error is returned. Check the memory of the ECS instance.

What do I do if the "Command failed" error is returned when I mount a CPFS file system?

When you mount a CPFS file system, if the following error is returned, it indicates that the available memory of the ECS instance is less than 4 GB. Upgrade the memory of the ECS instance and run the cpfs add ip command again to mount the file system.

[ WARN ] GPFS is not active on: hostname. Consult the install toolkit logs for possible errors 
during install. The GPFS service can also be started manually by running GPFS command 
'mmstartup -N Node[,Node...]'

[ FATAL ] GPFS NOT ACTIVE

What do I do if the "cpfs.sh is running already" error is returned when I mount a CPFS file system?

When you mount a CPFS file system, if the following error is returned, it indicates that a mount or unmount task is already running. Wait a while and try again.

cpfs.sh is running already, pid: xyz

What do I do if the "connect to host B port 22: Connection timed out" error is returned when I mount a CPFS file system?

When ECS instance A mounts a CPFS file system, if the following error is returned, it indicates that an ECS instance B with an abnormal status exists in the current CPFS file system cluster.

# cpfs add A

connect to host B port 22: Connection timed out

B hostname is invalid

Failed to add node.

Troubleshoot and fix ECS instance B by referring to the following methods, and then try the mount operation again.

Run mmgetstate -a on the admin node qr-001 to check whether ECS instance B is in a normal status (active indicates a normal status).

  • If instance B is in a normal status, submit a ticket to contact the CPFS team for further troubleshooting.

  • If instance B is in an abnormal status, determine whether to continue using the instance.

    • If you want to continue using this instance, submit a ticket to contact the CPFS team to repair the instance status.

    • If you no longer need the instance, run the mmdelnode -N <id> --force command to clear the node information.

      mmdelnode -N iZuf61mhwoc9flkufs0**** --force

  Do you want to continue? (yes/no) yes

mmdelnode: [W] Could not cleanup the following unreached nodes:

iZuf61mhwoc9flkufs0****

mmdelnode: Command successfully completed

mmdelnode: Propagating the cluster configuration data to all affected nodes. This is an 
asynchronous process.

      After the cleanup, delete the host information of instance B from /etc/hosts. In the command, iZuf61mhwoc9flkufs0**** is the ID of the target ECS instance.

What do I do if the "[FATAL] B: Could not connect to B via ssh" error is returned when I mount a CPFS file system?

When ECS instance A mounts a CPFS file system, if the following error is returned, it indicates that an ECS instance B with an interrupted installation exists in the current CPFS file system cluster and has residual configuration.

[ FATAL ] ssh: connect to host B port 22: Connection timed out

[ FATAL ] B: Could not connect to B via ssh.

  • Versions earlier than 2.2.0

    Delete the residual configuration information of ECS instance B from the /usr/lpp/mmfs/5.0.5.0/installer/configuration/clusterdefinition.txt file.

    [node4]
fqdn = B

os = rhel7
arch = x86_64

ip_address = 192.168.6.37
is_admin_node = False

is_object_store = False

is_nfs = False
is_smb = False

is_hdfs = False
is_protocol_node = False
is_nsd_server = False
access_ips =
is_quorum_node = False

is_manager_node = False
is_gui_server = False
is_ems_node = False

is_callhome_node = False

is_broker_node = False

is_node_offline = False

is_node_reachable = True

is_node_excluded = False

is_mestor_node = False

  • Version 2.2.0 and later

    Delete the residual configuration information of ECS instance B from the /usr/lpp/mmfs/5.1.2.0/ansible-toolkit/ansible/ibm-spectrum-scale-install-infra/vars/scale_clusterdefinition.json file.

       {
      "fqdn": "iZuf6hn0blj1g377w4xxxxZ",
      "os": "rhel7",
      "arch": "x86_64",
      "ip_address": "172.19.0.100",
      "is_admin_node": false,
      "is_object_store": false,
      "is_nfs": false,
      "is_smb": false,
      "is_hdfs": false,
      "is_protocol_node": false,
      "is_nsd_server": false,
      "is_quorum_node": false,
      "is_manager_node": false,
      "is_gui_server": false,
      "is_ems_node": false,
      "is_callhome_node": false,
      "is_broker_node": false,
      "is_node_offline": false,
      "is_node_reachable": true,
      "is_node_excluded": false,
      "is_mestor_node": false,
      "scale_daemon_nodename": "iZuf6hn0blj1g377w4xxxxZ"
    }

What do I do if the [ FATAL ] No GPFS admin node specified. specify an admin node using 'spectrumscale node add <node name or IP> -a'. error is returned when I mount a CPFS file system?

When you mount a CPFS file system, if the following error is returned, it indicates that you ran the command on the wrong node.

[ FATAL ] No GPFS admin node specified. specify an admin node using 'spectrumscale node add <node name or IP> -a'.

Check whether you ran the CPFS command on the qr-001 node.

[root@cpfs-08cd3xxx-000001-qr-001 ~]#

What do I do if the "Failed to resolve domain: file-system-id.region.cpfs.aliyuncs.com" error is returned when I mount a CPFS file system?

  • Cause

    The file-system-id.region.cpfs.aliyuncs.com parameter in the mount command is not replaced with the mount address of the export directory.

  • Solution

    Log on to the NAS console. In the Actions column of the target CPFS file system, click Manage to go to the Protocol Service page. In the Actions column, click Export Directory to open the Export Directory panel and obtain the mount address. Then, replace the file-system-id.region.cpfs.aliyuncs.com parameter in the mount command with the obtained mount address. Run the mount command again to mount the file system.

What do I do if an error occurs when I mount a CPFS file system by using an Elastic Desktop Service (EDS) client?

If an error message, such as 'cannot mount' or 'mount failed', appears when you try to mount the CPFS file system on the cloud desktop, follow these steps to check the network connectivity between the cloud desktop and the CPFS file system:

  1. Run the following command to query DNS for the IP address.

    The example command uses cpfs-009e40ab9c6476e6-001a3e8bf745b****.cn-hangzhou.cpfs.aliyuncs.com as the mount target domain name. Replace it with your actual domain name.

    dig  -t txt cpfs-009e40ab9c6476e6-001a3e8bf745b****.cn-hangzhou.cpfs.aliyuncs.com

  2. Run the ping command and use the IP address from the previous step to check for network connectivity.

    If the network is not connected, check your network configuration.

What do I do if a YUM repository error occurs when I use the cpfs add command to mount a CPFS file system?

When you mount a CPFS file system, if the following error is returned, it indicates that the YUM repository configuration for CentOS 8 has become invalid.

Errors during downloading metadata for repository 'appstream':
Status code: 404 for http://mirrors.cloud.aliyuncs.com/centos/8/AppStream/x86_64/os/repodata/repomd.xml (IP: 100.100.XX.XX)
Error: Failed to download metadata for repo 'appstream': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

CentOS 8 has reached its end of life (EOL), and the Linux community no longer maintains this operating system version. We recommend that you switch the YUM repository.

What do I do if the CPFS client fails to start?

  • Symptom:

    1. When you run mmgetstate -a on the admin node qr-001 to check the status of the ECS instance, the instance status is displayed as down.

    2. Run the /usr/lpp/mmfs/bin/mmstartup command. The following information is returned.

      … mmfslinux.ko kernel extension does not exist. Use mmbuildgpl
command to create the needed kernel extension for your kernel …

  • Cause:

    The kernel of the current ECS instance has been upgraded.

  • Solution:

    1. Run the /usr/lpp/mmfs/bin/mmbuildgpl command to rebuild the kernel extension.

      The following output is returned:

      mmbuildgpl: Building GPL (5.1.X.X) module begins at Fri Dec 3 16:05:33 CST 2021.
--------------------------------------------------------
Verifying Kernel Header...
 kernel version = 41800305 (418000305012001, 4.18.0-305.12.1.el8_4.x86_64, 4.18.0-305.12.1)
 module include dir = /lib/modules/4.18.0-305.12.1.el8_4.x86_64/build/include
 module build dir  = /lib/modules/4.18.0-305.12.1.el8_4.x86_64/build
 kernel source dir = /usr/src/linux-4.18.0-305.12.1.el8_4.x86_64/include
 Found valid kernel header file under /usr/src/kernels/4.18.0-305.12.1.el8_4.x86_64/include
Getting Kernel Cipher mode...
 Will use skcipher routines
Verifying Compiler...
 make is present at /bin/make
 cpp is present at /bin/cpp
 gcc is present at /bin/gcc
 g++ is present at /bin/g++
 ld is present at /bin/ld
Verifying libelf devel package...
 Verifying elfutils-libelf-devel is installed ...
  Command: /bin/rpm -q elfutils-libelf-devel
  The required package elfutils-libelf-devel is installed
Verifying Additional System Headers...
 Verifying kernel-headers is installed ...
  Command: /bin/rpm -q kernel-headers
  The required package kernel-headers is installed
make World ...
make InstallImages ...
--------------------------------------------------------
mmbuildgpl: Building GPL module completed successfully at Fri Dec 3 16:05:54 CST 2021.
--------------------------------------------------------

    2. Run the /usr/lpp/mmfs/bin/mmstartup command to restart the ECS instance.

    3. Run the /usr/lpp/mmfs/bin/mmmount all command to remount the file system.

How do I clean up residual configuration of an unmounted ECS instance?

First, confirm that the CPFS file system has been unmounted from the ECS instance. For more information, see Unmount a file system. Then, run the mmdelnode -N <id> --force command to clean up the residual configuration of the unmounted ECS instance. The following example is provided:

mmdelnode -N iZuf61mhwoc9flkufs0**** --force
  Do you want to continue? (yes/no) yes
mmdelnode: [W] Could not cleanup the following unreached nodes:

iZuf61mhwoc9flkufs0****

mmdelnode: Command successfully completed

mmdelnode: Propagating the cluster configuration data to all affected nodes. This is an 
asynchronous process.

In the command, iZuf61mhwoc9flkufs0**** is the ID of the target ECS instance.

What do I do if the "insufficient inventory" error is returned when I perform a scale-out operation?

CPFS file system scale-out depends on the inventory of CPFS storage nodes and the number of available IPs of the mount point vSwitch. Go to the VPC console to check the number of currently available IPs. A CPFS file system requires a maximum of 164 available IPs. Make sure that the vSwitch has sufficient available IPs.

Is data automatically rebalanced after a CPFS file system is scaled out?

After a CPFS file system is scaled out, data rebalancing is not performed by default. Existing data remains on the original storage nodes and is not automatically migrated to the newly added storage nodes.

Data rebalancing consumes the network and disk bandwidth of storage nodes, which degrades the frontend I/O performance of the file system. In addition, the more existing data a file system has, the longer data rebalancing takes. Considering that most workloads do not require automatic data rebalancing after a scale-out, CPFS file systems do not automatically perform data rebalancing after a scale-out.

Can CPFS only be mounted and accessed by using the POSIX client?

CPFS supports mounting and accessing file systems by using either the CPFS-POSIX client or the CPFS-NFS client. CPFS also supports cross-access between the CPFS-POSIX client and the CPFS-NFS client. For example, files created by using the CPFS-POSIX client and their modifications are visible to the CPFS-NFS client, and vice versa. For more information, see Client instructions.

Previous：Mount a CPFS file system on a cloud computerNext：File system management