FAQ

Even in regions where E-HPC is supported, cluster creation can fail if the underlying resources are unavailable. The two most common causes:

• No NAS file system available: File Storage NAS (NAS) can't be created in that region, or no NAS file system exists. E-HPC requires NAS to mount shared storage on the cluster.

• ECS instance types unavailable: The region or zone doesn't have the Elastic Compute Service (ECS) instance types needed for your compute nodes, or inventory is exhausted.

Select a different region and try again.

No. Each node runs on an ECS instance, but E-HPC manages the full lifecycle of those instances—batch creation, management system deployment, software pre-installation, and job scheduler configuration. Managing nodes directly in the ECS console bypasses these processes and can cause cluster or node failures, or make cluster resources unavailable.

It depends on whether the node and the ECS instance are in the same virtual private cloud (VPC):

• Same VPC: They communicate directly within the VPC.

• Different VPCs: Connect the VPCs first. Use Cloud Enterprise Network (CEN) or VPC peering to enable communication between private networks.

You may fail to log on to a cluster using Workbench or another Secure Shell (SSH) client for various reasons. Troubleshoot and resolve the issue based on your specific situation.

Work through these checks in order:

1. Verify the username and password are correct.

2. Check that your local network and carrier network are up.

3. Confirm the security group rule for the logon node allows access to port 22 (the default SSH port).

4. Check whether the security group of the node allows access from IP addresses related to the Workbench service. The security group rules vary based on the network type. For more information, see Log on to a Linux instance using Workbench.

5. Run iptables -nvL --line-number on the logon node to check whether a firewall or firewall rules are blocking connections.

If none of these resolve the issue, connect using Virtual Network Computing (VNC) instead. See FAQ about connecting to ECS instances.

Cause: This is a known systemd bug that surfaces when Network Information Service (NIS) is in use. Symptoms include slow SSH logins or node switching, occasional login failures, and an inability to restart sshd with the error Failed to activate service 'org.freedesktop.systemd1': timed out.

Solution:

1. Log in to the node as root.

2. Check /etc/nsswitch.conf to see whether [NOTFOUND=return] is missing from the passwd, shadow, and group entries:

   cat /etc/nsswitch.conf

   If the output looks like the following, proceed to the next step:

   passwd:     files sss nis
   shadow:     files sss nis
   group:      files sss nis

3. (Optional) Upgrade glibc:

   yum update glibc

4. Open /etc/nsswitch.conf and add [NOTFOUND=return] to the passwd, shadow, and group lines:

   vim /etc/nsswitch.conf

   Update the entries to:

   passwd:      files sss nis [NOTFOUND=return]
   shadow:      files sss nis [NOTFOUND=return]
   group:       files sss nis [NOTFOUND=return]

5. Save the file.

If you want to use Alibaba Cloud services in the Chinese mainland, complete account verification first. Without it, selecting a Chinese mainland region when purchasing services triggers an error on the buy page. For details, see FAQ about account verification of Alibaba Cloud accounts.

Problem description

A compute node in the cluster has an Abnormal status and cannot schedule tasks, as shown in the following figure:

Possible cause

This issue may occur because you enabled the system firewall on the management node. This prevents the ypbind service on the compute nodes from working correctly, which in turn prevents the system from scheduling tasks.

Solution

Disable the system firewall on the control plane node.

1. Log on to the management node as the root user.

2. Run the following command to stop the firewalld service.

   systemctl stop firewalld

3. Run the following command to prevent the firewalld service from starting on boot.

   systemctl disable firewalld

4. Run the following command to check the status of the firewalld service on the management node. Make sure that the service is stopped and does not restart on boot.

   systemctl status firewalld

When you create an E-HPC cluster, CentOS 7.6 is the default operating system for custom service nodes. You can select a different image as needed. For an existing E-HPC cluster, you can change the image by replacing the custom service for the node.

1. In the E-HPC console, find the cluster and click its name to go to its details page.

2. On the cluster details page, click the Custom Service tab. Find the node that you want to modify, and click Delete in the right-side column.

3. Click Add Custom Component, select Login (Version 1.0), and then click Edit Instance Specifications in the ECS Instance section.

4. After you select the instance specifications, click OK to finish adding the custom component.

An image provides the necessary information for cluster nodes (ECS instances), including the operating system and related configuration data. E-HPC supports public images, custom images, and community images. For more information, see Image overview.

• Public images: Official images provided by Alibaba Cloud.

• Custom images: Images that you create from an ECS instance or a snapshot, or import from an on-premises environment.

• Community images: Images published by an image provider on the Alibaba Cloud community image platform.

When creating or scaling out a cluster, or configuring an auto scaling policy, a custom image won't appear in the list if any of the following apply:

• Your account has no custom image in the current region. See Overview for how to create one.

• The custom image's operating system isn't supported by E-HPC.

• The node's instance type doesn't support the custom image.

• When configuring an auto scaling policy, the image in the global settings and queue settings must match.

Custom images must meet the following requirements. If your image doesn't meet them, creation or scale-out will fail:

• The yum source configuration in the image cannot be modified.

• /home and /opt cannot be mount directories or symbolic links in the image.

• If /etc/fstab contains mount entries for external file systems (such as NFS), the cluster must be able to access those file systems, or they must be in the same VPC. Remove any inaccessible entries from /etc/fstab before creating the cluster.

• The image must retain the group with account group ID 1000.

• The system disk size must be at least as large as the custom image.

E-HPC supports importing CentOS images only. For import steps, see Import an image.

When importing, make sure Check After Import is selected. Without it, the image may not appear in the E-HPC console.

Problem Description: When you use an Ubuntu image to scale out nodes, some compute nodes experience an abnormally long initialization time (more than 5 minutes), causing the nodes to fail to join the cluster for scheduling. In the node's deploy.log file, you may see an error message similar to the following:

E: Could not get lock /var/lib/dpkg/lock-frontend - open (11: Resource temporarily unavailable)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?

This issue is especially prominent in large-scale scale-out scenarios, such as when launching tens or hundreds of nodes simultaneously, and can affect most nodes.

Cause: The unattended-upgrades service is pre-installed and enabled by default in Ubuntu. It runs in the background to automatically check for and install security patches and critical updates without user intervention. The service is triggered by two systemd timers, apt-daily.timer and apt-daily-upgrade.timer, which typically run shortly after the system starts. During its operation, it acquires a lock on /var/lib/dpkg/lock-frontend to ensure the atomicity of package operations.

In E-HPC auto scaling scenarios, unattended-upgrades can cause the following issues:

• dpkg lock contention: After a node starts, unattended-upgrades runs immediately and acquires the /var/lib/dpkg/lock-frontend lock. This prevents the E-HPC deployment script from acquiring the lock to install required components like slurmd and nfs-common, blocking the installation process or causing it to fail.

• Initialization timeout: The runtime of unattended-upgrades is unpredictable and can last for several minutes. The deployment script must wait, causing the total node initialization time to exceed 5 minutes.

• Exacerbation in large-scale scale-outs: Each new node independently runs unattended-upgrades, fetching and verifying packages. When many nodes are launched simultaneously, the probability of dpkg lock contention between unattended-upgrades and the E-HPC deployment script approaches 100%. The larger the batch, the more severe the problem becomes. Nodes consume ECS resources but fail to complete initialization. This can also lead to wasted resources from node startup timeouts.

Solution: The most direct solution is to disable unattended-upgrades in your custom image. This prevents the background auto-update process from acquiring the dpkg lock when a node starts. Follow these steps:

1. When creating your custom image, log on to the base ECS instance as the root user.

2. Run the following commands to stop and disable the unattended-upgrades service:

   # Run these commands when creating the custom image
   systemctl disable unattended-upgrades
   systemctl mask unattended-upgrades
   apt-get remove -y unattended-upgrades

3. Run the following command to verify that the unattended-upgrades service is completely disabled:

   systemctl status unattended-upgrades

   In the expected output, the Loaded field should show masked or not-found, and the Active field should show inactive (dead). This indicates that the service has been successfully disabled.

4. Create the custom image based on this configuration. When you create an E-HPC cluster or elastically scale out nodes, select this custom image to avoid the dpkg lock contention issue.

E-HPC clusters use File Storage NAS to share data across compute nodes, so software installed in a shared directory is immediately accessible to all nodes.

Choose the installation location based on who needs access:

• All cluster users: Install to /opt.

• A single cluster user: Install to that user's home directory.

Problem: When you use spot instances for auto scaling, if an instance is reclaimed while compute jobs are still running on it, the scheduler may fail to remove the instance. This causes the node deletion to fail.

Solution: In auto scaling scenarios, the system automatically cleans up leftover nodes after a period of time. After the scheduler's status is updated, the node exits the BusyNodes state and can be deleted.

Problem: When you run the sinfo command, a compute node shows an alloc status. However, when you run the squeue command, no running jobs are found. This can leave the compute node idle but marked as allocated.

Cause: An internal SLURM communication issue or an abnormal job process exit can cause a node to remain in the ALLOCATED state even though the job process has not started or has already exited unexpectedly.

Solution

1. Create a directory to store the script.

   mkdir -p /root/recover

2. Run the following command to create the script file:

   cat << 'EOF' > /root/recover/slurm_abnormal_nodes_recover.sh
   #!/bin/bash
   
   if [ $# -lt 1 ]; then
       echo "Usage: $0 WORK_DIR"
       echo "Please exec with work dir"
       exit 0
   fi
   
   WORK_DIR=$1
   mkdir -p ${WORK_DIR}
   DATE=$(date +"%Y-%m-%d")
   exec > >(tee -a ${WORK_DIR}/node_recover_${DATE}.log) 2>&1
   
   cd ${WORK_DIR}
   TIME=$(date)
   echo "Slurm nodes check and recovery action start. time:${TIME}"
   scontrol show node --oneliner  > slurm_nodes
   
   > nodes_abnormal
   while read line; do
     if [[ $line =~ CPUAlloc=0 ]] && [[ $line =~ State=ALLOCATED\+DYNAMIC_NORM ]]; then
       echo $line >> nodes_abnormal
     fi
   done < slurm_nodes
   
   line_count=$(wc -l < "nodes_abnormal")
   if [[ $line_count -gt 0 ]]; then
       echo "The number of abnormal nodes: $line_count, restart service"
       systemctl restart slurmctld
       if [ $? -ne 0 ]; then
         echo "slurmctld service restart failed"
       else
         echo "slurmctld service restart success"
       fi
   else
       echo "The number of abnormal nodes: $line_count"
   fi
   
   echo "Slurm nodes check and recovery action complete."
   EOF

3. Grant execute permissions to the script.

   chmod +x /root/recover/slurm_abnormal_nodes_recover.sh

4. Set up a scheduled task to run the detection script every 5 minutes.

   Run the crontab -e command to edit the scheduled tasks and add the following content:

   */5 * * * * /root/recover/slurm_abnormal_nodes_recover.sh /root/recover

When creating an E-HPC cluster, specify both a mount target and a remote directory for the NAS file system. Example configuration:

ClusterId=ehpc-mrZSoWf****          # Cluster ID
VolumeMountpoint=045324****-m****.cn-hangzhou.nas.aliyuncs.com  # NAS mount target
RemotePath=/                         # Remote directory (NAS root)

To mount specific subdirectories on the cluster:

1. Create subdirectories under the NAS root before mounting:

   /ehpc-mrZSoWf****/opt
   /ehpc-mrZSoWf****/home

2. When creating (or after creating) the cluster, set the remote directory to the appropriate subdirectory. For example:

   /                          # Mounts to /ehpcdata on the cluster
   /ehpc-mrZSoWf****/home     # Mounts to /home on the cluster
   /ehpc-mrZSoWf****/opt      # Mounts to /opt on the cluster

   For full configuration steps, see Create a cluster by using the wizard and Manage storage resources.

Create the mount target and the mount directory before mounting the NAS file system.

The default limit is three clusters per region. To increase the quota, submit a ticketsubmit a ticketsubmit a ticketsubmit a ticket.

An E-HPC cluster supports up to 500 nodes. You can also add up to 500 compute nodes at a time. To increase either limit, submit a ticketsubmit a ticketsubmit a ticket.

Resource Access Management (RAM) provides a service-linked role called AliyunServiceRoleForEHPC for E-HPC. This role lets E-HPC access associated cloud resources—specifically ECS, VPC, and NAS.

If this role isn't attached to your account, complete role authorization before using E-HPC. See Manage a service-linked role.

If a RAM user hasn't been granted the AliyunEHPCReadOnlyAccess permission, the console displays the Switch to RAM for authorization message. Grant AliyunEHPCReadOnlyAccess to the RAM user to restore read access.

To allow a RAM user to create clusters, users, or jobs, grant both AliyunEHPCFullAccess and AliyunNASFullAccess. See Grant permissions to a RAM user.