DocsICP FilingConsole
Documentation
Home Page Elastic High Performance Computing E-HPC 1.0 Use Cases Create and deploy a hybrid cloud cluster Best practices for hybrid cloud master mode

Best practices for hybrid cloud master mode

Updated at:
Copy as MD
Product
My Favorites

In a hybrid cloud cluster running in master mode, the management node is part of a new cloud-based E-HPC cluster, while compute nodes include existing on-premises nodes and new cloud compute nodes. This topic describes how to deploy a master mode cluster that uses the SGE scheduler using an example scenario where an E-HPC cluster manages existing on-premises compute nodes.

Important

This deployment solution supports only the SGE scheduler. Before you proceed, you must submit a ticket to add your account to the allowlist for this feature.

Background information

Many enterprises, especially in fields like life sciences, already operate on-premises data centers for High-Performance Computing (HPC). However, these data centers may face challenges such as aging infrastructure and insufficient compute capacity. As business demands grow, they often seek to build cloud-based compute clusters while reusing their existing on-premises resources. To meet this demand, Alibaba Cloud E-HPC provides a hybrid cloud solution that enables unified, efficient management of both cloud and on-premises compute resources. This solution also leverages the elasticity of the cloud to support auto scaling.

The hybrid cloud solution offers two deployment modes: master mode and proxy mode.

  • Master mode: The management node is part of a new cluster created on the cloud.

  • Proxy mode: The management node is part of an existing cluster.

    Note

    In proxy mode, the existing cluster can be an on-premises cluster or a cluster in another region on the cloud.

Scenario

Assume an existing on-premises cluster with compute nodes that have the following configuration:

  • Operating system: CentOS 7.6

    Important

    Currently, only on-premises nodes that run a CentOS 7 series operating system are supported.

  • Specifications: 2 vCPUs, 4 GiB of memory

Procedure

This procedure includes the following steps:

Actions

Description

Step 1: Create an E-HPC cluster on the cloud

Set up an E-HPC cluster on the cloud and configure its network.

Step 2: Obtain the cloud cluster configuration information

Obtain the cloud cluster's configuration information to configure the on-premises compute nodes.

Step 3: Deploy the on-premises compute nodes

Deploy the scheduler client and user management client on the on-premises compute nodes, and mount the cloud storage.

Step 4: Add the on-premises compute nodes to the cloud cluster

Create a queue in the cloud cluster to manage the on-premises compute nodes, and then add the on-premises nodes to this queue.

Step 5: Configure auto scaling

Configure auto scaling for the cluster to automatically add or remove cloud compute nodes.

Step 6: Submit a job for testing

Submit a job to a queue in the cloud cluster to test whether the nodes work as expected.

(Optional) Remove on-premises compute nodes from the cloud cluster

You can remove on-premises compute nodes that are no longer needed from the cloud cluster and stop the related services on them.

Step 1: Create an E-HPC cluster

  1. Create an E-HPC cluster.

    For more information, see Create a cluster by using the wizard. When you configure the cluster, note the following parameters:

    • For Deployment Mode, select Lean. This deployment mode includes one management node and one optional compute node. We recommend that you select an instance type with at least 4 vCPUs and 8 GiB of memory for the management node.

    • For Scheduler, select opengridscheduler (SGE).

    • For Domain Account Service, select nis.

  2. Establish a network connection between the cloud cluster and the on-premises data center.

    You can use a VPN gateway or Express Connect to connect your cloud and on-premises networks.

  3. Configure a security group.

    Because the on-premises compute nodes need to join the cloud cluster for unified management, you must configure an inbound rule for the cluster's security group to allow access from the on-premises nodes.

    On the E-HPC cluster details page, click the security group ID to open the cluster's security group page. Add an inbound rule where the source is the CIDR block of your on-premises compute nodes.

    The inbound rule of the security group sets Priority to 1, Protocol Type to All, and Port Range to -1/-1 (all ports).

The following table describes the environment that is used in this topic.

Type

Description

Management node

1 ECS instance, ecs.c7a.xlarge.

Compute node

1 ECS instance, ecs.c7a.xlarge; 2 on-premises nodes. Operating system: CentOS 7.6.

Network

An SSL-VPN connection is used to connect the cloud cluster and the on-premises data center.

Step 2: Obtain cluster configuration

After creating the E-HPC cluster, obtain its management information to configure the on-premises compute nodes.

  1. Log on to the E-HPC cluster. For more information, see Log on to a cluster.

  2. Download the script used to obtain the cluster configuration information.

    wget https://public-ehpc-package.oss-cn-hangzhou.aliyuncs.com/scheduler/offpremise_config_fetch.sh

  3. Run the script to obtain the configuration information.

    source offpremise_config_fetch.sh

    The script generates a file named OFFPREMISE_PARAMS that contains the configuration information. Run the cat command to view the configuration information.

    [root@manager ~]# cat ONPREMISE_PARAMS
ehpc-hz-gInnxxx manager 10.0.xxx.xxx 0a4174xxx-jxxx.cn-hangzhou.nas.aliyuncs.com/ehpc-hz-gxxx xxxR manager 10.0.xxx.xxx

Step 3: Deploy on-premises nodes

You need to deploy the scheduler client and the user management client, and mount the cloud storage on the on-premises compute nodes.

Note

Before you begin, make sure the on-premises nodes can connect to the cloud cluster over the network. For example, if you use an SSL-VPN connection, you must install an SSL client on each on-premises compute node.

  • Single-node operation

    Log on to a compute node and perform the following operations:

    1. Download the service deployment script.

      wget https://public-ehpc-package.oss-cn-hangzhou.aliyuncs.com/scheduler/deploy_nis_sge_client.sh

    2. Run the script to deploy the services.

      sh deploy_nis_sge_client.sh <cluster_configuration_information>

      Replace <cluster_configuration_information> with the cluster configuration information that you obtained. For example:

      sh deploy_nis_sge_client.sh ehpc-hz-e5WF****** manager 10.18.XX.XX 000a******-vm***.cn-hangzhou.nas.aliyuncs.com:/ehpc-hz-e5WF****** manager 10.18.XX.XX

  • Multi-node operation

    Log on to a local configuration node or one of the compute nodes and perform the following operations. Make sure that this node can connect to the other on-premises compute nodes over the network.

    1. Download the service deployment script.

      wget https://public-ehpc-package.oss-cn-hangzhou.aliyuncs.com/scheduler/batch_install_clients.sh

    2. Create a hosts file.

      Each line in the hosts file corresponds to a compute node. The format is [host_name]  [host_ip]. For example:

      xi@xxx mcmode % cat hosts
sge-hyxxx-comp01 10.18.xxx.xxx
sge-hyxxx-comp02 10.11.xxx.xxx

    3. Run the script to deploy the services.

      sh batch_install_clients.sh <cluster_configuration_information> ./hosts

      Replace <cluster_configuration_information> with the cluster configuration information that you obtained. For example:

      sh batch_install_clients.sh ehpc-hz-e5WF****** manager 10.18.XX.XX 000a******-vm***.cn-hangzhou.nas.aliyuncs.com:/ehpc-hz-e5WF****** manager 10.18.XX.XX ./hosts

Step 4: Add on-premises nodes to the cluster

On-premises compute nodes cannot be moved between queues. Therefore, you must first create a dedicated queue for them and then add the nodes to that queue when resizing the cluster.

  1. Create a queue. For more information, see Manage queues.

  2. Add the on-premises compute nodes to the cluster.

    1. In the Elastic High Performance Computing console, go to the Clusters page. Find the destination cluster and click Resize in the upper-right corner.

    2. In the panel that appears, select the On-premises Node tab.

    3. Select the newly created queue, enter or batch import the information about the on-premises compute nodes, and then click OK.

      The following node information is required:

      • Hostname: Must be consistent with the actual hostname of the node.

      • Node IP: Must match the node's actual IP address.

      • CPU: The number of CPUs on the node. You can run the lscpu command to view this information.

      • Memory: The total memory size of the node.

      Important

      Ensure the entered hostname and node IP match the actual values on the node. Otherwise, the node cannot be added to the cluster.

      For nodes that are connected over SSL-VPN, the node IP address must be the IP address of the tun0 virtual network interface card (NIC). You can run the ifconfig command to view the IP address.

  3. On the Nodes & Queues page, find the destination cluster. Filter the nodes by type and queue to view the on-premises compute nodes.

    Refresh the page to update the node status. Wait for 1 to 2 minutes. When their status changes to Running, the nodes have been successfully added.

Step 5: Configure auto scaling

E-HPC supports auto scaling, which automatically adds or removes cloud compute nodes based on real-time workload. For more information, see Configure an auto scaling policy.

The auto scaling configuration page contains two areas: Global Configuration and Queue Configuration. Global Configuration includes:

  • Enable Scale-out and Enable Scale-in switches

  • Cluster Compute Nodes range slider (0–200)

  • Scale-in Time (Minutes) slider

  • Image Type dropdown (e.g., Public Image CentOS_7.6_64)

  • Excluded Node List dropdown

The Queue Configuration area displays queues (e.g., workq, local) in separate tabs. Each queue can be configured with Max Nodes, Min Nodes, Auto Scale-out, Auto Scale-in, Hostname Prefix, Image ID, Hostname Suffix, and Instance Configuration (e.g., ecs.c7.large 2 vCPUs, 4 GiB). After configuration, select E-HPC Terms of Service and click OK.

Note

Auto scaling is not supported for on-premises compute nodes and their queues.

Step 6: Submit a test job

  1. Submit a job to the queue that contains the cloud compute nodes for testing.

    qsub sample1.sh

    The following is an example sample1.sh test script. Replace the queue (-q) and paths (-o and -e) based on your environment.

    #!/bin/bash

#$ -cwd
#$ -l vf=1G
#$ -q workq
#$ -N sge-demo1
#$ -o /home/test/
#$ -e /home/test/

echo "demo start, sleep 10s and ping localhost"
sleep 10
ping -c 10 localhost
echo "demo complete"

  2. Submit a job to the queue that contains the on-premises compute nodes for testing.

    qsub sample2.sh

    The following is an example sample2.sh test script. Replace the queue (-q) and paths (-o and -e) based on your environment.

    #!/bin/bash

#$ -cwd
#$ -l vf=1G
#$ -q local
#$ -N sge-demo2
#$ -o /home/test/
#$ -e /home/test/

echo "demo start, sleep 10s and ping localhost"
sleep 10
ping -c 10 localhost
echo "demo complete"

(Optional) Remove on-premises nodes

  1. Remove the on-premises compute nodes from the cluster.

    For more information, see Delete a node.

  2. Stop the related services on the on-premises compute nodes.

    • Single-node operation

      Log on to a compute node and perform the following operations:

      1. Download the script used to stop the services.

        wget https://public-ehpc-package.oss-cn-hangzhou.aliyuncs.com/scheduler/nis_sge_client_stop.sh

      2. Run the script to stop the services.

        sh nis_sge_client_stop.sh <cluster_configuration_information>

        Replace <cluster_configuration_information> with the cluster configuration information that you obtained. For example:

        sh nis_sge_client_stop.sh ehpc-hz-e5WF****** manager 10.18.XX.XX 000a******-vm***.cn-hangzhou.nas.aliyuncs.com:/ehpc-hz-e5WF****** manager 10.18.XX.XX

    • Multi-node operation

      Log on to a local configuration node or one of the compute nodes and perform the following operations. Make sure that this node can connect to the other on-premises compute nodes over the network.

      1. Download the script used to stop the services.

        wget https://public-ehpc-package.oss-cn-hangzhou.aliyuncs.com/scheduler/batch_stop_clients.sh

      2. Create a hosts file.

        Each line in the hosts file corresponds to a compute node. The format is [host_name]  [host_ip]. For example:

        cat hosts
sge-hyxxx-comp01 10.18.xxx.xxx
sge-hyxxx-comp02 10.11.xxx.xxx

      3. Run the script to stop the services.

        sh batch_stop_clients.sh <cluster_configuration_information> ./hosts

        Replace <cluster_configuration_information> with the cluster configuration information that you obtained. For example:

        sh batch_stop_clients.sh  ehpc-hz-e5WF****** manager 10.18.XXX.XX 000a******-vm***.cn-hangzhou.nas.aliyuncs.com:/ehpc-hz-e5WF****** manager 10.18.XX.XX ./hosts
Previous：Create and deploy a hybrid cloud clusterNext：Best practices for hybrid cloud proxy mode