SAP on Alibaba Cloud high availability architecture

1.3.1 Pacemaker cluster

Pacemaker is an open source high availability cluster resource manager for Linux, which is built into SUSE Linux Enterprise Server for SAP Applications (SLES for SAP). Pacemaker handles:

• Monitoring the health of cluster nodes and resources

• Automatically migrating resources when it detects a failure

• Managing virtual IP address failover

• Coordinating quorum among cluster members

1.3.2 Corosync

Corosync provides cluster communication. It handles heartbeat detection and message passing between nodes. In Alibaba Cloud environments, use unicast (UCAST) mode for communication.

1.3.3 STONITH/fencing

STONITH (Shoot The Other Node In The Head) isolates failed nodes to prevent split-brain scenarios. Alibaba Cloud provides a dedicated STONITH device fence_aliyun. It uses Alibaba Cloud OpenAPI to force restart or shut down ECS instances.

1.3.4 SAP Enqueue Replication

The SAP Enqueue Server manages lock tables in SAP systems. Enqueue Replication maintains a synchronized copy of the lock table on the standby node to preserve lock information if the primary node fails. Two versions exist:

• ENSA1 (Enqueue Replication Server 1): For SAP NetWeaver 7.40 and 7.50

• ENSA2 (Enqueue Replication Server 2): For SAP S/4HANA 1809 and later. This is the current standard.

1.3.5 SAP HANA System Replication

SAP HANA System Replication (HSR) is a built-in data replication mechanism for HANA databases. It supports both synchronous (sync) and asynchronous (async) modes. In HA deployments, use synchronous mode. This ensures that the primary database waits for the secondary database to persist each commit before acknowledging the transaction.

2.1.1 Selection guidelines

Note: Compared with standard SLES, SLES for SAP Applications has the following key differences:

• Pre-integrated with SAP-specific high availability resource agent packages (such as SAPInstance and SAPHanaSR).

• Access to a joint SAP and SUSE technical support channel.

• SUSE and SAP jointly verify its compatibility with SAP software.

2.1.2 Recommended versions

Note: For specific versions, refer to the SAP Product Availability Matrix (PAM) and official SUSE support statements.

2.1.3 Obtaining the operating system

You can obtain SLES or SLES for SAP Applications on Alibaba Cloud in two ways:

• Purchase a subscription from Marketplace: When you create an ECS instance, select an SLES or SLES for SAP Applications image from the Marketplace. The operating system subscription fee is billed with the ECS instance on a pay-as-you-go or subscription basis, eliminating the need for separate OS license management. This method is suitable for new cloud deployments or for simplifying license management.

• Bring Your Own License (BYOL): If you already have a valid SUSE subscription, you can use a custom image to apply your existing OS license to Alibaba Cloud ECS instances. You are responsible for registering your SUSE subscription and configuring the software repositories. This method is suitable if you have an existing enterprise agreement with SUSE or are migrating from an on-premises environment.

Important:

• All nodes in a cluster must run the same operating system version. Mixed-version deployments are not supported.

• The Pacemaker cluster configuration steps in the following sections are based on SLES for SAP Applications.

  These steps do not apply to nodes that use the standard SLES version.

2.2.1 ASCS/ERS nodes

ASCS and ERS nodes have low resource requirements. These nodes primarily consume CPU and memory for lock management and message routing.

Recommended instance families (ordered by generation priority):

Typical ASCS/ERS instance selection recommendations:

Note: Use the same instance type for both ASCS and ERS nodes. Either node might run both ASCS and ERS instances.

2.2.2 PAS/AAS nodes

PAS (Primary Application Server) and AAS (Additional Application Server) process actual business workloads. Select instance types based on your SAP sizing results. These nodes do not need clustering. Add more AAS instances to improve processing capacity and availability.

Common instance families include the following:

2.4.1 IP address planning

Key point: Use Overlay IP for virtual IPs on Alibaba Cloud (implemented via route tables). Manage these VIPs with the aliyun-vpc-move-ip resource agent. VIPs do not need to overlap with physical subnets. You can use a separate IP range.

2.4.2 Alibaba Cloud OpenAPI endpoints

Cluster STONITH devices and VIP resource agents use Alibaba Cloud OpenAPI. Within a VPC, access endpoints over the internal network. No NAT Gateway is required:

• ECS endpoint: ecs-vpc.<region-id>.aliyuncs.com

• VPC endpoint: vpc-vpc.<region-id>.aliyuncs.com

2.4.3 /etc/hosts configuration

All cluster nodes must include complete name resolution in /etc/hosts (example):

10.0.1.10      sapapp1          # ASCS node
10.0.2.10      sapapp2          # ERS node
10.0.100.11    vsapascs         # ASCS virtual hostname
10.0.100.12    vsapers          # ERS virtual hostname

2.5.1 Simple Mount architecture

The Simple Mount architecture uses NFS shares instead of cluster-managed file system resources. This approach greatly simplifies cluster configuration and maintenance.

NFS Server (Alibaba Cloud NAS)
├── /sapmnt/<SID>             ──  Mounted by all nodes over NFS
└── /usr/sap/<SID>            ──  Mounted by all cluster nodes over NFS
    ├── ASCS<xx>/             ──  ASCS instance directory
    ├── ERS<xx>/              ──  ERS instance directory
    └── SYS/                  ──  System directory

Local file system on each node:
/usr/sap/                     ──  Local XFS (contains sapservices, saphostagent)

Configure NFS mounts in each node's /etc/fstab:

<nas-endpoint>:/sapmnt/<SID>     /sapmnt/<SID>     nfs  defaults  0 0
<nas-endpoint>:/usr/sap/<SID>    /usr/sap/<SID>    nfs  defaults  0 0

2.5.2 Traditional architecture

In the traditional architecture, ASCS and ERS each use a separate NAS file system. Pacemaker's Filesystem resource agent controls mounting and unmounting:

NAS file system 1 --> /usr/sap/<SID>/ASCS<xx>  (mounted by cluster)
NAS file system 2 --> /usr/sap/<SID>/ERS<xx>   (mounted by cluster)
NFS share        --> /sapmnt                  (mounted at OS level)
NFS share        --> /usr/sap/<SID>/SYS       (mounted at OS level)

2.6.1 Prerequisites

• Operating system: SUSE Linux Enterprise Server for SAP Applications 15 SP1 or later

• Packages: sap-suse-cluster-connector >= 3.1.0, sapstartsrv-resource-agents >= 0.9.1, resource-agents >= 4.x

• Disable the systemd auto-start service for ASCS and ERS

• Install the Alibaba Cloud STONITH device (fence_aliyun) and VIP resource agent (aliyun-vpc-move-ip)

2.6.2 Alibaba Cloud-specific components

Install required dependency packages:

# Install libcurl-devel, fence-agents, pycurl, and pexpect
zypper install libcurl-devel
zypper install fence-agents
pip3 install pycurl pexpect

# Install Alibaba Cloud SDK
pip install aliyun-python-sdk-core aliyun-python-sdk-vpc aliyun-python-sdk-ecs

Install the latest fence_aliyun (STONITH device):

# Download fence_aliyun
curl https://raw.githubusercontent.com/ClusterLabs/fence-agents/refs/heads/main/agents/aliyun/fence_aliyun.py \
  -o /usr/sbin/fence_aliyun
chmod 755 /usr/sbin/fence_aliyun
chown root:root /usr/sbin/fence_aliyun
# Set Python path
sed -i "s|@PYTHON@|$(which python3 2>/dev/null || which python 2>/dev/null)|" /usr/sbin/fence_aliyun
sed -i "s|@FENCEAGENTSLIBDIR@|/usr/share/fence|" /usr/sbin/fence_aliyun

Install aliyun-vpc-move-ip (VIP resource agent):

mkdir -p /usr/lib/ocf/resource.d/aliyun
curl https://raw.githubusercontent.com/ClusterLabs/resource-agents/refs/heads/main/heartbeat/aliyun-vpc-move-ip \
  -o /usr/lib/ocf/resource.d/aliyun/vpc-move-ip
chmod 755 /usr/lib/ocf/resource.d/aliyun/vpc-move-ip
chown root:root /usr/lib/ocf/resource.d/aliyun/vpc-move-ip

Configure RAM role authentication:

Create a RAM policy named SAP-HA-ROLE-POLICY with these permissions:

{
    "Version": "1",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecs:StartInstance",
                "ecs:StopInstance",
                "ecs:RebootInstance",
                "ecs:Describe*"
            ],
            "Resource": ["*"]
        },
        {
            "Effect": "Allow",
            "Action": [
                "vpc:CreateRouteEntry",
                "vpc:DeleteRouteEntry",
                "vpc:Describe*"
            ],
            "Resource": ["*"]
        }
    ]
}

Create the RAM role SAP-HA-ROLE and attach the policy to it. Then, attach this RAM role to all ECS instances in the cluster:

In the console, select the target ECS instance, and then choose **Grant/Revoke RAM Role** from the **More** menu.

In the dialog box that appears, select the SAP-HA-ROLE that you created, and then apply the role.

Perform the same steps for the ECS nodes of the HANA cluster (as described in Section 3).

Configure Alibaba Cloud CLI authorization:

Install Alibaba Cloud CLI on the ECS instance:

/bin/bash -c "$(curl -fsSL https://aliyuncli.alicdn.com/install.sh)"

Configure Alibaba Cloud CLI authorization:

# aliyun configure --profile ecsRamRoleProfile --mode EcsRamRole
Configuring profile 'ecsRamRoleProfile' in 'EcsRamRole' authenticate mode...
Ecs Ram Role []:
Default Region Id []:
Default Output Format [json]: json (Only support json)
Default Language [zh|en] en: 
Saving profile[ecsRamRoleProfile] ...Done.

• Ecs Ram Role: SAP-HA-ROLE

• Default Region Id: Enter the region ID where your ECS instances reside, such as cn-shanghai

2.6.3 SSH trust between nodes

Before initializing the cluster, you must configure passwordless SSH access (also known as SSH trust) between all cluster nodes. The ha-cluster-join command uses SSH to connect to existing nodes, and cannot run non-interactively without this trust relationship.

The following steps use a two-node cluster (node 1: sap-ascs, node 2: sap-ers) as an example.

ssh-keygen -t rsa -b 4096 -N "" -f /root/.ssh/id_rsa

ssh-copy-id -i /root/.ssh/id_rsa.pub root@sap-ers

ssh-copy-id -i /root/.ssh/id_rsa.pub root@sap-ascs

ssh root@sap-ers "hostname"

ssh root@sap-ascs "hostname"

ssh-keyscan -H sap-ers >> /root/.ssh/known_hosts
ssh-keyscan -H sap-ascs >> /root/.ssh/known_hosts

2.6.4 Cluster initialization

Run the ha-cluster-init script on node 1.

ha-cluster-init -y -i eth0 -u

Run the ha-cluster-join script on node 2.

ha-cluster-join -y -c <node1_name_or_ip_address> -i eth0

In Alibaba Cloud environments, use unicast (UCAST) mode with the udpu parameter ("-u").

2.6.5 ENSA2 Simple Mount resource configuration

Prerequisite check

In Simple Mount configurations, the sapstartsrv service becomes a cluster resource. The cluster independently manages which node runs the ASCS and ERS sapstartsrv services.

Verify that the sapstartsrv-resource-agents resource agent is installed on your operating system.

# Check whether the sapstartsrv-resource-agents resource agent is installed
ls /usr/lib/ocf/resource.d/suse/SAPStartSrv

# If the file above is missing, install the package
zypper install sapstartsrv-resource-agents

This package comes from the SLE-Module-SAP-Applications module. If zypper cannot find the package, verify that the module is enabled:

SUSEConnect --list-extensions | grep SAP

If the module is not active, enable it:

SUSEConnect -p sle-module-sap-applications/<version>/x86_64

Because the cluster controls the sapstartsrv service, disable the SAP system systemd auto-start service. This prevents conflicts between sapstartsrv processes during failover, which could stop the cluster from starting SAP resources correctly.

# Check whether the SAP instance systemd auto-start service is enabled on this machine
systemctl list-unit-files | grep SAP

# Disable the service's auto-start based on the output above
systemctl disable SAP<SID>_<instance number>   # ASCS
systemctl disable SAP<SID>_<instance number>   # ERS

Sample cluster script

The following example shows a complete Pacemaker resource configuration for ENSA2 Simple Mount on Alibaba Cloud (SID=EN2, ASCS instance number=00, ERS instance number=10):

# === Cluster global properties ===
property cib-bootstrap-options: \
    stonith-enabled="true" \
    stonith-action="reboot" \
    stonith-timeout="150"

rsc_defaults rsc-options: \
    resource-stickiness="1" \
    migration-threshold="3"

op_defaults op-options: \
    timeout="600" \
    record-pending=true

# === STONITH resource (Alibaba Cloud fence_aliyun) ===
primitive res_ALIYUN_STONITH_1 stonith:fence_aliyun \
        op monitor interval=120 timeout=60 \
        params filter="InstanceIds=[\"<ECS Id1>\", \"<ECS Id2>\"]" plug=<ECS Id1> ram_role=SAP-HA-ROLE region=<region ID> \
        meta target-role=Started

primitive res_ALIYUN_STONITH_2 stonith:fence_aliyun \
        op monitor interval=120 timeout=60 \
        params filter="InstanceIds=[\"<ECS Id1>\", \"<ECS Id2>\"]" plug=<ECS Id2> ram_role=SAP-HA-ROLE region=<region ID> \
        meta target-role=Started

# STONITH location constraints: A node does not run its own STONITH resource
location loc_stonith1_not_on_sapapp1 res_ALIYUN_STONITH_1 -inf: sapapp1
location loc_stonith2_not_on_sapapp2 res_ALIYUN_STONITH_2 -inf: sapapp2

# === ASCS resources ===
# ASCS virtual IP (using Alibaba Cloud vpc-move-ip)
# The routing_table parameter value is the default route table ID of the VPC where the ECS instances reside
primitive rsc_ip_EN2_ASCS00 ocf:aliyun:vpc-move-ip \
    params address=<ascs-vip> \
           routing_table=<routing-table-id> \
           endpoint=vpc-vpc.<region-id>.aliyuncs.com \
           interface=eth0 \
    op monitor interval=10s timeout=20s

# ASCS SAPStartSrv resource (Simple Mount architecture only)
primitive rsc_SAPStartSrv_EN2_ASCS00 ocf:suse:SAPStartSrv \
    params InstanceName=EN2_ASCS00_sapen2as

# ASCS SAPInstance resource
primitive rsc_sap_EN2_ASCS00 SAPInstance \
    op monitor interval=11 timeout=60 on-fail=restart \
    params InstanceName=EN2_ASCS00_sapen2as \
           START_PROFILE="/sapmnt/EN2/profile/EN2_ASCS00_sapen2as" \
           AUTOMATIC_RECOVER=false \
    meta resource-stickiness=5000 failure-timeout=60 \
         migration-threshold=1 priority=10

# === ERS resources ===
# ERS virtual IP
# The routing_table parameter value is the default route table ID of the VPC where the ECS instances reside
primitive rsc_ip_EN2_ERS10 ocf:aliyun:vpc-move-ip \
    params address=<ers-vip> \
           routing_table=<routing-table-id> \
           endpoint=vpc-vpc.<region-id>.aliyuncs.com \
           interface=eth0 \
    op monitor interval=10s timeout=20s

# ERS SAPStartSrv resource
primitive rsc_SAPStartSrv_EN2_ERS10 ocf:suse:SAPStartSrv \
    params InstanceName=EN2_ERS10_sapen2er

# ERS SAPInstance resource
primitive rsc_sap_EN2_ERS10 SAPInstance \
    op monitor interval=11 timeout=60 on-fail=restart \
    params InstanceName=EN2_ERS10_sapen2er \
           START_PROFILE="/sapmnt/EN2/profile/EN2_ERS10_sapen2er" \
           AUTOMATIC_RECOVER=false IS_ERS=true \
    meta priority=1000

# === Resource groups ===
group grp_EN2_ASCS00 rsc_ip_EN2_ASCS00 rsc_SAPStartSrv_EN2_ASCS00 rsc_sap_EN2_ASCS00 \
    meta resource-stickiness=3000
group grp_EN2_ERS10 rsc_ip_EN2_ERS10 rsc_SAPStartSrv_EN2_ERS10 rsc_sap_EN2_ERS10

# === Constraints ===
# ASCS and ERS cannot run on the same node
colocation col_sap_EN2_not_both -5000: grp_EN2_ERS10 grp_EN2_ASCS00

# ASCS starts on the ERS node after failover
location loc_sap_EN2_failover_to_ers rsc_sap_EN2_ASCS00 \
    rule 2000: runs_ers_EN2 eq 1

# ASCS starts before ERS
order ord_sap_EN2_first_ascs Optional: rsc_sap_EN2_ASCS00:start rsc_sap_EN2_ERS10:stop

2.6.6 ENSA1 traditional resource configuration

For SAP NetWeaver 7.40/7.50, use the ENSA1 architecture. Key differences from ENSA2 include the following:

• Use the Filesystem resource agent to manage mounting and unmounting ASCS and ERS file systems

• Do not use the SAPStartSrv resource agent

• Include Filesystem resources in resource groups

# ASCS file system resource (traditional architecture)
primitive rsc_fs_EN2_ASCS00 Filesystem \
    params device="<nas-mount-point>:/" \
           directory="/usr/sap/EN2/ASCS00" \
           fstype=nfs \
           options="vers=4,minorversion=0,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2" \
    op start timeout=60s interval=0 \
    op stop timeout=60s interval=0 \
    op monitor interval=20s timeout=40s

# Resource group includes file system
group grp_EN2_ASCS00 rsc_ip_EN2_ASCS00 rsc_fs_EN2_ASCS00 rsc_sap_EN2_ASCS00 \
    meta resource-stickiness=3000

2.6.7 sap-suse-cluster-connector integration

The sap-suse-cluster-connector implements the API v3 communication interface between SAP sapstartsrv and the Pacemaker cluster. This ensures that operations performed by SAP administrators using sapcontrol or SAP MMC notify the cluster correctly. This avoids false cluster alerts.

Install and configure:

# Install
zypper install sap-suse-cluster-connector

# Add the <sid>adm user to the haclient group
usermod -a -G haclient <sid>adm

# Add these lines to the SAP instance profile
service/halib = $(DIR_EXECUTABLE)/saphascriptco.so
service/halib_cluster_connector = /usr/bin/sap_suse_cluster_connector

3.1.1 Performance optimized solution

Preload tables on the secondary node so data stays in memory. Failover time is usually short. Supports the logreplay_readaccess operation mode. This allows read-only queries on the secondary node.

3.1.2 Cost optimized solution

During normal operation, the secondary node also runs a non-replicated SAP HANA instance (such as QAS). Pacemaker anti-colocation constraints manage this. When the primary database needs to fail over, the cluster first stops the non-replicated instance, then performs takeover.

3.7.1 Configure HANA System Replication

Before configuring the cluster, set up HANA System Replication:

# Enable SR on the Primary node (hana01)
hdbnsutil -sr_enable --name=SiteA

# Register on the Secondary node (hana02)
hdbnsutil -sr_register --remoteHost=hana01 --remoteInstance=<inst_nr> \
    --replicationMode=sync --operationMode=logreplay --name=SiteB

3.7.2 Configure HA/DR provider hook scripts

SUSE provides three key HA/DR provider hook scripts. Configure them in HANA's global.ini:

susHanaSR.py — Monitors SR connection state changes (required):

[ha_dr_provider_sushanasr]
provider = susHanaSR
path = /usr/share/SAPHanaSR-angi/
execution_order = 1

[trace]
ha_dr_sushanasr = info

susTkOver.py — Checks before takeover to block unexpected manual takeovers:

[ha_dr_provider_sustkover]
provider = susTkOver
path = /usr/share/SAPHanaSR-angi/
execution_order = 2

[trace]
ha_dr_sustkover = info

susChkSrv.py — Monitors service state changes to speed up failover when the indexserver fails:

[ha_dr_provider_suschksrv]
provider = susChkSrv
path = /usr/share/SAPHanaSR-angi/
execution_order = 3
action_on_lost = stop

[trace]
ha_dr_suschksrv = info

Configure sudo permissions:

Create /etc/sudoers.d/SAPHanaSR:

<sid>adm ALL=(ALL) NOPASSWD: /usr/sbin/crm_attribute -n hana_<sid>_*
<sid>adm ALL=(ALL) NOPASSWD: /usr/bin/SAPHanaSR-hookHelper --sid=<SID> *

3.7.3 Configure STONITH/Fencing

Refer to the fence_aliyun solution described in section 2.6.2 Installing Alibaba Cloud-Specific Components.

3.7.4 Configure HANA cluster resources (performance optimized)

Here is a complete Pacemaker cluster resource configuration example for HANA (SID=HA1, instance number=10):

# === Cluster Global Properties ===
property cib-bootstrap-options: \
    stonith-enabled="true" \
    stonith-action="reboot" \
    stonith-timeout="150" \
    priority-fencing-delay="30"

rsc_defaults rsc-options: \
    resource-stickiness="1000" \
    migration-threshold="5000"

op_defaults op-options: \
    timeout="600" \
    record-pending=true

# === STONITH ===

# Alibaba Cloud fence_aliyun (similar to the application layer)
primitive res_ALIYUN_STONITH_1 stonith:fence_aliyun \
        op monitor interval=120 timeout=60 \
        params filter="InstanceIds=[\"<ECS Id1>\", \"<ECS Id2>\"]" plug=<ECS Id1> ram_role=SAP-HA-ROLE region=<region ID> \
        meta target-role=Started
primitive res_ALIYUN_STONITH_2 stonith:fence_aliyun \
        op monitor interval=120 timeout=60 \
        params filter="InstanceIds=[\"<ECS Id1>\", \"<ECS Id2>\"]" plug=<ECS Id2> ram_role=SAP-HA-ROLE region=<region ID> \
        meta target-role=Started
location loc_node1_stonith_not_on_node1 res_ALIYUN_STONITH_1 -inf: <node1>
location loc_node2_stonith_not_on_node2 res_ALIYUN_STONITH_2 -inf: <node2>

# === SAPHanaTopology ===
primitive rsc_SAPHanaTop_HA1_HDB10 ocf:suse:SAPHanaTopology \
    op start interval=0 timeout=600 \
    op stop interval=0 timeout=300 \
    op monitor interval=50 timeout=600 \
    params SID=HA1 InstanceNumber=10

clone cln_SAPHanaTop_HA1_HDB10 rsc_SAPHanaTop_HA1_HDB10 \
    meta clone-node-max=1 interleave=true

# === SAPHanaController ===
primitive rsc_SAPHanaCon_HA1_HDB10 ocf:suse:SAPHanaController \
    op start interval=0 timeout=3600 \
    op stop interval=0 timeout=3600 \
    op promote interval=0 timeout=900 \
    op demote interval=0 timeout=320 \
    op monitor interval=60 role=Promoted timeout=700 \
    op monitor interval=61 role=Unpromoted timeout=700 \
    params SID=HA1 InstanceNumber=10 \
           PREFER_SITE_TAKEOVER=true \
           DUPLICATE_PRIMARY_TIMEOUT=7200 \
           AUTOMATED_REGISTER=false \
    meta priority=100

clone mst_SAPHanaCon_HA1_HDB10 rsc_SAPHanaCon_HA1_HDB10 \
    meta clone-node-max=1 promotable=true interleave=true maintenance=true

# === Virtual IP ===
# Primary VIP
primitive rsc_ip_HA1_HDB10 ocf:aliyun:vpc-move-ip \
     params address=10.0.100.20 routing_table=<rt-id> \
            endpoint=vpc-vpc.<region-id>.aliyuncs.com interface=eth0

# === Constraints ===
# The VIP follows the promoted HANA instance
colocation col_saphana_ip_HA1_HDB10 2000: \
    rsc_ip_HA1_HDB10:Started mst_SAPHanaCon_HA1_HDB10:Promoted

# The topology resource must start before the controller resource
order ord_saphana_HA1_HDB10 Optional: \
    cln_SAPHanaTop_HA1_HDB10 mst_SAPHanaCon_HA1_HDB10

3.7.5 Key parameters

3.7.6 Configure Active/Active Read-Enabled (optional)

If you enable the logreplay_readaccess operation mode, add a read-only VIP for the secondary node:

primitive rsc_ip_HA1_HDB10_readenabled ocf:aliyun:vpc-move-ip \
     params address=10.0.100.20 routing_table=<rt-id> \
            endpoint=vpc-vpc.<region-id>.aliyuncs.com interface=eth0

colocation col_saphana_ip_HA1_HDB10_readenabled 2000: \
    rsc_ip_HA1_HDB10_readenabled:Started mst_SAPHanaCon_HA1_HDB10:Unpromoted