Sending failure alerts with a Feishu bot

Background information

E-HPC integrates with EventBridge to deliver events, such as cluster node faults, to a dedicated event bus. This integration allows you to receive, query, and consume fault events. You can configure rules for E-HPC fault events, such as GPU or eRDMA faults, to send notifications to a Feishu group through a Feishu bot, ensuring you receive timely information about cluster node failures.

Notification workflow: A fault occurs in a cluster node → An EventBridge event rule is triggered → EventBridge sends the event to the Feishu bot webhook address → The Feishu bot sends a message to the Feishu group.

Prerequisites

Before you begin, complete the following steps:

1. Activate EventBridge
   Log on to the EventBridge console and click Activate Now to activate the service.
   

2. (Optional) Upgrade the cluster client version
   To use the fault event notification feature, upgrade the EHPCUtils client on your cluster to a supported version. For instructions on how to upgrade a single node or the entire cluster, see the Upgrade the cluster client version section of this topic.
   

Step 1: Create a Feishu bot

Create a Feishu bot to get a unique webhook address. This address is used to connect to EventBridge and receive notifications.

1. Log on to the Feishu desktop client and select the group chat where you want to receive fault notifications.

2. On the group chat page, click the Settings icon in the upper-right corner to go to the Settings page.

3. On the Settings page, click Group Bots.

4. On the Add Bots page, click Custom Bot.

5. On the Create Bot page, enter a custom name and description for the bot, and then click Add.

6. On the Webhook address page, copy and save the webhook address. In the Security Settings section, you can select Custom Keywords and enter a keyword, such as "cluster". Then, click Finish.

For more information, see the Feishu for Developers documentation: Add a custom bot.

Step 2: Configure an E-HPC fault event rule

An event rule listens for specific types of fault events. When an event matches the rule, EventBridge sends the event to the Feishu bot, which in turn delivers it to the Feishu group.

1. Log on to the EventBridge console.

2. In the left-side navigation pane, click Event Bus.

3. In the top navigation bar, select a region.
   Note: You cannot configure events across regions.
   

4. On the Event Bus page, click default to go to the default event bus configuration page.

5. In the left-side navigation pane, click Event Rules, and then click Create Rule.

6. In the Create Rule panel, follow the on-screen wizard to complete the configuration.

{
    "Uid": "$.data.Uid",
    "SeverityLevel": "$.data.SeverityLevel",
    "ExecuteCmd": "$.data.ExecuteCmd",
    "ClusterId": "$.data.ClusterId",
    "InstanceId": "$.data.InstanceId",
    "InstanceIp": "$.data.InstanceIp",
    "InstanceName": "$.data.InstanceName",
    "FaultDetail": "$.data.FaultDetail",
    "FaultType": "$.data.FaultType",
    "FaultTarget": "$.data.FaultTarget",
    "InstanceType": "$.data.InstanceType",
    "MessageId": "$.data.MessageId",
    "DetectedAt": "$.data.DetectedAt",
    "aliyunregionid": "$.aliyunregionid",
    "eventID": "$.id"
}

{
  "msgtype": "text",
  "text": {
    "content": "Dear Alibaba Cloud customer ${Uid},\nThis is a notification that a node fault occurred in your Elastic High Performance Computing cluster ${ClusterId} in the ${aliyunregionid} region at ${DetectedAt}. Details are as follows:\n\tInstance ID: ${InstanceId}\n\tInstance IP: ${InstanceIp}\n\tInstance hostname: ${InstanceName}\n\tInstance type: ${InstanceType}\n\tFault type: ${FaultType}\n\tFault target: ${FaultTarget}\n\tMessage ID: ${MessageId}\n\tSeverity level: ${SeverityLevel}\n\tFault details: ${FaultDetail}\n\nPS: The event ID for this incident is ${eventID}. If you have any questions about using E-HPC, please provide this event ID when you submit a ticket."
  }
}

Verify the results

After you complete the configuration, EventBridge delivers matching fault events, such as ehpc:Maintenance:NodeGpuFault or ehpc:Maintenance:NodeErdmaFault, to the Feishu bot, which then sends a notification to your Feishu group. You can also use the test event sending feature in the EventBridge console to verify that Feishu can receive messages correctly.

Fault details and troubleshooting steps

After you receive a fault notification, you can use the following tables to identify the fault type and perform initial troubleshooting.

Run checks in bulk (optional)

If you need to proactively check multiple nodes for errors, you can use the Run Command feature in the E-HPC console to run a diagnostic script on multiple nodes simultaneously.

1. Go to the E-HPC console, click Cluster in the left-side navigation pane, and go to your target cluster.

2. In the list of nodes, select the nodes you want to check, and then click Run Command.

3. In the command window, enter the following script, and then click Execute:

#!/bin/bash
hpcacc gpu check --format json
hpcacc erdma check --format readable

4. In the command output, view the check results for each node. If there are errors, refer to the tables in the "Fault details and troubleshooting steps" section for recommended actions.

(Optional) Upgrade the cluster client version

To use the fault event notification feature, ensure the E-HPC client on your cluster nodes, including the ehpc-faultevent.service, meets the required version.

/usr/local/ehpc/bin/ehpcutil upgrade --target_version 2.0.47 --checksum 2.0.47
echo "2.0.41" > /usr/local/ehpc_2.0.47/agent/version
systemctl restart ehpc-agent

# Update HPC-ACC
/usr/local/ehpc/bin/ehpcutil tools hpc_acc install --update
bash /usr/local/hpc_acc_util_latest/deploy.sh

echo "2.0.47" > /usr/local/ehpc_2.0.47/agent/version
/usr/local/ehpc/bin/ehpcutil upgrade --target_version 2.0.41 --checksum 2.0.41
systemctl restart ehpc-agent
systemctl stop ehpc-faultevent
systemctl disable ehpc-faultevent
rm -rf /etc/systemd/system/ehpc-faultevent.service