Automate GPU health checks and remediation via IM channels-Elastic Compute Service(ECS)-阿里云帮助中心

In large-scale GPU cluster operations, hardware failures, driver anomalies, and NVLink issues can directly disrupt model training and inference. By combining Elastic Compute Agent scheduled tasks with IM channel integration, you can proactively inspect GPU instance health at scale. When abnormalities are detected, Agent automatically pushes diagnostics to your operations chat group (Lark in this example), where operators can troubleshoot, apply one-click remediation, and respond to system events via conversation. You can also mention the bot at any time for ad-hoc resource inventory and health checks.

Solution Architecture and Flow

The end-to-end solution consists of four stages, forming a closed loop of "periodic inspection → abnormality push → in-chat conversation → closed-loop remediation":

Periodic Inspection: Elastic Compute Agent automatically runs 12 GPU health diagnostics against the specified instance scope at the configured schedule.
Abnormality Push: When issues are detected, Agent pushes the diagnostic results and suggested remediation to the designated IM operations group (Lark / DingTalk / WeCom).
In-chat Conversation: Operators mention the bot or reply to messages in the group. Agent uses the inspection context to further localize the issue and return actionable instructions.
Closed-loop Remediation: After confirming the suggestion, operators issue the remediation command directly in the group. Agent applies the one-click fix or responds to the ECS system event, and the console status is synced accordingly.

Diagnostic Coverage

Elastic Compute Agent ships with 12 GPU-specific diagnostics, covering the following two categories of root causes:

A. Driver and Software Component (3 items): Primarily in-guest software configuration issues, resolvable by issuing self-help remediation commands in the chat group (for example, installing or upgrading drivers/components).
B. GPU Hardware / Firmware / Low-level Faults (9 items): Underlying issues in hardware, firmware, memory, or interconnects. These typically require responding to ECS system events, scheduling a restart, or migrating the instance.

Category	Diagnostic Item	Detected Issue	Typical Impact
A Driver/Component	GPU driver not installed	GPU driver has not been installed	GPU card unavailable
A Driver/Component	GPU driver version mismatch	Driver version does not match the OS kernel version	GPU card cannot function normally
A Driver/Component	FabricManager component anomaly	FabricManager fails to load (commonly due to Tesla driver version mismatch)	GPU card cannot function normally
B Hardware/Firmware	GPU dropped (DeviceLost)	GPU dropped due to hardware failure	GPU card unavailable
B Hardware/Firmware	GPU power cable anomaly	GPU power cable connection abnormal	GPU card unavailable
B Hardware/Firmware	Memory ECC error	GPU memory shows ECC (correctable / uncorrectable) errors	Partial workloads run abnormally
B Hardware/Firmware	NVLink anomaly	NVLink status between GPUs is abnormal	GPU anomaly
B Hardware/Firmware	Xid error	Xid error during GPU hardware/driver load	GPU card works abnormally
B Hardware/Firmware	NVXid software/hardware risk	Xid inspection detects software or hardware risk	GPU card cannot function normally
B Hardware/Firmware	RmInitAdapter failure	"RmInitAdapter failed" occurs during loading	GPU card works abnormally
B Hardware/Firmware	infoROM corruption	"infoROM is corrupted at gpu" occurs	GPU card works abnormally
B Hardware/Firmware	Info ROM version anomaly	GPU firmware Info ROM version is abnormal	GPU card cannot function normally

Prerequisites

Before you begin, ensure the following conditions are met:

Elastic Compute Agent is activated. On your first visit to the Elastic Compute Agent console, review the service agreement and click Activate Now.
You have GPU instances that need to be inspected under your account.
You have administrator or developer access to an enterprise or organizational IM platform (Lark is used in this article; the configuration flow is similar for DingTalk, WeCom, etc. See Configure IM Channels for details).

Procedure

Step 1: Create a Lark Application (Bot)

Sign in to the Lark Open Platform and click Use template to create an AI agent app.
On the template page, click Create Now. The system automatically generates the app. Record and keep the App ID and App Secret for later use when configuring the IM channel.

If you need to use the bot in an external group, do the following:

Under the "App Release" section on the left navigation, click Version Management and Release and configure the version information. In the External Sharing area, select Allow the bot to be added to external groups. If you also want external users to have 1-on-1 chats with the bot, select Allow external users to chat with the bot.
Click Save, then submit for review and release following the Lark workflow.

For detailed Lark app configuration, refer to the latest official Lark documentation. This article only lists the key items required for the integration with Elastic Compute Agent.

Step 2: Configure the Lark IM Channel in the Console

Go to the Elastic Compute Agent console. In the lower-left corner of the console, click Settings > IM Channels.
Click Add IM Channel. In the dialog, configure the following parameters:
- Channel Name: Enter a custom channel name.
- Platform: Select Lark.
- App ID / App Secret: Fill in the parameters obtained in Step 1.
- Role Configuration: Select the RAM role used by the IM channel. If this is your first time, go to the RAM console and refer to Create a RAM Role to create one.
Click OK to save. The system automatically verifies the callback URL. If the verification passes, the IM channel is available.

The bot's operation permissions within the IM group are determined by the associated RAM role. We recommend associating the IM channel with a RAM role that has operations permissions, so that remediation and event responses can be completed directly in the group. For details, see Permission Configuration.

Step 3: Verify Bot Connectivity and Add It to the Group

In the Lark client, sign in to the target enterprise/organization account (must be the same tenant used to create the app in Step 1).
Search for the bot you created in Step 1 in Lark and start a 1-on-1 chat directly. Verify that the bot responds to basic interactions.
Enter the target operations group. Click the ... icon in the upper-right corner, click Add Bot, search for the bot you created, and add it.
In the group, mention the bot and send a test message (for example: Hello). Confirm that the bot responds to group messages.

Step 4: Create a GPU Health Inspection Scheduled Task

In the Elastic Compute Agent console, click Scheduled Tasks > Task Templates on the left navigation bar.
Find the GPU Resource Health Inspection template and click Create Scheduled Task.
In the Create Scheduled Task dialog, configure the parameters:
- Resource Scope: Select the range of GPU instances to inspect. You can filter by region, tag, instance ID, and other dimensions.
- Schedule: Configure the inspection frequency. We recommend every 2 or 4 hours; adjust based on your business window.
- Notification Channel: First enable IM Channel, then select the Lark channel you configured in Step 2 and choose the target group from the drop-down list.
  
  To let the group appear in the drop-down list, mention the bot at least once in the target group first (see Step 3).
Click Save to create the task. The task runs automatically at the configured schedule. You can check the task card on the Configured Tasks tab, and view execution history and logs on the Execution Records tab.

After creation, we recommend clicking Run Now on the task card once to verify that the inspection scope, diagnostic items, and group notifications work as expected.

The inspection task publishes each abnormal resource as a pending-response event. Customer systems can also subscribe to these events to detect issues and handle remediation in a self-built operations platform. In that case, you can leave the IM channel disabled to avoid duplicate notifications.

Step 5: View Results and Close the Loop in the IM Group

Once the scheduled task runs, Agent automatically pushes abnormal instance details and remediation suggestions to the IM operations group. Operators can complete the entire handling loop by replying or quoting messages directly in the group, without logging in to the console:

1. Locate and One-Click Remediate

Operators reply to Agent directly in the group. Agent uses the inspection context to further localize the issue and return a remediation plan.

Example prompts:

List the abnormal instances in this inspection
Diagnose the GPU issue for i-bp1xxxxxxx

After confirming the suggestion, issue the remediation command directly in the group, and Agent will run the underlying operation:

Fix all issues
Help me fix it

2. Respond to System Operations Events

For GPU issues caused by Alibaba Cloud proactive events (such as hardware maintenance or system restart), Agent surfaces the related event info in the inspection result. You can confirm and respond to the event directly in the group.

Example prompts:

Show pending system events for i-bp1xxxxxxx
Accept the operations event for i-bp1xxxxxxx

After the event is handled, Agent posts the result back to the group, and the event status in the ECS console is synced automatically.

Additional Scenario: Ad-hoc Operations via Bot Mention

In addition to scheduled push, operators can mention the bot in the IM group at any time to issue ad-hoc operations or knowledge queries:

Resource Inventory: Count all ECS instances under this account grouped by region
Single-instance Diagnosis: Diagnose the current health status of i-bp1xxxxxxx
Product Consultation: Ask Agent about GPU instance types, driver compatibility, and other topics to get standard knowledge answers.

Cost Reference

GPU inspection is metered in Credits. A basic single-instance inspection consumes approximately 1 Credit (approximately USD 0.007). Deep GPU inspection modes will be introduced in the future.

Taking a fleet of 100 GPU instances as an example, the estimated cost is as follows:

Instance Count	Cost per Inspection	Every 2 Hours (Daily Cost)	Every 4 Hours (Daily Cost)
100	Approx. USD 0.7	Approx. USD 8.4	Approx. USD 4.2

The costs above are for reference only; actual charges are based on console metering.

In the lower-left corner of the Elastic Compute Agent console, click Settings > Usage and Quota to view the total Credits, consumed Credits, remaining quota, and usage trend in real time. For details, see Billing.