Container Service for Kubernetes (ACK) node pools support customizable auto repair. On ECS and Lingjun nodes, including super nodes, ACK automatically detects GPU faults and underlying node faults, and repairs them by using the steps that you define. Track the repair progress through Kubernetes Events.
Node auto repair is a whitelist feature. To enable it, contact technical support.
Scope
-
Node pool types: Elastic Compute Service (ECS) node pools and Lingjun node pools, including super nodes.
-
Fault types: node faults, GPU faults, and system and component faults.
-
Add-on dependency: the
ack-node-problem-detector(NPD) add-on 1.2.36 or later, required for detecting GPU faults and system and component faults. -
Region limitations: not yet available in dedicated cloud regions.
Auto repair mechanism
Node auto repair continuously monitors the fault signals reported by nodes, such as CloudMonitor events, underlying system events, and the GPU, system, and component faults detected by NPD. When a fault is detected, ACK converts it into a Kubernetes Event and a Node Condition, then follows the configured auto repair rules to repair the node. Repair progress is reported through Kubernetes Events.
Auto repair process
-
Fault detection: ACK continuously monitors underlying node events and the anomalies reported by NPD. When a fault is detected, ACK adds the corresponding Node Condition to the node and generates a repair plan.
-
Fault handling: ACK runs the repair steps in sequence based on the auto repair rule. Each step can be enabled or skipped, and any enabled step can require manual intervention, in which case ACK proceeds only after authorization is granted.
-
Node isolation: Marks the faulty node as unschedulable to prevent new workloads from being scheduled on it.
-
Node drain: Evicts the workloads that run on the node.
-
Automatic repair: Initiates a repair action, such as a hardware repair or node reboot, based on the fault type.
-
Remove node isolation: Restores the node to a schedulable state after the repair is complete.
-
ACK handles faults by priority. Repair-type faults are high priority and are handled first. Reboot-type faults are low priority and are deferred until all high-priority faults are resolved.
Precautions
-
If the cluster already runs other software that provides node auto repair, perform a thorough evaluation before you use this feature to avoid conflicts between multiple auto repair mechanisms.
-
If manual intervention is enabled for any repair step, grant the authorization without delay. Otherwise, ACK pauses the current step and all subsequent steps, and the node may remain in an unhealthy state.
-
When a node-type fault occurs on a Lingjun node, ACK repairs it through the Lingjun approval-based O&M operation during the repair phase. Closely monitor both the Lingjun node's own status and its node status in the ACK cluster to prevent prolonged service disruption.
-
If the node auto repair process fails, log on to the cluster as soon as possible, repair the node manually, and restore it to a schedulable state. For assistance, contact technical support.
-
After you enable node auto repair, make sure that you enable alert management and activate the GPU monitoring and alert rules, Alert Rules for Node Pool O&M Events, and Cluster Node Auto Repair Alert Rule Set alert rule sets. For more information, see Configure alerts.
Configure auto repair rules
Procedure
From activation to taking effect, node auto repair typically involves the following steps:
-
Configure auto repair rules: In the managed configurations of a new or existing node pool, create an auto repair rule and specify the fault types and repair process. For more information, see Auto repair configuration.
-
Configure throttling and circuit breaking (optional): In the managed configurations, set the maximum number of nodes repaired in parallel and the circuit breaking threshold to prevent repair operations from overloading the cluster during large-scale faults. For more information, see Auto repair throttling and circuit breaking.
-
Enable automatic fault diagnosis (optional, for ECS node pools only): In the managed configurations, select Automatic Fault Diagnosis to enable in-depth diagnosis of GPU hardware faults. For more information, see Enable automatic fault diagnosis (ECS in-depth diagnosis).
-
Enable alerts: Activate alert management and the alert rule sets for node auto repair. For more information, see Configure alerts.
-
Monitor and handle events: Monitor the auto repair progress through Kubernetes Events (for more information, see Kubernetes events for node auto repair). Grant authorization as soon as ACK requests it, and intervene manually if the process fails.
Auto repair configuration
Enable node auto repair and define auto repair rules when you create a node pool or modify an existing node pool. After the feature is enabled, you can also add and manage rules centrally on the auto repair rule management page.
Entry points
-
Method 1: Configure a new node pool. When you create a node pool, in the Configure Managed Node Pool area, select Custom Node Management. Then follow the on-screen instructions to enable Auto Repair and create auto repair rules. For a full description of the configuration items, see Create and manage node pools.
-
Method 2: Configure an existing node pool. In the node pool list, find the target node pool and click
> Enable Managed Node Pool (if managed mode is not enabled) or Configure Managed Node Pool (if managed mode is already enabled) in the Actions column. Follow the on-screen instructions to create auto repair rules and enable node auto repair.
Configure an auto repair rule
-
Access the rule configuration page: Open the Create Self-Healing Rule panel from the node pool creation or configuration page, or click Create Self-Healing Rule on the Self-Healing Rule page.
-
Configure sub-rules:
-
Select the fault type and the repair action: Based on the fault type (node faults, GPU faults, or system and component faults), select the corresponding repair action (see Faults).
-
Configure the repair process: All of the following steps support manual intervention. When manual intervention is enabled, ACK runs the step only after authorization is granted.
Parameter
Description
Node isolation
Marks the faulty node as unschedulable (
spec.unschedulable=true) to prevent new workloads from being scheduled on it.Node drain
Requires Node isolation. Evicts the workloads on the node one by one and respects Pod Disruption Budget (PDB) constraints. The default timeout is 300 seconds and is configurable. Pods managed by a DaemonSet or an OpenKruise Advanced DaemonSet, and Pods labeled
alibabacloud.com/evict-pod=skip, are not evicted. The process advances to the next step after the timeout elapses.Automatic repair
Executes the repair action that corresponds to the fault type (for more information, see Faults).
Remove node isolation
Requires Node isolation. Makes the node schedulable again (
spec.unschedulable=false). If the node was already unschedulable before the repair process started, this step does not restore it to a schedulable state.
-
Auto repair throttling and circuit breaking
After you select Custom Node Management and enable Auto Repair on the Configure Managed Node Pool page, the auto repair throttling and circuit breaking policy takes effect with its default settings.
Default policy
By default, when multiple nodes become abnormal at the same time, ACK repairs the abnormal nodes in the node pool one at a time, and circuit breaking is not triggered.
Modify the default policy
If the default policy does not meet your requirements, click Modify Auto Repair Throttling and Circuit Breaking Policy and configure the following parameters:
|
Parameter |
Description |
|
Maximum number of nodes repaired in parallel |
The maximum number of nodes that ACK repairs at the same time when many nodes in the node pool are abnormal. |
|
Circuit breaking condition |
When the number of faulty nodes exceeds this threshold, auto repair enters the circuit breaking state and stops initiating new repair operations. |
-
Make sure that the circuit breaking threshold is greater than or equal to the maximum number of nodes repaired in parallel. Otherwise, ACK may trigger circuit breaking and stop repair operations before the number of faulty nodes reaches the parallel repair limit.
-
When a repair-type or reboot-type fault that affects a super node occurs in the node pool, ACK must repair all sub-nodes of the super node at the same time. If the maximum number of nodes repaired in parallel is less than the number of sub-nodes in a single super node, such faults cannot be repaired. Set the maximum number of nodes repaired in parallel to an appropriate value.
Enable automatic fault diagnosis (ECS in-depth diagnosis)
After automatic fault diagnosis is enabled, ACK uses the ack-node-problem-detector (NPD) node-side monitoring probe to detect potential anomalies and proactively initiate in-depth diagnostics of the GPU hardware. When an anomaly is confirmed, ACK raises alerts through Kubernetes Events, Node Conditions, and ECS system events. Combine these alerts with auto repair rules for automatic response.
Applicability
-
The node pool must be an ECS node pool.
-
The NPD add-on version must be 1.2.36 or later.
How to enable
New node pool
When you create a node pool, in the Configure Managed Node Pool area, select Custom Node Management and then select the Automatic Fault Diagnosis checkbox.
Existing node pool
For managed node pools: In the node pool list, click 
> Configure Managed Node Pool in the Actions column for the target node pool. In the Configure Managed Node Pool dialog box, select the Automatic Fault Diagnosis checkbox.
For unmanaged node pools: In the node pool list, click > Enable Managed Node Pool in the Actions column for the target node pool. In the Enable Managed Node Pool dialog box, select Custom Node Management and then select the Automatic Fault Diagnosis checkbox.
The following NPD detection items trigger an ECS in-depth diagnosis:
|
Error item |
Description |
|
|
Double Bit ECC Error (DBE). This event is logged when the GPU detects an uncorrectable error. |
|
|
GPU internal microcontroller halt. |
|
|
NVLink error. |
|
|
GPU has fallen off the bus. |
|
|
Context Switch Timeout Error. |
|
|
A timeout occurs while waiting for the GSP core to respond to an RPC message. |
|
|
An error occurs in the code running on the GPU's GSP core. |
|
|
NVLink: RLW Error. |
|
|
GPU Recovery Action Changed. |
|
|
The device's external power cable is not properly connected. |
Configure alerts
After you enable node auto repair, make sure that you enable alert management and activate the following alert rule sets: GPU monitoring and alert rules, Alert Rules for Node Pool O&M Events, and Cluster Node Auto Repair Alert Rule Set. Configure a dedicated notification group or contact so that you receive alert notifications without delay when an exception occurs.
For information about how to enable rule sets, see ACK alert management.
Faults
The following tables describe the Node Conditions that trigger auto repair, the remediation actions, and the recovery criteria for each fault type.
Node faults
This section applies to underlying hardware anomalies and faults that require a reboot on ECS and Lingjun nodes, including super nodes.
Lingjun
|
Fault name |
Underlying event name |
Kubernetes Node Condition |
Description |
Remediation action |
Recovery criteria |
|
Lingjun node requires repair due to an underlying hardware anomaly |
resourceType: Node |
|
The Lingjun node has a hardware fault and requires repair. The repair may cause data loss on the node's local disks. |
Repair Lingjun node Important
Repairing a Lingjun node may cause data loss on its local disks. Back up your data in advance. If the container runtime and Kubelet directories are mounted on a local disk, remove the node from the ACK cluster and add it back after the repair is complete. |
Both of the following conditions are met:
|
|
Lingjun node requires a reboot due to an underlying anomaly |
resourceType: Node |
|
The Lingjun node has a fault. Reboot the node as soon as possible to attempt a repair. |
Reboot node |
Both of the following conditions are met:
|
|
Lingjun node requires repair due to a hardware anomaly (affected by a super node) |
resourceType: HyperNode |
|
The Lingjun super node has a hardware fault that requires repair. Authorize the repair as soon as possible. ACK initiates a separate auto repair flow for each sub-node. When all sub-node repair processes reach the repair step, ACK requests a single authorization to repair the entire super node. |
Repair Lingjun node Important
Repairing a Lingjun node may cause data loss on its local disks. Back up your data in advance. If the container runtime and Kubelet directories are mounted on a local disk, remove the node from the ACK cluster and add it back after the repair is complete. |
Both of the following conditions are met:
|
|
Lingjun node requires a reboot due to an underlying anomaly (affected by a super node) |
resourceType: HyperNode |
|
The Lingjun super node has a fault. Authorize the repair as soon as possible. ACK initiates a separate auto repair flow for each sub-node. When all sub-node repair processes reach the repair step, ACK requests a single authorization to repair the entire super node. |
Reboot node |
Both of the following conditions are met:
|
ECS
|
Fault name |
Underlying event name |
Kubernetes Node Condition |
Description |
Remediation action |
Recovery criteria |
|
Instance redeployment required for system maintenance (Scheduled) |
|
|
Alibaba Cloud has detected a potential software or hardware risk on the host machine of the ECS instance, which may require an instance redeployment (not yet a fault). Initiate the redeployment immediately to mitigate the risk. Otherwise, ECS will trigger the redeployment after 24 to 48 hours. For more information, see Overview of ECS system events. |
Redeploy node Important
Data on the local disk may be lost during the node redeployment. Back up your data in advance. |
Both of the following conditions are met:
|
|
Instance redeployment required for system maintenance (Inquiring) |
|
|
Alibaba Cloud has detected a potential software or hardware risk on the host machine of the ECS instance, which may require an instance redeployment (not yet a fault). Initiate the redeployment immediately to mitigate the risk. For more information, see Overview of ECS system events. |
Redeploy node Important
Data on the local disk may be lost during the node redeployment. Back up your data in advance. |
Both of the following conditions are met:
|
|
Instance reboot required for system maintenance (Scheduled) |
|
|
Alibaba Cloud has detected a potential software or hardware risk on the host machine of the ECS instance, which may require an instance reboot (not yet a fault). Initiate the reboot immediately to mitigate the risk. Otherwise, ECS will trigger the reboot after 24 to 48 hours. For more information, see Overview of ECS system events. |
Reboot node |
Both of the following conditions are met:
|
|
Instance reboot required for system maintenance (Inquiring) |
|
|
Alibaba Cloud has detected a potential software or hardware risk on the host machine of the ECS instance, which may require an instance reboot (not yet a fault). Initiate the reboot immediately to mitigate the risk. For more information, see Overview of ECS system events. |
Reboot node |
Both of the following conditions are met:
|
For a complete mapping of underlying events to Node Conditions and Kubernetes Events, see Underlying event to Node Condition conversion.
GPU faults
Detection of these faults depends on the NPD add-on. All GPU faults in this category share the same recovery criteria: the node status is Ready, and the triggering Node Condition is resolved.
Lingjun
|
Fault name |
Kubernetes Node Condition |
Description |
Remediation action |
|
|
|
Double Bit ECC Error (DBE). This event is logged when the GPU detects an uncorrectable error. |
Reboot node |
|
|
|
GPU internal microcontroller halt. |
Reboot node |
|
|
|
The GPU driver detects an uncorrectable error in the GPU memory. |
Reboot node |
|
|
|
The GPU driver detects an uncorrectable error in the GPU memory, and suppression has failed. |
Reboot node |
|
|
|
An error occurs in the code running on the GPU's GSP core. |
Reboot node |
|
|
|
GPU memory page fault. |
Reboot node |
|
|
|
The GPU has retired pages in a pending state. |
Reboot node |
|
|
|
Row remapping is pending and will be performed on the next GPU reset. |
Reboot node |
|
|
|
A timeout occurs while waiting for the GSP core to respond to an RPC message. |
Reboot node |
The remediation action for all GPU faults is to reboot the node. If a GPU fault occurs at the same time as a Lingjun node system event that requires repair, ACK does not automatically reboot the node, because a reboot during the repair may cause data loss on the local disk. In this case, the auto repair process fails. Configure a separate auto repair rule for repair-type faults to cover this scenario. For more information, see Lingjun node requires repair due to an underlying hardware anomaly.
ECS
|
Fault name |
Kubernetes Node Condition |
Description |
Remediation action |
|
|
|
Double Bit ECC Error (DBE). This event is logged when the GPU detects an uncorrectable error. |
Reboot node |
|
|
|
GPU internal microcontroller halt. |
Reboot node |
|
|
|
The GPU driver detects an uncorrectable error in the GPU memory. |
Reboot node |
|
|
|
The GPU driver detects an uncorrectable error in the GPU memory, and suppression has failed. |
Reboot node |
|
|
|
An error occurs in the code running on the GPU's GSP core. |
Reboot node |
|
|
|
GPU memory page fault. |
Reboot node |
|
|
|
The GPU has retired pages in a pending state. |
Reboot node |
|
|
|
Row remapping is pending and will be performed on the next GPU reset. |
Reboot node |
|
|
|
A timeout occurs while waiting for the GSP core to respond to an RPC message. |
Reboot node |
System and component faults
Detection of these faults depends on the NPD add-on.
|
Fault name |
Kubernetes Node Condition |
Description |
Remediation action |
Fault detection threshold |
Recovery criteria |
|
Node time synchronization service is abnormal |
|
The time synchronization service (ntpd or chronyd) is abnormal. |
Log on to the node and run repair commands |
10s |
The node status is |
|
Node container runtime is unavailable |
|
The container runtime (Containerd or Docker Engine) has stopped, and the node is unavailable. |
Log on to the node and run repair commands |
90s |
The node status is |
|
Node systemd service is abnormal |
|
The systemd service is abnormal, preventing containers from being started or destroyed. |
Reboot node |
90s |
The node status is |
|
Node is in NotReady state |
|
The Kubelet has stopped or its heartbeat is abnormal, causing the node to enter the NotReady state. |
Log on to the node and run repair commands |
180s |
For three consecutive probes, the node status is |
|
Node file system is in read-only state |
|
The node's file system has become read-only. |
Reboot node |
90s |
The node status is |
If you enable the option to allow node reboots during the Automatic repair stage, ACK attempts to reboot the node if the anomaly persists after the automatic repair action is executed.
Underlying event to Node Condition conversion
The ACK node auto repair component automatically converts the underlying events generated by Lingjun nodes, Lingjun super nodes, and ECS nodes into the corresponding Node Conditions, and generates Kubernetes Events. This section provides the complete status mapping, which you can refer to when you troubleshoot faults.
Underlying events are mapped to Node Conditions by using the following fields:
-
Type: Identifies the fault type. -
Reason: Identifies the current phase of the event.
Lingjun CloudMonitor events
The Lingjun control plane publishes hardware and system anomalies for Lingjun bare metal nodes as CloudMonitor (CMS) events. The lifecycle of these events transitions through Inquiring → Executing → Completed or Terminated. ACK periodically pulls the events, identifies the event type (such as Node.HardwareFault.HOST for a required repair or Node.FaultNeedReboot.HOST for a required reboot), and converts them into the corresponding Node Conditions on the node.
The following example uses the Node.HardwareFault.HOST CMS event for a Lingjun node hardware fault:
ECS system events
ECS nodes reuse the ECS system event framework. The event lifecycle includes states such as Inquiring, Scheduled, Executing, Completed, Avoided, Canceled, and Failed. ACK calls the ECS system event API to retrieve instance maintenance events, such as SystemMaintenance.Reboot and SystemMaintenance.Redeploy, and maps them to the corresponding Node Condition on the node.
The following describes the status changes for a SystemMaintenance.Reboot scheduled reboot maintenance event with an initial status of Inquiring:
Underlying event conversion for regular Lingjun nodes
Each underlying event for a regular Lingjun node is converted into the corresponding Node Condition and Kubernetes Event of that node in the cluster.
|
Event status |
Description |
Node Condition |
Kubernetes Event |
|
resourceType: Node |
A check for a node hardware fault is in progress. |
|
|
|
resourceType: Node |
Maintenance for a node hardware fault has started. |
|
|
|
resourceType: Node |
Maintenance for a node hardware fault is complete. |
|
|
|
resourceType: Node |
Maintenance for a node hardware fault is terminated. |
|
|
|
resourceType: Node |
A check for a fault that requires a node reboot is in progress. |
|
|
|
resourceType: Node |
The node reboot to fix a fault has started. |
|
|
|
resourceType: Node |
The node reboot to fix a fault is complete. |
|
|
|
resourceType: Node |
Maintenance for a fault that requires a node reboot is terminated. |
|
|
Underlying event conversion for Lingjun super nodes
Each underlying event for a Lingjun super node is converted into corresponding Node Conditions and Kubernetes Events for all sub-nodes within that super node.
|
Event status |
Description |
Node Condition |
Kubernetes Event |
|
resourceType: HyperNode |
A check for a node hardware fault is in progress. |
|
|
|
resourceType: HyperNode |
Maintenance for a node hardware fault has started. |
|
|
|
resourceType: HyperNode |
Maintenance for a node hardware fault is complete. |
|
|
|
resourceType: HyperNode |
Maintenance for a node hardware fault is terminated. |
|
|
|
resourceType: HyperNode |
A check for a fault that requires a node reboot is in progress. |
|
|
|
resourceType: HyperNode |
The node reboot to fix a fault has started. |
|
|
|
resourceType: HyperNode |
The node reboot to fix a fault is complete. |
|
|
|
resourceType: HyperNode |
Maintenance for a fault that requires a node reboot is terminated. |
|
|
Underlying maintenance event conversion for ECS nodes
ACK converts system maintenance events for ECS nodes into corresponding Node Conditions and Kubernetes Events.
|
Event status |
Description |
Node Condition |
Kubernetes Event |
|
|
A check for an instance reboot due to system maintenance is in progress. |
|
|
|
|
An instance reboot due to system maintenance is scheduled. |
|
|
|
|
An instance reboot due to system maintenance is in progress. |
|
|
|
|
The instance reboot due to system maintenance is complete. |
|
|
|
|
The instance reboot due to system maintenance has been avoided. |
|
|
|
|
The instance reboot due to system maintenance has failed. |
|
|
|
|
The instance reboot due to system maintenance is canceled, as the risk has been mitigated. |
|
|
|
|
A check for an instance redeployment due to system maintenance is in progress. |
|
|
|
|
An instance redeployment due to system maintenance is scheduled. |
|
|
|
|
An instance redeployment due to system maintenance is in progress. |
|
|
|
|
The instance redeployment due to system maintenance is complete. |
|
|
|
|
The instance redeployment due to system maintenance has been avoided. |
|
|
|
|
The instance redeployment due to system maintenance is canceled. |
|
|
Kubernetes events for node auto repair
After managed mode is enabled and an auto repair rule is associated with a node pool, ACK converts node fault events into Kubernetes Events and Node Conditions, and reports the repair plan progress through Kubernetes Events. View them with kubectl describe node <node-name> or kubectl get events.
Determine whether auto repair is successful
Auto repair is complete when the node's status returns to Ready, the triggering Node Condition is cleared (or the fault-specific success criteria are met), and a RepairPlanCompleted event with a success message is generated. If a NodeNotRecovery or RepairPlanFailed event occurs, the node has not recovered. In this case, log on to the cluster and repair the node manually as described in Precautions.
Common auto repair process events
|
Event (Reason) |
Level |
Description |
|
|
Warning |
A Lingjun node fault event is detected, or the status of a fault event is updated. |
|
|
Warning |
A repair plan is created. |
|
|
Warning |
The current step requires manual authorization. The event message describes the operation to authorize and how to authorize it. |
|
|
Warning |
The authorization label is added to the node labels. |
|
|
Normal |
The repair plan status is updated. This event records the current step and its result. |
|
|
Normal |
The Lingjun control plane is authorized to perform the repair. |
|
|
Warning |
After the repair is completed, the node status is checked every 30 minutes. This event is triggered if the node has not recovered. |
|
|
Warning |
The repair plan is complete. The event message indicates whether the repair succeeded or failed. |
|
|
Warning |
An ECS node fault event is detected, or the status of a fault event is updated. |
|
|
Warning |
A Lingjun super node fault event is detected, or the status of a fault event is updated. |
|
|
Warning |
Auto repair for a super node event is waiting for all sub-nodes of the super node to enter the repair phase. |
Events for special scenarios
|
Event (Reason) |
Level |
Description |
|
|
Normal |
The previous repair plan succeeded, but the current fault type differs from the previous one. The system waits for a 10-minute silence period before it creates a new repair plan. |
|
|
Warning |
The most recent auto repair process handled the current fault successfully, but the node has not recovered. Manual troubleshooting is required. |
|
|
Warning |
The repair plan for the current fault failed. The node remains unrecovered and requires manual troubleshooting. |
|
|
Warning |
The repair plan status update failed. |
|
|
Normal |
The authorization label failed to be set, and the auto repair process is aborted. |