Enable auto repair for nodes in a managed node pool-Container Service for Kubernetes(ACK)-阿里云帮助中心

For Lingjun node pools, see Enable node auto repair.

How it works

The auto repair process covers:

Execution flow: End-to-end flow from fault detection to repair completion.
Node conditions that trigger auto repair: Fault types, risk levels, thresholds, and repair actions.
Node status during auto repair: Status transitions during and after repair.

Execution flow

The workflow from fault detection through notification to auto repair:

Fault diagnosis and detection	The ack-node-problem-detector (NPD) add-on checks for node exceptions. If a node remains unhealthy for a specified period, ACK identifies it as faulty.
Fault notification	ACK generates a Node Condition and a Kubernetes Event. Configure alerts in the Event Center to receive notifications.
(For exclusive GPU scenarios) Fault isolation	After a GPU exception is detected, ACK isolates the faulty GPU card. See GPU exception detection and automatic isolation.
Auto repair for system and Kubernetes component exceptions	ACK determines whether to initiate a repair task based on the node's `condition` and other information. Run `kubectl describe node` to check the `condition` field. When a system or Kubernetes component exception persists beyond the configured threshold, ACK executes a repair task: ACK repairs the faulty system and Kubernetes components, for example, by restarting the kubelet or the container runtime. If you select Enable self-healing for system and node components only (node reboots allowed), ACK performs these additional steps if initial repairs fail: ACK automatically sets the faulty node to unschedulable. ACK drains the node. The drain timeout is 10 minutes or the maximum `TerminationGracePeriodSeconds` of all Pods to evict, whichever is longer, up to 30 minutes. ACK evicts Pods to other nodes per the Pod Disruption Budget (PDB). For high availability, use multi-replica deployments, distribute workloads across nodes, and configure PDBs for critical services. If the drain fails, ACK continues the repair. ACK restarts the node. After the node recovers, ACK restores it to schedulable. Nodes already unschedulable before auto repair remain unschedulable after repair completes.

In clusters with multiple node pools, auto repair runs one node pool at a time.

Node conditions that trigger auto repair

Node Condition	Description	Risk level	Threshold	Repair action
KubeletNotReady(KubeletHung)	The kubelet is unresponsive, causing the node to become NotReady.	High	180s	Restart the kubelet. If the Enable self-healing for system and node components only (node reboots allowed) option is enabled, restart the ECS instance.
KubeletNotReady(PLEG)	The Pod Lifecycle Event Generator (PLEG) health check has failed, causing the node to report a NotReady status.	Medium	180s	Restart containerd or Docker. Restart the kubelet. If the Enable self-healing for system and node components only (node reboots allowed) option is enabled, restart the ECS instance.
KubeletNotReady(SandboxError)	PodSandbox not found, preventing the kubelet from starting correctly.	High	180s	Delete the corresponding sandbox container. Restart the kubelet.
RuntimeOffline	The container runtime (containerd or Docker) has stopped, making the node unavailable.	High	90s	Restart containerd or Docker. If the Enable self-healing for system and node components only (node reboots allowed) option is enabled, restart the ECS instance.
NTPProblem	The time synchronization service (ntpd or chronyd) is not working correctly.	High	10s	Restart ntpd or chronyd.
SystemdOffline	Abnormal systemd state prevents starting or destroying containers.	High	90s	If the Enable self-healing for system and node components only (node reboots allowed) option is enabled, restart the ECS instance.
ReadonlyFilesystem	The node's file system has become read-only.	High	90s	If the Enable self-healing for system and node components only (node reboots allowed) option is enabled, restart the ECS instance.

Node status during auto repair

During repair, the node status is Repairing.
If the fault is resolved after repair, the node returns to normal.
If the fault persists after repair, the node status becomes Recovery failed.

A Recovery failed node does not trigger another auto repair until the fault is resolved.

Usage notes

This feature requires the Event Center for alerts and ack-node-problem-detector (NPD) for exception detection. See Event monitoring.
Available only in ACK managed clusters for managed node pools.
The following features are in phased rollout. To request access, submit a ticket.
- Auto repair for node instance exceptions: subject to an allowlist.
- Alert rule sets: After enabling node auto repair, enable alert management and activate the Cluster Node Auto Repair Alert Rule Set and GPU monitoring and alert rules to receive exception alerts. These rule sets are in a phased release.
  
  See Container Service Alert Management.
- NPD version: Auto repair for node instance exceptions requires ack-node-problem-detector (NPD) 1.2.26 or later, currently in a phased release.

Enable node auto repair

Enable node auto repair when creating a node pool or for an existing managed node pool.

New node pool

On the ACK Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Node Pools.
Click Create Node Pool. In the Configure Managed Node Pool section, select Managed Node Pool, enable node auto repair, configure the node reboot policy for component repair, and create the node pool.

For configuration details, see Create and manage a node pool. For node reboot considerations, see below.

Existing node pool

On the ACK Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Node Pools.
In the node pool list, find the target node pool. In the Actions column, click the icon and select Enable Managed Node Pool (for a regular node pool) or Configure Managed Node Pool (for a managed node pool). Set the Configure Managed Node Pool mode to Managed Node Pool and enable node auto repair.

For configuration details, see Create and manage a node pool. For node reboot considerations, see below.

Auto repair events

ACK writes auto repair events to the Event Center. On the cluster details page, choose Operations > Event Center. On the Node Events tab, view auto repair records. Subscribe to these events via Event monitoring.

Event	Level	Description
NodeRepairStart	Normal	Node auto repair has started.
NodeRepairAction	Normal	A repair action was performed, such as restarting the kubelet.
NodeRepairSucceed	Normal	Node auto repair succeeded.
NodeRepairFailed	Warning	Repair failed. See FAQ.
NodeRepairIgnore	Normal	Skipped because the ECS instance was not running.

FAQ

Auto repair failure

Auto repair cannot resolve all failures. If repair fails or the fault persists, ACK sets the node status to Recovery failed.

If a node's auto repair fails, subsequent repairs for that node pool pause until the fault is resolved. You can submit a ticket for assistance.

References

Enable NPD and monitor cluster events via Event monitoring.
See GPU exception detection and automatic isolation and Diagnose GPU node issues.
To remove and re-add a faulty node, follow the standard procedures in the ACK console to avoid unexpected behavior. See Remove a node and Add existing nodes.