Configure automated rollouts for Pods with high disk usage in ACS-Container Compute Service(ACS)-阿里云帮助中心

When the ephemeral storage for Pods in an ACS cluster runs low, you can use the acs-instance-helper component to configure an automated monitoring policy. This policy ensures business continuity by gracefully rolling out Pods with high disk usage to maintain cluster stability.

How it works

By default, ACS instances provide 30 GiB of free ephemeral storage. If this storage is exhausted, your services may be disrupted. Deploy the acs-instance-helper component to automatically roll out Pods that reach a 98% disk usage threshold, preventing service disruptions from disk exhaustion.

Disk pressure monitoring: acs-instance-helper continuously monitors the Pod condition. When the ephemeral storage usage of a Pod exceeds the threshold, the Pod's disk sufficient condition (ContainerHasSufficientDisk) is updated to False.
Trigger automated rollout: When this condition changes, the component automatically initiates a graceful Pod rollout. It combines the scale-up-before-scale-down strategy of a stateless workload like a Deployment with the scheduling policy of a PodDisruptionBudget (PDB) for a seamless rollout.

Prerequisites

Available for ACS clusters running Kubernetes version 1.28 or later.

Install the component

In the ACS console, click the name of your cluster. In the left navigation pane, choose Applications > Helm.
On the Helm page, click Deploy.
- Basic information: In the Chart search box, enter acs-instance-helper and select it from the results.
- Parameters: For Chart Version, select the latest version.

Configure global settings

Console

In the navigation pane on the left, choose Configurations > ConfigMaps.
On the ConfigMaps page, click Create from YAML. Copy the following manifest into the Template area and click Create.

kubectl

Get the cluster KubeConfig and connect to the cluster by using kubectl.
Save the following YAML content as the acs-instance-helper-global-configmap.yaml file, and then run the kubectl apply -f acs-instance-helper-global-configmap.yaml command.

kind: ConfigMap
apiVersion: v1
metadata:
  namespace: kube-system
  name: acs-instance-helper-global-config
data:
  insufficientDiskEvictionSeconds: "300"
  customOnlineWorkloads: foo.io/SomeWorkload,bar.io/AnotherWorkload

insufficientDiskEvictionSeconds: The grace period in seconds from when the component first detects disk pressure on a Pod until it begins graceful termination. The default value is "300" (5 minutes).
customOnlineWorkloads: The component supports Deployment workloads by default. You can configure this parameter to support other custom workload types by providing them as a comma-separated list.
Important
- Ensure that the custom workload controller can maintain the desired number of replicas through scaling, for example, by automatically creating new replicas when existing ones are terminated.
- A zero-downtime automated rollout cannot be guaranteed for all custom workload types when the disk pressure threshold is reached. Test thoroughly before enabling this feature for a custom workload in a production environment.

Create and configure a workload

To enable acs-instance-helper to manage applications, add the following annotation to the workload: ops.alibabacloud.com/enable-insufficient-disk-helper: "true".
Create a PDB and specify that at least 100% of the replicas are available at all times. As shown in the following example, the rolling update process ensures that at least two replicas are available.

Console

In the navigation pane on the left, choose Workloads > Deployments.

On the Deployments page, click Create from YAML. Copy the following manifest into the Template area and click Create.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: app
  template:
    metadata:
      labels:
        app: app
      annotations:
        # Key annotation: Enables the feature for this workload.
        ops.alibabacloud.com/enable-insufficient-disk-helper: "true"
    spec:
      containers:
        - image: registry-cn-hangzhou.ack.aliyuncs.com/dev/hello-world:v1
          name: main-container
          resources:
            limits:
              cpu: "4"
              memory: 8Gi
      restartPolicy: Always
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: app-pdb
spec:
  minAvailable: 100%
  selector:
    matchLabels:
      app: app

In the pop-up window, find the target stateless workload, click View , and verify that the Pod status is Running.

kubectl

Save the following YAML manifest as an app.yaml file, and then run the kubectl apply -f app.yaml command.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: app
  template:
    metadata:
      labels:
        app: app
      annotations:
        # Key annotation: Enables the feature for this workload.
        ops.alibabacloud.com/enable-insufficient-disk-helper: "true"
    spec:
      containers:
        - image: registry-cn-hangzhou.ack.aliyuncs.com/dev/hello-world:v1
          name: main-container
          resources:
            limits:
              cpu: "4"
              memory: 8Gi
      restartPolicy: Always
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: app-pdb
spec:
  minAvailable: 100%
  selector:
    matchLabels:
      app: app

Confirm that the status of the target application Pod is Running.
```
kubectl get pods -l app=app
```

Verify the automated rollout

Replace app-xxx and app-yyy with the actual Pod names and write data to the two Pods in separate terminals to simulate multiple replicas experiencing disk pressure simultaneously.

# Terminal 1
kubectl exec app-xxx -- sh -c 'dd if=/dev/zero of=/largefile bs=1G count=30'
# Terminal 2
kubectl exec app-yyy -- sh -c 'dd if=/dev/zero of=/largefile bs=1G count=30'

Observe for several minutes. You will see that when the disk pressure threshold is reached, the two Pods are sequentially evicted and replaced. During this process, two Pods always remain in the Running state:

$ kubectl get po
NAME         READY   STATUS        RESTARTS   AGE
app-xxx      1/1     Terminating   0          12m
app-yyy      1/1     Running       0          12m
app-xxx-new  1/1     Running       0          11s

After app-xxx is evicted and replaced, app-yyy is also evicted and replaced, and two new Pods are eventually created.

$ kubectl get po
NAME          READY   STATUS        RESTARTS   AGE
app-yyy       1/1     Terminating   0          13m
app-xxx-new   1/1     Running       0          1m
app-yyy-new   1/1     Running       0          11s

Production considerations

Proactive disk management: Treat this automated rollout feature as a last resort, not a primary strategy. Proactively manage disk usage by optimizing your application. For example, configure proper ACS container log rotation or increase the ephemeral storage size.
Monitoring and alerts: Configure alerts for the Pod events NewInstanceCreationTriggered and InstanceEvictedGracefully to stay informed about automatic recovery operations triggered by disk pressure. For more information, see Monitor the status of an ACS cluster by using Alibaba Cloud Prometheus.

Billing

Installing the acs-instance-helper component deploys a Deployment with two replicas in your cluster. Each replica is allocated 1 vCPU and 2 GiB of memory. These resources are consumed from your cluster and will incur fees. For details, see ACS computing power billing.

FAQ

Difference from Kubelet native eviction

The Kubelet's native Pod eviction is a reactive mechanism that triggers only after the disk is full. This process is not protected by safeguards such as PodDisruptionBudgets. Additionally, because the disk is already full, the Pod's graceful termination logic may fail to execute properly. This feature avoids these issues by proactively initiating an eviction before the disk is full.