Usage notes and risky operations

Category

Risky operation

Impact

Recovery solution

API Server

You can reuse the CLB instance of the API server for other scenarios, such as a Service of type LoadBalancer.

The cluster becomes unavailable and service traffic is affected.

Revert to the original configuration or contact after-sales support.

Modifying the listener, vServer group, access control list (ACL), or other configurations that control SLB forwarding, or modifying the tag configuration of the SLB instance used by the API Server.

The cluster becomes abnormal.

Revert to the original configuration.

Delete the CLB instance that the API server uses.

Operations on the cluster fail.

This operation cannot be reversed. You must re-create the cluster. For more information, see Create an ACK managed cluster.

Worker node

Modifying the security group of a node in the cluster.

The node may become unavailable.

Add the node back to the security group that was automatically created for the cluster. For more information, see Associate a security group with an instance (primary ENI).

The node expires or is destroyed.

The node becomes unavailable.

This operation cannot be reversed.

Reinstalling the operating system.

Components on the node are deleted.

Remove the node from the cluster and then add it back. For more information, see Remove a node and Add existing nodes to a cluster.

Upgrading node component versions on your own.

The node may become unusable.

Roll back to the original version.

Changing the node IP address.

The node becomes unavailable.

Revert the IP address to the original one.

Modifying parameters of core components, such as kubelet, docker, and containerd, on your own.

The node may become unavailable.

Configure the parameters as recommended in the official documentation.

Modifying the operating system configuration.

The node may become unavailable.

Try to revert the configuration item. Otherwise, delete the node and purchase a new one.

Modifying the node time.

Components on the node may work abnormally.

Revert the node time.

Adding computing power to the cluster in a way that is not supported by ACK.

ACK provides methods such as the console, OpenAPI, and command-line interface (CLI) to add computing power to a cluster. For more information, see Add existing nodes to a cluster. If you add nodes to the cluster using other methods, ACK cannot identify the source of these nodes and cannot provide product capabilities such as node lifecycle management, automated O&M, and technical support. For a detailed risk description, see Why does the console show that the source of the node pool to which a node belongs is "Other Nodes"?.

We recommend that you manage computing power using node pools. If you want to continue using the current method, ensure the compatibility of the nodes with various cluster components, such as Kubernetes, network, storage, and security components.

Master node (ACK dedicated cluster)

Modifying the security group of a node in the cluster.

The master node may become unavailable.

Add the node back to the security group that was automatically created for the cluster. For more information, see Associate a security group with an instance (primary ENI).

The node expires or is destroyed.

The master node becomes unavailable.

This operation cannot be reversed.

Reinstalling the operating system.

Components on the master node are deleted.

This operation cannot be reversed.

Upgrading the Master or etcd component versions on your own.

The cluster may become unusable.

Roll back to the original version.

Deleting or formatting data in core directories such as /etc/kubernetes on the node.

The master node becomes unavailable.

This operation cannot be reversed.

Changing the node IP address.

The master node becomes unavailable.

Revert the IP address to the original one.

Modifying parameters of core components, such as etcd, kube-apiserver, and docker, on your own.

The master node may become unavailable.

Configure the parameters as recommended in the official documentation.

Replacing the Master or etcd certificate on your own.

The cluster may become unusable.

This operation cannot be reversed.

Adding or removing master nodes on your own.

The cluster may become unusable.

This operation cannot be reversed.

Modifying the node time.

Components on the node may work abnormally.

Revert the node time.

Other

Changing permissions or performing modifications through RAM.

Some cluster resources, such as SLB instances, may fail to be created.

Revert to the original permissions.

Modifying or deleting pre-configured PodSecurityPolicy-related resources in the cluster. This includes the PodSecurityPolicy resource named ack.privileged, and ClusterRole, ClusterRoleBinding, Role, and RoleBinding resources with names that start with ack:podsecuritypolicy:.

Core cluster components may become abnormal. You may not be able to create or update pod resources in the cluster.

Recover the related resources. For more information, see Configure or recover the default pod security policies of ACK.

Risky operation

Impact

Recovery solution

Deleting a scaling group.

The node pool becomes abnormal.

This operation cannot be reversed. You can only re-create the node pool. For more information, see Create a node pool.

Removing a node using kubectl.

The displayed number of nodes in the node pool does not match the actual number.

Remove the specified node from the ACK console or using the node pool-related API (see Remove a node), or scale in by modifying the expected number of nodes in the node pool (see Create and manage node pools).

Directly releasing an ECS instance.

The node pool product page may display information abnormally. For a node pool with a specific number of expected nodes, the pool automatically scales out to the expected number of nodes based on the node pool configuration to maintain that number.

This operation cannot be reversed. The correct procedure is to scale in by modifying the expected number of nodes in the node pool from the ACK console or using the node pool-related API (see Create and manage node pools) or to remove a specified node (see Remove a node).

Manually scaling out or scaling in a node pool with auto scaling enabled.

The auto scaling component automatically adjusts the number of nodes based on the policy. This causes the result to not match your expectations.

This operation cannot be reversed. Auto-scaling node pools do not require manual intervention.

Modifying the maximum or minimum number of instances in an ESS scaling group.

Scaling may become abnormal.

• For a node pool without auto scaling enabled, change the maximum and minimum number of instances in the ESS scaling group to the default values of 2000 and 0.

• For a node pool with auto scaling enabled, change the maximum and minimum number of instances in the ESS scaling group to be consistent with the maximum and minimum number of nodes in the node pool.

Not backing up data before adding an existing node.

Any data on the instance will be lost.

This operation cannot be reversed.

• Before you manually add an existing node, you must back up all data that you want to keep.

• When a node is added automatically, the system disk is replaced. You need to back up useful data that is stored on the system disk in advance.

Saving important data on the node's system disk.

The self-healing operation of a node pool may repair a node by resetting its configuration. This can lead to data loss on the system disk.

This operation cannot be reversed. The correct practice is to store important data on an extra data disk or on a cloud disk, NAS, or OSS.

Risky operation

Impact

Recovery solution

Uninstalling the virtual node component.

The serverless pod management feature becomes abnormal: created ECI pods and ACS pods cannot be deleted normally, and new ECI pods and ACS pods cannot be created normally.

Re-install the virtual node component.

Risky operation

Impact

Recovery solution

Modifying the kernel parameter net.ipv4.ip_forward=0.

Network connectivity fails.

Modify the kernel parameter to net.ipv4.ip_forward=1.

Modifying the kernel parameters:

• net.ipv4.conf.all.rp_filter = 1|2

• net.ipv4.conf.[ethX].rp_filter = 1|2

  Note

  ethX represents all network interface controllers (NICs) that start with eth.

Network connectivity fails.

Modify the kernel parameters to:

• net.ipv4.conf.all.rp_filter = 0

• net.ipv4.conf.[ethX].rp_filter = 0

Modifying the kernel parameter net.ipv4.tcp_tw_reuse = 1.

Pod health checks become abnormal.

Modify the kernel parameter to net.ipv4.tcp_tw_reuse = 0.

Modifying the kernel parameter net.ipv4.tcp_tw_recycle = 1.

NAT becomes abnormal.

Modify the kernel parameter to net.ipv4.tcp_tw_recycle = 0.

Modifying the kernel parameter net.ipv4.ip_local_port_range.

Intermittent network connectivity failures occur.

Modify the kernel parameter to the default value net.ipv4.ip_local_port_range="32768 60999".

Installing firewall software, such as Firewalld or ufw.

Container network connectivity fails.

Uninstall the firewall software and restart the node.

The node security group configuration does not allow UDP traffic on port 53 for the container CIDR block.

DNS in the cluster does not work correctly.

Configure the security group to allow traffic as recommended in the official documentation.

Modifying or deleting the tags of an SLB instance added by ACK.

The SLB instance becomes abnormal.

Revert the tags of the SLB instance.

Modifying the configuration of an SLB instance managed by ACK from the SLB console, including the SLB instance, listener, and vServer group.

The SLB instance becomes abnormal.

Revert the configuration of the SLB instance.

Removing the annotation for reusing an existing SLB instance from a Service, which is service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: ${YOUR_LB_ID}.

The SLB instance becomes abnormal.

Add the annotation for reusing an existing SLB instance to the Service.

Deleting an SLB instance created by ACK from the SLB console.

The cluster network may become abnormal.

Delete the SLB instance by deleting the Service. For more information, see Delete a Service.

Manually deleting the nginx-ingress-lb Service in the kube-system namespace when the Nginx Ingress Controller component is installed.

The Ingress Controller does not work correctly and may crash in severe cases.

Create a new Service with the same name using the following YAML.

apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    app: nginx-ingress-lb
  name: nginx-ingress-lb
  namespace: kube-system
spec:
  externalTrafficPolicy: Local
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: ingress-nginx
  type: LoadBalancer

Adding or modifying the nameserver option in the DNS configuration file /etc/resolv.conf on an ECS node.

If the configured DNS server is not configured properly, DNS resolution may fail. This affects the normal operation of the cluster.

If you want to use a self-managed DNS server as an upstream server, we recommend that you configure it on the CoreDNS side. For more information, see Configure an unmanaged CoreDNS component.

Modifying or deleting elastic network interfaces (ENIs) or Lingjun ENIs created by ACK.

The pod network is interrupted.

This operation cannot be reversed.

Modifying or deleting network-related CRDs.

podnetworkings.network.alibabacloud.com
podenis.network.alibabacloud.com
networkinterfaces.network.alibabacloud.com
nodes.network.alibabacloud.com
noderuntimes.network.alibabacloud.com
*.cilium.io
*.crd.projectcalico.org

The Terway component will not work. This may lead to network interruptions and pod abnormalities in severe cases.

This operation cannot be reversed.

Creating, modifying, or deleting network-related system CRs.

podenis.network.alibabacloud.com
networkinterfaces.network.alibabacloud.com
nodes.network.alibabacloud.com
noderuntimes.network.alibabacloud.com
*.cilium.io
*.crd.projectcalico.org

The Terway component will not work. This may lead to network interruptions and pod abnormalities in severe cases.

Delete the custom CR and re-create the associated pod.

Modifying or deleting fields that are not allowed to be modified in the Terway network configuration. For parameter declarations, see Customize Terway configurations.

The Terway component will not work. This may lead to network interruptions and pod abnormalities in severe cases.

Revert to the original configuration and restart the node.

Risky operation

Impact

Recovery solution

Manually detaching a cloud disk from the console.

Pod write operations report an I/O error.

Restart the pod and manually clean up mount point residuals on the node.

Running umount on the disk mount path on the node.

The pod writes to the local disk.

Restart the pod.

Directly operating on a cloud disk on the node.

The pod writes to the local disk.

This operation cannot be reversed.

Mounting the same cloud disk to multiple pods.

The pod writes to the local disk or reports an I/O error.

Ensure that one cloud disk is used by only one pod.

Manually deleting a NAS mount directory.

Pod write operations report an I/O error.

Restart the pod.

Deleting a NAS disk or mount target that is in use.

The pod experiences an I/O hang.

Restart the ECS node. For more information, see Restart an ECS instance.

Risky operation

Impact

Recovery solution

Deleting the /tmp/ccs-log-collector/pos directory on the host.

Duplicate log collection.

This operation cannot be reversed. The files in this directory record the log collection positions.

Deleting the /tmp/ccs-log-collector/buffer directory on the host.

Log data loss.

This operation cannot be reversed. This directory is a cache file for logs that are waiting to be consumed.

Deleting the aliyunlogconfig CRD resource.

Log collection fails.

Re-create the deleted CRD and its corresponding resources. However, logs from the failure period cannot be recovered.

Deleting a CRD also deletes all its corresponding instances. Even if the CRD is recovered, you must manually create the deleted instances.

Deleting the log component.

Log collection fails.

Reinstall the log component and manually recover the aliyunlogconfig CRD instances. Logs from the deletion period cannot be recovered.

Deleting the log component is equivalent to deleting the aliyunlogconfig CRD and the Logtail collector. All log collection capabilities are lost during this period.