ACS supports the scheduled inspection feature provided by CIS. You can set scheduled rules to periodically inspect your clusters for risk alerts. This topic describes common risk alerts identified during cluster inspections and their solutions.
Cluster inspection items and alerts
To learn how to use the cluster inspection feature, see Use cluster diagnostics.
Specific inspection items may vary by cluster configuration. The inspection report provides the definitive results.
Check Type | Inspection Item | Alert |
Resource Quotas ResourceQuotas | ||
SLB backend server quota check | ||
SLB listener quota check | ||
Resource Watermark ResourceLevel | SLB bandwidth usage check | |
SLB maximum connections check | ||
Versions & Certificates Versions&Certificates | Cluster Kubernetes version check | |
Cluster Risk ClusterRisk | API server SLB instance exists | |
API server SLB instance status | ||
API server SLB port 6443 listener configuration | ||
API server SLB access control configuration | ||
DNS service ClusterIP | ||
DNS service backend endpoints | ||
Multiple Services sharing an SLB port |
Insufficient SLB quota in VPC
Impact: The remaining quota for SLB instances in the VPC is less than five. Each LoadBalancer Service creates an SLB instance. If the quota is exhausted, you cannot create new LoadBalancer Services.
Solution: By default, an account can have up to 60 SLB instances. If you need a higher quota, log on to the Quota Center console and submit an application to increase the quota. For more information, see Quotas.
Insufficient SLB backend server quota
Impact: An SLB instance supports a limited number of ECS instances as backend servers. For a large-scale LoadBalancer Service, its pods may be distributed across multiple ECS instances. If the number of ECS instances exceeds this quota, the SLB instance cannot add the excess instances.
Solution: By default, an SLB instance can have up to 200 backend servers. If you need a higher quota, log on to the Quota Center console and submit an application to increase the quota. For more information, see Quotas.
Insufficient SLB listener quota
Impact: An SLB instance supports a limited number of listeners. For a LoadBalancer Service, each exposed port corresponds to one SLB listener. When the number of ports reaches this limit, additional ports cannot serve traffic.
Solution: By default, you can add up to 50 listeners to an SLB instance. If you need a higher quota, log on to the Quota Center console and submit an application to increase the quota. For more information, see Quotas.
SLB instance quota check
Impact: This check verifies if the remaining quota for SLB instances is less than five. Each LoadBalancer Service creates an SLB instance. If the quota is exhausted, you cannot create new LoadBalancer Services.
Solution: By default, an account can have up to 60 SLB instances. If you need a higher quota, submit a request in Quota Center to increase the quota.
High SLB bandwidth usage
Impact: The peak outbound bandwidth over the last three days has exceeded 80% of the limit. When the bandwidth limit is reached, the SLB instance may drop packets, causing request jitter and high latency.
Solution: If the network bandwidth usage of your SLB instance is too high, you must upgrade the instance specification. For more information, see Expose an application by using an existing SLB instance.
High SLB maximum connections
Impact: The maximum number of connections over the last three days has reached 80% of the limit. When the connection limit is reached, new connections cannot be established, which prevents clients from accessing the service.
Solution: If the number of connections has reached 80% of the maximum over the last three days, upgrade the instance specification to prevent service disruptions. Exceeding the maximum connections limit prevents clients from accessing the service. For more information, see Expose an application by using an existing SLB instance.
High SLB new connection rate
Impact: The maximum new connection rate over the last three days has reached 80% of the limit. When this limit is reached, new connections cannot be established for a short period, preventing clients from accessing the service.
Solution: If the new connection rate for your SLB instance has reached 80% of the maximum new connections per second over the last three days, you must upgrade the instance specification to prevent service disruptions. Reaching the limit for new connections per second prevents clients from accessing the service. For more information, see Expose an application by using an existing SLB instance.
High SLB QPS
Impact: The maximum QPS over the last three days has reached 80% of the limit. When the QPS limit is reached, clients may be unable to access the service.
Solution: If the QPS of your SLB instance has reached 80% of the maximum QPS over the last three days, you must upgrade the instance specification to prevent service disruptions. Exceeding the maximum QPS limit prevents clients from accessing the service. For more information, see Expose an application by using an existing SLB instance.
SLB new connection rate check
Impact: This check verifies if the maximum new connection rate for the SLB instance over the last three days has reached 80% of the limit. When this limit is reached, new connections cannot be established for a short period, and clients may be unable to access the service.
Solution: If the new connection rate of your SLB instance is too high, you must upgrade the instance specification to avoid service disruptions. For more information, see Expose an application by using an existing SLB instance.
API server SLB instance exists
Impact: This check verifies if an SLB instance exists for the cluster API server. Without a load balancer, the API server runs as a single instance, creating a single point of failure (SPOF). If this instance fails, the entire cluster becomes inoperable.
Solution: Configure a load balancer.
API server SLB instance status
Impact: All cluster operations, such as pod scheduling, Service deployment, and scaling, are interrupted or delayed. The service discovery mechanism within the cluster depends on the API server. If the SLB instance is in an abnormal state, service discovery may fail.
Solution: Verify that the SLB instance configuration is correct, including the backend server configuration, listener ports, and health check mechanism.
API server SLB port 6443 listener configuration
Impact: This check verifies the listener configuration on port 6443 of the SLB instance for the cluster API server. If the configuration is incorrect, all requests to the API server through the SLB instance fail, including kubectl operations, dashboard access, and API requests from other services. Services within the cluster may also fail to resolve other services by name, as this process also depends on the API server.
Solution: Verify that the SLB instance configuration is correct, including the backend server configuration, listener ports, and health check mechanism. Ensure that a listener for port 6443 is configured and uses the HTTPS protocol.
API server SLB access control configuration
Impact: This check verifies the access control configuration of the SLB instance for the cluster API server. If the configuration is incorrect, cluster management and operations, such as node management, pod scheduling, and Service deployment, may be interrupted or restricted. Because workloads rely on the API server for communication and service discovery, incorrect access control settings can cause these requests to fail.
Solution: Check the access control configurations of the SLB instance, such as security groups and ACLs. Ensure that legitimate IP addresses and ports, especially port 6443, are allowed to access the API server. Verify that the TLS/SSL configuration between the SLB and the API server is correct and that the certificates are valid and not expired.
DNS service ClusterIP
Impact: This check verifies if a ClusterIP is correctly assigned to the cluster DNS service. An abnormal DNS service can cause cluster functions to fail and affect workloads.
Solution: Check the Kubernetes cluster's network plugins and configurations for conflicts or errors. Redeploy the DNS service, such as CoreDNS, to ensure it is correctly configured and assigned.
DNS service backend endpoints
Impact: This check verifies the number of backend DNS servers associated with the cluster DNS service. If the number is 0, the DNS service is unavailable.
Solution: Check whether the CoreDNS configuration file (Corefile) is correctly configured. Make sure that the forward or proxy directive points to a valid set of backend DNS servers.
SLB QPS check
Impact: This check verifies if the maximum QPS for the SLB instance over the last three days has reached 80% of the limit. When the QPS limit is reached, clients may be unable to access the service.
Solution: If the QPS of your SLB instance is too high, you must upgrade the instance specification to prevent service disruptions. For more information, see Expose an application by using an existing SLB instance.
Outdated cluster version
Impact: The Kubernetes version of your cluster is approaching or has reached its end of support.
Solution: Upgrade the cluster as soon as possible if its version is approaching or has reached the end of support.
Multiple Services sharing an SLB port
Impact: If multiple Services use the same port on a single SLB instance, they can become unavailable.
Solution: Modify or delete the conflicting Services to ensure that each Service uses a different port when reusing the same SLB instance.