Troubleshoot service issues

Uneven load distribution across CLB backends

Cause: CLB scheduling algorithm is not suited to the traffic pattern.

Symptom: Uneven request distribution across backend servers.

Solution:

• For services with externalTrafficPolicy: Local, add the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler:"wrr" annotation to use weighted round-robin scheduling.

• For services using persistent connections, add the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler:"wlc" annotation for weighted least connections scheduling. Prevents one long-lived connection from monopolizing traffic.

To capture container network packets for load distribution analysis, see this Alibaba Cloud Developer community article.

503 error during application updates

Cause: Connection draining or pod graceful termination is not configured. During rolling updates, CLB may route traffic to terminating pods.

Symptom: 503 error when accessing the CLB during an application update.

Solution:

1. Add the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain annotation to enable connection draining. See Common operations to manage listeners.

2. Configure readinessProbe and preStop on the pod:

   • readinessProbe : Pods join CLB backends only after passing the probe. Set probe frequency, delay, and failure threshold to match your application's startup time. Too-short timeouts cause repeated pod restarts.

   • preStop and terminationGracePeriodSeconds : Set preStop to the time your application needs to drain in-flight requests. Set terminationGracePeriodSeconds to at least 30 seconds longer than preStop.

   apiVersion: v1
   kind: Pod
   metadata:
     name: nginx
     namespace: default
   spec:
     containers:
     - name: nginx
       image: nginx
       # Liveness probe
       livenessProbe:
         failureThreshold: 3
         initialDelaySeconds: 30
         periodSeconds: 30
         successThreshold: 1
         tcpSocket:
           port: 5084
         timeoutSeconds: 1
       # Readiness probe
       readinessProbe:
         failureThreshold: 3
         initialDelaySeconds: 30
         periodSeconds: 30
         successThreshold: 1
         tcpSocket:
           port: 5084
         timeoutSeconds: 1
       # Graceful termination
       lifecycle:
         preStop:
           exec:
             command:
             - sleep
             - 30
     terminationGracePeriodSeconds: 60

CLB inaccessible from within the cluster

Cause: externalTrafficPolicy: Local is set on the service. kube-proxy only forwards traffic to pods on the same node as the request origin. If the node has no backend pod for the service, the connection fails. This affects in-cluster traffic routed to the CLB address. See kube-proxy adds external-lb address to node-local iptables rule.

Symptom: CLB is accessible from outside the cluster but connections fail from within.

Solution: Use one of the following approaches:

• Access via ClusterIP or service name (recommended for in-cluster access): Use the service's ClusterIP or DNS name instead of the CLB address. For Ingress, the service name is nginx-ingress-lb.kube-system.

• Switch to `externalTrafficPolicy: Cluster`: In-cluster traffic reaches the service regardless of pod placement, but client source IP is not preserved. To modify the Ingress service:

  With an Ingress CLB, pods can only access Ingress/CLB-exposed services from the node running the Ingress pod.

  kubectl edit svc nginx-ingress-lb -n kube-system

• Use `externalTrafficPolicy: Cluster` with ENI pass-through (Terway only): If your cluster uses Terway with ENIs or multiple IPs per ENI, set externalTrafficPolicy: Cluster and add the service.beta.kubernetes.io/backend-type: "eni" annotation. This preserves source IP and enables in-cluster access. See Use annotations to configure a Classic Load Balancer (CLB) instance.

  apiVersion: v1
  kind: Service
  metadata:
    annotations:
      service.beta.kubernetes.io/backend-type: eni
    labels:
      app: nginx-ingress-lb
    name: nginx-ingress-lb
    namespace: kube-system
  spec:
    externalTrafficPolicy: Cluster

CLB inaccessible from outside the cluster

Cause: An ACL blocks the client IP, the CLB vServer group has no backends, or the health check is failing.

Symptom: The CLB instance cannot be reached from outside the cluster.

Solution:

1. Check for service error events and resolve them. See Service error events and solutions.

   kubectl -n {your-namespace} describe svc {your-svc-name}

2. Check whether an ACL is configured on the CLB instance. If so, verify it allows inbound traffic from the client IP. See Resource Access Management.

3. Check whether the CLB vServer group is empty. If empty, verify a pod is associated with the service and running. If unhealthy, resolve the pod issue first. See Troubleshoot pod issues.

4. Check whether the CLB listener health check passes. If failing, verify the pod responds correctly. See CLB health check FAQ.

Cannot connect to the backend HTTPS service

Cause: With a certificate on the CLB listener, CLB terminates TLS and forwards HTTP to backends. If targetPort points to an HTTPS port (e.g., 443), the pod rejects the plaintext request with "The plain HTTP request was sent to HTTPS port."

Symptom: Backend connections fail after configuring HTTPS on the CLB listener.

Solution: Set targetPort to the pod's HTTP port. For example, if Nginx serves HTTPS on 443, set targetPort to 80.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${YOUR_CERT_ID}"
  name: nginx
  namespace: default
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer