Service FAQ

Default CLB instances in an ACK cluster

If you install the Nginx Ingress add-on when you create a Kubernetes cluster, two Classic Load Balancer (CLB) instances are created by default.

The two CLB instances serve the following purposes:

• API Server CLB: This CLB instance serves as the access endpoint for the API server. All requests to the cluster are routed through this CLB instance. It listens on TCP port 6443. The backend servers are API server pods or master ECS instances.

• Nginx Ingress Controller CLB: This CLB instance is associated with the nginx-ingress-controller Service in the kube-system namespace. The vServer group dynamically binds to Nginx Ingress Controller pods to load balance and route external requests. It listens on TCP ports 80 and 443.

For more information, see The specific purposes of CLB instances in an ACK cluster.

Choosing between Local and Cluster policies

The Local and Cluster external traffic policies behave differently depending on the network plugin. For more information about the differences between the two policies, see External traffic policies.

No events for Service-LoadBalancer sync

If no Events are displayed after you run the kubectl -n {your-namespace} describe svc {your-svc-name} command, check your CCM component version.

• Versions earlier than v1.9.3.276-g372aa98-aliyun: Earlier versions of the CCM component do not display Events for the synchronization between Services and LoadBalancers. You must upgrade the component. For information about how to view and upgrade the CCM version, see Upgrade the CCM component.

• Versions v1.9.3.276-g372aa98-aliyun and later: Submit a ticket.

CLB instance stuck in Pending state

1. Run the kubectl -n {your-namespace} describe svc {your-svc-name} command to view Events.

2. Resolve any errors reported in the Events. For information about how to handle different error messages, see Error Events and solutions.

   If no error messages are displayed, see Why are no Events displayed for the synchronization between Services and LoadBalancers?.

CLB vServer group not updated

1. Run the kubectl -n {your-namespace} describe svc {your-svc-name} command to view Events.

2. Resolve any errors reported in the Events. For information about how to handle different error messages, see Error Events and solutions.

   If no error messages are displayed, see Why are no Events displayed for the synchronization between Services and LoadBalancers?.

Service annotation not taking effect

1. Check for error messages.

   1. Run the kubectl -n {your-namespace} describe svc {your-svc-name} command to view Events.

   2. Resolve any errors reported in the Events. For information about how to handle different error messages, see Error Events and solutions.

2. If no error messages are reported, resolve the issue based on one of the following scenarios:

   • Verify that your CCM version meets the requirements for the annotation. For more information about the version mappings between annotations and CCM, see Use annotations to configure a Classic Load Balancer (CLB).

   • Log on to the Container Service for Kubernetes console. On the Services page, click the name of the target Service and check whether the Service contains the required annotation. If not, add the annotation.

     For more information about how to add annotations, see Use annotations to configure a Classic Load Balancer (CLB).

     For more information about how to view the list of Services, see Manage Services.

   • Verify that the annotation is configured correctly.

Unintended CLB configuration changes

The CCM component uses a declarative API and automatically updates the CLB configuration based on the Service configuration under specific conditions. Any changes that you make in the CLB console are at risk of being overwritten. We recommend that you configure the CLB instance by using annotations. For more information, see Use annotations to configure a Classic Load Balancer (CLB).

CLB IP inaccessible from within cluster

• Scenario 1: You use a private IP address for a CLB instance that was not created by a Service. A client pod and a backend pod might be scheduled on the same node, which causes access from within the cluster to fail.

  For Layer 4 load balancing, a backend ECS instance cannot act as both a client and a backend server. You can use one of the following methods to resolve this issue:

  • Change the CLB IP address to a public IP address.

  • Create the CLB instance by using a Service and set the external traffic policy to Cluster. In this case, kube-proxy intercepts traffic that is sent to the CLB instance from within the cluster and bypasses the CLB instance.

• Scenario 2: You set the external traffic policy to Local when you expose a Service. This causes access to the CLB IP address from within the cluster to fail.

  For more information about the cause and solution, see How do I resolve connectivity issues to a CLB address exposed by a LoadBalancer Service in a Kubernetes cluster?.

Connectivity issues to a LoadBalancer Service

When are CLB instances automatically deleted?

Service deletion and CLB instances

If you reuse an existing CLB instance by specifying the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: {your-slb-id} annotation in the Service, deleting the Service does not delete the CLB instance. Otherwise, the CLB instance is deleted when you delete the Service.

If you change the Service type (for example, from LoadBalancer to NodePort), the CLB instance that was automatically created by CCM is also deleted.

Accidental CLB instance deletion

• Scenario 1: You accidentally delete the API Server CLB instance

  The instance cannot be recovered. You must recreate the cluster. For more information, see Create an ACK Pro cluster.

• Scenario 2: You accidentally delete the Ingress CLB instance

  The following steps describe how to recreate the CLB instance by using an Nginx Ingress as an example.

  1. Log on to the ACK console. In the left navigation pane, click Clusters.

  2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Network > Services.

  3. On the Service page, select kube-system from the Namespace drop-down list.

     1. Find the nginx-ingress-lb Service, click Edit YAML in the Actions column, remove the status.loadBalancer field and its sub-fields, and then click OK, which prompts CCM to recreate the CLB instance.

     2. If the nginx-ingress-lb Service is not in the list, click Create from YAML in the upper-left corner and use the following template to create a Service named nginx-ingress-lb.

        apiVersion: v1
        kind: Service
        metadata:
          labels:
            app: nginx-ingress-lb
          name: nginx-ingress-lb
          namespace: kube-system
        spec:
          externalTrafficPolicy: Local
          ports:
          - name: http
            port: 80
            protocol: TCP
            targetPort: 80
          - name: https
            port: 443
            protocol: TCP
            targetPort: 443
          selector:
            app: ingress-nginx
          type: LoadBalancer

• Scenario 3: You accidentally delete a business-related CLB instance

  • If you no longer need the Service that corresponds to the CLB instance, delete the Service.

  • If the Service is still in use, perform the following steps:

    1. Log on to the ACK console. In the left navigation pane, click Clusters.

    2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Network > Services.

    3. On the Service page, select All Namespaces from the Namespace drop-down list and find your business Service.

    4. In the Actions column for the target Service, click Edit YAML, remove the status field, and then click Update, which prompts CCM to recreate the CLB instance.

Enable CLB renaming for old CCM versions

1. Log on to a master node of the Kubernetes cluster. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

2. Run the kubectl get svc -n ${namespace} ${service} command to view the Service type and IP address.

   Note

   Replace namespace and service with the actual namespace and Service name.

3. Run the following command to generate the required tag for the CLB instance.

   kubectl get svc -n ${namespace} ${service} -o jsonpath="{.metadata.uid}"|awk -F "-" '{print "kubernetes.do.not.delete: "substr("a"$1$2$3$4$5,1,32)}'

   

4. Log on to the CLB console. Search for the CLB instance in the corresponding region based on the IP address that you obtained in Step 2.

5. Add a tag to the CLB instance by using the key and value generated in Step 3. The key and value correspond to 1 and 2 in the preceding figure. For more information, see Create and manage CLB instances.

Automatic node weight calculation in Local mode

This section uses an example where an application pod (app=nginx) is deployed on three ECS instances and exposed through a Service to explain how node weights are calculated in Local mode.

Querying CLB instance details

1. Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

2. Run the following command to obtain the name, IP address, and address type of each LoadBalancer Service in all namespaces.

   kubectl get services -A -ojson | jq '.items[] | select(.spec.type == "LoadBalancer") | {name: .metadata.name, namespace: .metadata.namespace, ip: .status.loadBalancer.ingress[0].ip, lb_type: .metadata.annotations."service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type"}'

   Sample output:

   {
     "name": "test",
     "namespace": "default",
     "ip": "192.168.*.*",
     "lb_type": "intranet"
   }
   {
     "name": "nginx-ingress-lb",
     "namespace": "kube-system",
     "ip": "47.97.*.*",
     "lb_type": "null"
   }

Enable graceful connection termination

You can configure connection draining by using the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain and service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain-timeout annotations. After a backend server is removed from the Service, the CLB instance continues to process existing connections for the specified drain timeout period. For more information, see Enable connection draining for a listener.

Why does reusing a CLB instance fail?

• Check the CCM version. Versions earlier than v1.9.3.105-gfd4e547-aliyun do not support reusing CLB instances. To upgrade, see Upgrade the CCM component.

• You cannot reuse a CLB instance created by the cluster.

• You cannot reuse the CLB instance associated with the API server.

• If you reuse a private-facing CLB instance, it must be in the same VPC as the cluster. Cross-VPC reuse is not supported.

Missing listeners when reusing a CLB

What if a CCM upgrade fails?

For troubleshooting steps, see Cloud Controller Manager (CCM) component upgrade check fails.

Service error messages and solutions

Configuring a listener for a NodePort Service

CCM supports listener configuration only for LoadBalancer Services. You must change the Service type from NodePort to LoadBalancer.

How do I access a NodePort Service?

• If you access a Service from within the cluster (on a cluster node), you can use ClusterIP+Port or Node IP+the Service's NodePort. The default NodePort is greater than 30000.

• If you access the Service from outside the cluster, you can use the IP address of a node and the NodePort of the Service. The default NodePort is greater than 30000.

• From outside the VPC (from another VPC or the internet): You must change the Service type to LoadBalancer and then access it through the Service's external endpoint.

  Note

  If your Service has its external traffic policy set to Local, ensure that the node you are accessing hosts one of the Service's backend pods. For more information, see External traffic policies.

How do I configure the NodePort range?

In Kubernetes, the API server provides the ServiceNodePortRange parameter (the --service-node-port-range command-line argument), which restricts the range of NodePorts that can be used by NodePort or LoadBalancer Services. The default range is 30000–32767. In an ACK Pro cluster, you can modify this range by customizing the control plane parameters. For more information, see Customize control plane parameters for an ACK Pro cluster.

• Exercise caution when you modify the NodePort range. Ensure that the NodePort range does not conflict with the range specified by the net.ipv4.ip_local_port_range Linux kernel parameter on your cluster nodes. The ip_local_port_range parameter controls the range of ephemeral ports that applications on the system can use. The default value for ip_local_port_range is 32768–60999.

• In a default ACK cluster configuration, the ServiceNodePortRange and ip_local_port_range parameters do not conflict. If you previously modified either of these parameters to increase the number of available ports and their ranges now overlap, sporadic network issues may occur. In severe cases, this can lead to health check failures and nodes going offline. We recommend that you either restore the default values or adjust the port ranges so they no longer overlap.

• After you adjust the port range, some NodePort or LoadBalancer Services in the cluster may still be using a NodePort that falls within the ip_local_port_range. You must reconfigure these Services to avoid conflicts. You can run kubectl edit <service-name> and change the value of the spec.ports.nodePort field to an unused NodePort.

ACK dedicated cluster: How do I add the required permissions when upgrading the CCM component to a new version?

To improve performance, newer versions of CCM introduce Alibaba Cloud APIs that require additional Resource Access Management (RAM) permissions. For example, v2.11.2 adds route management for Flannel networks, and v2.12.1 adds batch management for Network Load Balancer (NLB).

Therefore, before you upgrade the CCM component to v2.11.2 or later in an ACK dedicated cluster, you must grant the required permissions to its master RAM role for the component to function correctly.

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Cluster Information.

3. Click the Basic Information tab. In the Cluster Resources section, click the role name for Master RAM Role.

4. On the role details page in the RAM console, click Permissions. In the list of authorized policies, find the custom policy that starts with k8sMasterRolePolicy-Ccm- and click the policy name to go to the policy management page.

   For older clusters, this policy may not exist. You can select a custom policy whose name starts with k8sMasterRolePolicy-.

5. Click Edit and add the nlb:ListAsynJobs permission to the NLB-related permissions.

   

For Flannel clusters, you must also add the vpc:CreateRouteEntries and vpc:DeleteRouteEntries permissions to the VPC-related permissions.

After you add the permissions and save the policy, you can proceed with the CCM component upgrade.