Troubleshoot Nginx Ingress issues

Table of contents

Background information

The Kubernetes community officially maintains the Ingress NGINX Controller. ACK's Nginx Ingress Controller uses the version provided by the Kubernetes community and supports all community Annotations configurations.

For Nginx Ingress resources to work properly, you must deploy an Nginx Ingress Controller in the cluster to parse the Ingress forwarding rules. The Nginx Ingress Controller receives requests, matches them against the Ingress rules, and then forwards them to the corresponding backend Service pods for processing. In Kubernetes, the relationship among a Service, an Nginx Ingress, and the Nginx Ingress Controller is as follows:

• A Service is an abstraction of backend services. One Service can represent multiple identical backend services.

• An Nginx Ingress defines reverse proxy rules that specify which Service pods receive HTTP or HTTPS requests. For example, requests are routed to different Service pods based on the Host and URL path in each request.

• The Nginx Ingress Controller is a component in the Kubernetes cluster that parses Nginx Ingress reverse proxy rules. When an Ingress is added, deleted, or modified, the Nginx Ingress Controller immediately updates its forwarding rules. When the controller receives a request, it forwards the request to the appropriate Service pod based on these rules.

The Nginx Ingress Controller retrieves Ingress resource changes from the API Server, dynamically generates configuration files required by the load balancer (such as nginx.conf), and reloads the load balancer (for example, by running nginx -s reload to reload Nginx) to apply new routing rules.

Diagnostic workflow

1. Follow these steps to check if the issue is caused by Ingress and ensure the Ingress Controller configuration is correct.

   1. Verify that access from the Controller Pod meets expectations. For details, see Manually access the Ingress and backend pods from the Controller Pod.

   2. Confirm that you use the Nginx Ingress Controller correctly. For details, see the Nginx Ingress community documentation.

2. Use the Ingress diagnostics feature to check Ingress and component configurations, and make modifications based on the prompts. For details about the Ingress diagnostics feature, see Use the Ingress diagnostics feature.

3. Follow the troubleshooting approach to identify related issues and solutions.

4. If the preceding steps do not resolve the issue, follow these steps:

   • For HTTPS certificate issues:

     1. Check whether WAF or WAF in transparent proxy mode is enabled for the domain name.

        • If enabled, confirm that WAF or transparent WAF does not have a TLS certificate configured.

        • If not enabled, proceed to the next step.

     2. Check whether the SLB uses Layer 7 listeners.

        • If yes, confirm that no TLS certificate is configured on the SLB Layer 7 listener.

        • If no, proceed to the next step.

   • For non-HTTPS certificate issues, check the error logs in the Controller Pod. For details, see Check error logs in the Controller Pod.

5. If the preceding steps do not resolve the issue, capture packets in both the Controller Pod and the corresponding backend application pod to identify the problem. For details, see Capture packets.

Troubleshooting approach

Common inspection methods

$remote_addr - [$remote_addr] - $remote_user [$time_local]
    "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length
    $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length
    $upstream_response_time $upstream_status $req_id $host [$proxy_alternative_upstream_name]

kubectl logs <controller pod name> -n <namespace> | less

42.11.**.** - [42.11.**.**]--[25/Nov/2021:11:40:30 +0800]"GET / HTTP/1.1" 200 615 "_" "curl/7.64.1" 76 0.001 [default-nginx-svc-80] 172.16.254.208:80 615 0.000 200 46b79dkahflhakjhdhfkah**** 47.11.**.**[]
42.11.**.** - [42.11.**.**]--[25/Nov/2021:11:40:31 +0800]"GET / HTTP/1.1" 200 615 "_" "curl/7.64.1" 76 0.001 [default-nginx-svc-80] 172.16.254.208:80 615 0.000 200 fadgrerthflhakjhdhfkah**** 47.11.**.**[]

Capture packets

When you cannot locate the issue, capture packets for auxiliary diagnosis.

1. Based on preliminary issue identification, determine whether the network issue occurs in the Ingress Pod or the application Pod. If information is insufficient, capture packets from both sides.

2. Log on to the node where the problematic application Pod or Ingress Pod resides.

3. On the ECS instance (outside the container), run the following command to capture Ingress traffic to a file.

   tcpdump -i any host <application Pod IP or Ingress Pod IP> -C 20 -W 200 -w /tmp/ingress.pcap

4. Monitor logs and stop packet capture when the expected error occurs.

5. Combine error logs with business logs to locate the exact packet information at the time of the error.

   Note

   • Under normal circumstances, packet capture does not affect business operations and only slightly increases CPU load and disk writes.

   • The preceding command rotates captured packets, generating up to 200 .pcap files of 20 MB each.

Unable to access the cluster LoadBalancer external address from within the cluster

Issue symptoms

Some pods under certain nodes in the cluster cannot access backend pods through the external address (SLB instance IP address) of the Nginx Ingress Controller, while others can.

Root cause

This issue is caused by the externalTrafficPolicy configuration of the Service associated with the Controller. This configuration determines how external traffic is handled: when set to Local, only backend pods on the same node as the Controller pod can successfully receive requests; when set to Cluster, access works normally. Requests from within the cluster to the LoadBalancer external address are treated as external traffic.

Solutions

• (Recommended) Access from within the Kubernetes cluster using ClusterIP or the Ingress service name. The Ingress service name is nginx-ingress-lb.kube-system.

• Run the kubectl edit svc nginx-ingress-lb -n kube-system command to modify the Ingress service. Change externalTrafficPolicy in the LoadBalancer Service to Cluster. If the cluster uses the Flannel container network plugin, the client source IP will be lost. If Terway is used, the source IP is preserved.

• Example:

  apiVersion: v1
  kind: Service
  metadata:
    annotations:
      service.beta.kubernetes.io/backend-type: eni   # Direct ENI.
    labels:
      app: nginx-ingress-lb
    name: nginx-ingress-lb
    namespace: kube-system
  spec:
    externalTrafficPolicy: Cluster

  For more information about Service Annotations, see Configure Classic Load Balancer (CLB) using Annotations.

Unable to access the Ingress Controller itself

Issue symptoms

In Flannel clusters, accessing the Ingress itself from the Ingress Pod using the domain name, SLB IP, or Cluster IP results in partial or complete request failures.

Root cause

The default Flannel configuration does not allow loopback access.

Solutions

• (Recommended) If possible, rebuild the cluster and use the Terway network plugin. Migrate existing cluster workloads to the Terway-mode cluster.

• If rebuilding the cluster is not feasible, modify the Flannel configuration to enable hairpinMode. After modifying the configuration, recreate the Flannel Pod to apply the changes.

  1. Run the following command to edit Flannel.

     kubectl edit cm kube-flannel-cfg -n kube-system

  2. In the returned cni-conf.json file, add "hairpinMode": true to the delegate section.

     Example:

     cni-conf.json: |
         {
           "name": "cb0",
           "cniVersion":"0.3.1",
           "type": "flannel",
           "delegate": {
             "isDefaultGateway": true,
             "hairpinMode": true
           }
         }

  3. Run the following command to delete and recreate Flannel.

     kubectl delete pod -n kube-system -l app=flannel   

The default or old TLS certificate is still used after adding or modifying a TLS certificate in the cluster

Issue symptoms

After adding or modifying a Secret in the cluster and specifying secretName in the Ingress, access still uses the default certificate (Kubernetes Ingress Controller Fake Certificate) or the old certificate.

Root cause

• The certificate is not returned by the Ingress Controller in the cluster.

• The certificate is invalid and not correctly loaded by the Controller.

• The Ingress Controller returns certificates based on SNI, but the TLS handshake might not carry SNI.

Solutions

• Use one of the following methods to confirm whether the SNI field is included during TLS connection establishment:

  • Use a newer browser version that supports SNI.

  • When testing certificates with the openssl s_client command, include the -servername parameter.

  • When using the curl command, add hosts or use the --resolve parameter to map the domain name instead of using IP + Host request header.

• Confirm that no TLS certificate is configured on WAF, WAF in transparent proxy mode, or the SLB Layer 7 listener. The TLS certificate should be returned by the Ingress Controller in the cluster.

• Use the Ingress diagnostics feature in the intelligent operations console to check for configuration errors and error logs. For details, see Use the Ingress diagnostics feature.

• Run the following command to manually view error logs in the Ingress Pod and make modifications based on the prompts in the error logs.

  kubectl logs <ingress pod name> -n <pod namespace> | grep -E ^[EW]

Unable to connect to a gRPC service exposed through Ingress

Issue symptoms

Unable to access the gRPC service behind the Ingress.

Root cause

• Annotation specifying the backend protocol type is not set in the Ingress resource.

• The gRPC service can only be accessed through a TLS port.

Solutions

• Set the following Annotation in the corresponding Ingress resource: nginx.ingress.kubernetes.io/backend-protocol:"GRPC".

• Confirm that the client sends requests using the TLS port and encrypts the traffic.

Unable to connect to a backend HTTPS service

Issue symptoms

• Unable to access the HTTPS service behind the Ingress.

• The response might be 400 with the message The plain HTTP request was sent to HTTPS port.

Root cause

The Ingress Controller sends requests to backend pods using the default HTTP protocol.

Solution

Set the following Annotation in the Ingress resource: nginx.ingress.kubernetes.io/backend-protocol:"HTTPS".

Unable to preserve the client source IP in the Ingress pod

Issue symptoms

The Ingress Pod cannot preserve the client's originating IP address and displays the node IP, an IP in the 100.XX.XX.XX CIDR block, or another address.

Root cause

• The externalTrafficPolicy of the Service used by Ingress is set to Cluster.

• Layer 7 proxy is used on the SLB.

• WAF or WAF in transparent proxy mode is used.

Solutions

• For cases where externalTrafficPolicy is set to Cluster and a Layer 4 SLB is used on the frontend:

  Change externalTrafficPolicy to Local. However, this might cause internal cluster access to the Ingress via SLB IP to fail. For the solution, see Unable to access the cluster LoadBalancer external address from within the cluster.

• For cases using Layer 7 proxy (Layer 7 SLB, WAF, or transparent WAF):

  1. Ensure that the Layer 7 proxy is used and the X-Forwarded-For request header is enabled.

  2. Add enable-real-ip: "true" to the Ingress Controller's ConfigMap (default is nginx-configuration in the kube-system namespace).

  3. Check logs to verify whether the source IP can be obtained.

• For long chains with multiple forwards (for example, an additional reverse proxy service configured before the Ingress Controller), enable enable-real-ip and observe the value of remote_addr in the logs to determine whether the real IP is passed to the Ingress container via the X-Forwarded-For request header. If not, use X-Forwarded-For or similar methods to carry the client's real IP before the request reaches the Ingress Controller.

Phased release rules do not take effect

Issue symptoms

Phased release is configured in the cluster, but the phased release rules do not take effect.

Root cause

Possible causes:

• When using canary-* related Annotations, nginx.ingress.kubernetes.io/canary: "true" is not set.

• When using canary-* related Annotations, Nginx Ingress Controller versions earlier than 0.47.0 require the Host field in the Ingress rule to contain your business domain name and cannot be empty.

Solutions

• Modify nginx.ingress.kubernetes.io/canary: "true" or the Host field in the Ingress rule based on the preceding causes. For details, see Routing rules.

• If the preceding situations do not apply, see Traffic distribution does not match phased release rules, or other traffic enters the phased release service.

Traffic distribution does not match phased release rules, or other traffic enters the phased release service

Issue symptoms

Phased release rules are configured, but traffic is not distributed according to the rules, or normal Ingress traffic enters the phased release service.

Root cause

In the Nginx Ingress Controller, phased release rules apply to all Ingress resources that use the same Service, not just a single Ingress.

For details about this issue, see Ingress with phased release rules affects all Ingress resources that use the same Service.

Solution

For Ingress resources that require phased release (including those using service-match and canary-* related Annotations), create independent Services (including both production and phased release Services) pointing to the original pods. Then enable phased release for the Ingress. For details, see Implement phased release and blue-green deployment using Nginx Ingress.

Error "failed calling webhook" when creating an Ingress resource

Issue symptoms

When adding an Ingress resource, the error "Internal error occurred: failed calling webhook..." appears, as shown in the following figure.

Root cause

When adding an Ingress resource, the system validates the Ingress resource through a service (default is ingress-nginx-controller-admission). If the validation chain fails (for example, the service is deleted or the Ingress controller is deleted), validation fails and the Ingress resource addition is rejected.

Solutions

• Check along the webhook chain to ensure all resources exist and work properly. The chain is ValidatingWebhookConfiguration → Service → Pod.

• Confirm that the admission feature of the Ingress Controller Pod is enabled and the Pod can be accessed externally.

• If the Ingress Controller has been deleted or webhook functionality is not needed, delete the ValidatingWebhookConfiguration resource directly.

SSL_ERROR_RX_RECORD_TOO_LONG error during HTTPS access

Issue symptoms

During HTTPS access, the error SSL_ERROR_RX_RECORD_TOO_LONG or routines:CONNECT_CR_SRVR_HELLO:wrong version number appears.

Root cause

The HTTPS request is sent to a non-HTTPS port, such as an HTTP port.

Solution

Modify the SLB or Service settings based on the actual situation to ensure HTTPS requests reach the correct port.

Common HTTP error codes appear

Issue symptoms

Requests return non-2xx and non-3xx errors, such as 502, 503, 413, or 499.

Root cause and solutions

Check logs to determine whether the error is returned by the Ingress Controller. For details, see View access logs from the Controller Pod in Simple Log Service (SLS). If so, refer to the following solutions:

• 413 error

  • Root cause: The Nginx Ingress Controller works normally, but the request payload exceeds the maximum allowed size.

  • Solution: Modify the Controller configuration using kubectl edit cm -n kube-system nginx-configuration and adjust the values of nginx.ingress.kubernetes.io/client-max-body-size and nginx.ingress.kubernetes.io/proxy-body-size as needed (default value 20m).

• 499 error

  • Root cause: The client disconnects early for some reason, which is not necessarily a component or backend issue.

  • Solutions:

    • If there are few 499 errors, they may be normal depending on the business and can be ignored.

    • If there are many 499 errors, check whether the backend processing time and frontend request timeout meet expectations.

• 502 error

  • Root cause: The Nginx Ingress works normally, but its pod cannot connect to the target backend pod.

  • Solutions:

    • Consistent occurrence:

      • Possibly caused by backend Service or pod configuration errors. Check the backend Service port configuration and application code in the container.

    • Intermittent occurrence:

      • Possibly caused by high load on the Nginx Ingress Controller pod. Evaluate the load using request and connection statistics from the associated SLB instance and refer to Configure Nginx Ingress Controller for high-load scenarios to allocate more resources to the Controller.

      • Possibly caused by the backend pod actively closing sessions. The Nginx Ingress Controller enables persistent connections by default. Confirm that the backend persistent connection idle timeout is longer than the Controller's idle timeout (default 900 seconds).

    • If none of the above methods identify the issue, perform packet capture analysis.

• 503 error

  • Root cause: The Ingress Controller cannot find backend pods or all pods are inaccessible.

  • Solutions:

    • Intermittent occurrence:

      • Refer to the 502 error solutions.

      • Check the backend application readiness status and configure appropriate health checks.

    • Consistent occurrence:

      Check whether the backend Service configuration is correct and whether Endpoints exist.

net::ERR_HTTP2_SERVER_REFUSED_STREAM error appears

Issue symptoms

When accessing a webpage, some resources fail to load correctly, and the console shows net::ERR_HTTP2_SERVER_REFUSED_STREAM or net::ERR_FAILED errors.

Root cause

The number of concurrent resource requests exceeds the HTTP/2 maximum stream limit.

Solutions

• (Recommended) Adjust http2-max-concurrent-streams to a higher value (default 128) in the ConfigMap based on actual needs. For details, see http2-max-concurrent-streams.

• Disable HTTP/2 support directly in the ConfigMap by setting use-http2 to false. For details, see use-http2.

Error "The param of ServerGroupName is illegal" appears

Root cause

The ServerGroupName is generated in the format namespace+svcName+port. The server group name must be 2–128 characters long, start with a letter (uppercase or lowercase) or Chinese character, and can contain digits, periods (.), underscores (_), and hyphens (-).

Solution

Modify the server group name to comply with the format requirements.

Error "certificate signed by unknown authority" when creating an Ingress

Root cause

When creating an Ingress, the error shown in the preceding figure appears because multiple Ingress deployments share the same resources (such as Secrets, services, and webhook configurations), causing SSL certificate mismatches during webhook execution and communication with backend services.

Solution

Redeploy the two Ingress sets with non-overlapping resources. For information about resources included in Ingress, see What updates does the system perform when upgrading the Nginx Ingress Controller component in ACK component management?.

Ingress Pod health check fails and causes restarts

Issue symptoms

The Controller Pod restarts due to health check failures.

Root cause

• High load on the Ingress Pod or its node causes health check failures.

• Kernel parameters tcp_tw_reuse or tcp_timestamps are set on cluster nodes, which may cause health check failures.

Solutions

• Scale out the Ingress Pod and observe whether the issue persists. For details, see High-availability deployment of Nginx Ingress Controller.

• Disable tcp_tw_reuse or set it to 2, and simultaneously disable tcp_timestamps. Observe whether the issue persists.

Add TCP or UDP services

1. Add corresponding entries to the relevant ConfigMap (default is tcp-services and udp-services in the kube-system namespace).

   For example, to map port 8080 of example-go in the default namespace to port 9000, use the following example.

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: tcp-services
     namespace: ingress-nginx
   data:
     9000: "default/example-go:8080"  # Map port 8080 to port 9000.

2. Add the mapped ports to the Ingress Deployment (default is nginx-ingress-controller in the kube-system namespace).

3. Add the mapped ports to the Service corresponding to the Ingress.

   Expand to view sample code

   apiVersion: v1
   kind: Service
   metadata:
     name: ingress-nginx
     namespace: ingress-nginx
     labels:
       app.kubernetes.io/name: ingress-nginx
       app.kubernetes.io/part-of: ingress-nginx
   spec:
     type: LoadBalancer
     ports:
       - name: http
         port: 80
         targetPort: 80
         protocol: TCP
       - name: https
         port: 443
         targetPort: 443
         protocol: TCP
       - name: proxied-tcp-9000
         port: 9000
         targetPort: 9000
         protocol: TCP
     selector:
       app.kubernetes.io/name: ingress-nginx
       app.kubernetes.io/part-of: ingress-nginx

   For more information about adding TCP and UDP services, see Expose TCP and UDP services.

Ingress rules do not take effect

Issue symptoms

Added or modified Ingress rules do not take effect.

Root cause

• Ingress configuration errors prevent new rules from loading correctly.

• Ingress resource configuration errors do not match expected configurations.

• Permission issues with the Ingress Controller prevent it from monitoring Ingress resource changes.

• An old Ingress uses the server-alias to configure a domain name that conflicts with the new Ingress, causing the rules to be ignored.

Some resources fail to load or a blank screen appears after rewriting to the root directory

Issue symptoms

After rewriting access using the Ingress rewrite-target annotation, some page resources fail to load or a blank screen appears.

Root cause

• rewrite-target might not be configured using regular expressions.

• The application hardcodes resource paths to the root directory.

Solutions

• Check whether rewrite-target is used with regular expressions and capturing groups. For details, see Rewrite.

• Check whether frontend requests access the correct paths.

How to fix abnormal SLS log parsing after a version upgrade

Issue symptoms

The ingress-nginx-controller component has two main versions: 0.20 and 0.30. After upgrading from version 0.20 to 0.30 using the Add-ons feature in the console, the Ingress Dashboard fails to correctly display actual backend service access during phased release or blue-green deployment.

Root cause

Versions 0.20 and 0.30 have different default output formats, causing the Ingress Dashboard to fail to correctly display actual backend service access during phased release or blue-green deployment.

Solution

Perform the following steps to fix the issue by updating the nginx-configuration configmap and k8s-nginx-ingress configuration.

1. Update the nginx-configuration configmap.

   • If you have not modified the nginx-configuration configmap, save the following content as nginx-configuration.yaml and run the kubectl apply -f nginx-configuration.yaml command to deploy it.

     apiVersion: v1
     kind: ConfigMap
     data:
       allow-backend-server-header: "true"
       enable-underscores-in-headers: "true"
       generate-request-id: "true"
       ignore-invalid-headers: "true"
       log-format-upstream: $remote_addr - [$remote_addr] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id $host [$proxy_alternative_upstream_name]
       max-worker-connections: "65536"
       proxy-body-size: 20m
       proxy-connect-timeout: "10"
       reuse-port: "true"
       server-tokens: "false"
       ssl-redirect: "false"
       upstream-keepalive-timeout: "900"
       worker-cpu-affinity: auto
     metadata:
       labels:
         app: ingress-nginx
       name: nginx-configuration
       namespace: kube-system

   • If you have modified the nginx-configuration configmap, run the following command to avoid overwriting your configuration:

     kubectl edit configmap nginx-configuration -n kube-system

   Add [$proxy_alternative_upstream_name] to the end of the log-format-upstream field, then save and exit.

2. Update the k8s-nginx-ingress configuration.

   Save the following content as k8s-nginx-ingress.yaml and run the kubectl apply -f k8s-nginx-ingress.yaml command to deploy it.

   Expand to view YAML content

   apiVersion: log.alibabacloud.com/v1alpha1
   kind: AliyunLogConfig
   metadata:
     namespace: kube-system
     # your config name, must be unique in you k8s cluster
     name: k8s-nginx-ingress
   spec:
     # logstore name to upload log
     logstore: nginx-ingress
     # product code, only for k8s nginx ingress
     productCode: k8s-nginx-ingress
     # logtail config detail
     logtailConfig:
       inputType: plugin
       # logtail config name, should be same with [metadata.name]
       configName: k8s-nginx-ingress
       inputDetail:
         plugin:
           inputs:
           - type: service_docker_stdout
             detail:
               IncludeLabel:
                 io.kubernetes.container.name: nginx-ingress-controller
               Stderr: false
               Stdout: true
           processors:
           - type: processor_regex
             detail:
               KeepSource: false
               Keys:
               - client_ip
               - x_forward_for
               - remote_user
               - time
               - method
               - url
               - version
               - status
               - body_bytes_sent
               - http_referer
               - http_user_agent
               - request_length
               - request_time
               - proxy_upstream_name
               - upstream_addr
               - upstream_response_length
               - upstream_response_time
               - upstream_status
               - req_id
               - host
               - proxy_alternative_upstream_name
               NoKeyError: true
               NoMatchError: true
               Regex: ^(\S+)\s-\s\[([^]]+)]\s-\s(\S+)\s\[(\S+)\s\S+\s"(\w+)\s(\S+)\s([^"]+)"\s(\d+)\s(\d+)\s"([^"]*)"\s"([^"]*)"\s(\S+)\s(\S+)+\s\[([^]]*)]\s(\S+)\s(\S+)\s(\S+)\s(\S+)\s(\S+)\s*(\S*)\s*\[*([^]]*)\]*.*
               SourceKey: content

Error "cannot list/get/update resource" appears

Issue symptoms

Using the method described in Check error logs in the Controller Pod, you find Controller error logs in the Pod similar to the following:

User "system:serviceaccount:kube-system:ingress-nginx" cannot list/get/update resource "xxx" in API group "xxx" at the cluster scope/ in the namespace "kube-system"

Root cause

The Nginx Ingress Controller lacks permissions to update related resources.

Solutions

• Confirm from the logs whether the issue stems from a ClusterRole or Role.

  • If the log contains at the cluster scope, the issue originates from the ClusterRole (ingress-nginx).

  • If the log contains in the namespace "kube-system", the issue originates from the Role (kube-system/ingress-nginx).

• Confirm that the corresponding permissions and bindings are complete.

  • For ClusterRole:

    • Ensure that the ClusterRole ingress-nginx and ClusterRoleBinding ingress-nginx exist. If they do not exist, consider recreating them, restoring them, or reinstalling the component.

    • Ensure that the ClusterRole ingress-nginx includes the permissions mentioned in the logs (for example, List permission for networking.k8s.io/ingresses). If the permission is missing, add it manually to the ClusterRole.

  • For Role:

    • Confirm that the Role kube-system/ingress-nginx and RoleBinding kube-system/ingress-nginx exist. If they do not exist, consider recreating them, restoring them, or reinstalling the component.

    • Confirm that the Role ingress-nginx includes the permissions mentioned in the logs (for example, Update permission for the ConfigMap ingress-controller-leader-nginx). If the permission is missing, add it manually to the Role.

Error "configuration file failed" appears

Issue symptoms

Using the method described in Check error logs in the Controller Pod, you find Controller error logs in the Pod similar to the following:

requeuing……nginx: configuration file xxx test failed（multiple lines）

Root cause

Configuration errors cause Nginx configuration reload to fail, usually due to syntax errors in Ingress rules or Snippets inserted in the ConfigMap.

Solutions

• Check the error messages in the logs (ignore warn-level messages) to roughly locate the issue. If the error message is unclear, use the line number in the error to enter the Pod and view the corresponding file. In the following example, the file is /tmp/nginx/nginx-cfg2825306115, line 449.

  Run the following command to check for configuration errors near the specified line.

  # Enter the Pod to run commands.
  kubectl exec -n <namespace> <controller pod name> -it -- bash
  # View the erroneous file with line numbers and check for configuration errors near the specified line.
  cat -n /tmp/nginx/nginx-cfg2825306115

• Based on the error message and configuration file, identify and fix the error according to your actual configuration.

Error "Unexpected error validating SSL certificate" appears

Issue symptoms

Using the method described in Check error logs in the Controller Pod, you find Controller error logs in the Pod similar to the following:

Unexpected error validating SSL certificate "xxx" for server "xxx"

Root cause

Certificate configuration errors. Common causes include domain names in the certificate not matching those configured in the Ingress. Some Warning-level logs (such as missing SAN extensions in the certificate) do not affect normal certificate usage. Determine whether an issue exists based on the actual situation.

Solution

Check certificate issues in the cluster based on the error content.

• Verify that the cert and key formats and contents are correct.

• Verify that the domain names in the certificate match those configured in the Ingress.

• Check whether the certificate has expired.

Issue with many uncleaned configuration files in the Controller

Troubleshoot persistent Pending status of pods after Controller upgrade

kubectl -n kube-system describe pod <pending-pod-name>

kubectl -n kube-system get events

TCP stream mixing issue with multiple CLBs under high concurrency in Flannel CNI + IPVS clusters