Common troubleshooting procedures

Troubleshoot service registration and discovery issues

Follow these steps:

1. First, confirm that the following information is correct and belongs to the correct workspace:

   • InstanceId: The unique identifier of the middleware instance.

   • AntVIP: The middleware uses AntVIP for addressing.

   • AccessKey ID: The access key ID.

   • AccessKey Secret: The access key secret.

2. If the information is correct, troubleshoot the service registration and discovery process. The troubleshooting order is: Application > MOSN > Registry Center.

   Important

   To perform the following troubleshooting steps in a container, you must obtain the kubeconfig for the corresponding user workspace.

   • Troubleshoot the application

     1. On the node, run the following command to access the application container:

        kubectl exec -it sofa-service-thrf4-whtpc -c <application_name>

        

     2. Run the following command to check if the MOSN registered port is open:

        curl -v http://127.0.0.1:13330

     3. Go to the /home/admin/logs directory.

        The framework and log file names may differ, but the log directory is always the same. For example, the log file for both Dubbo and Spring Cloud is spring.log.

     4. Search for the keywords dataId or EnterpriseMeshRegistry.

   • MOSN:

     1. Enter the mosn-sidecar-container.

     2. Find the log file: /home/admin/logs/mosn/default.log

     3. Search for system config init to check if the user-configured four-tuple information is correct.

     4. Search for dataId or registry to check if the service was registered successfully.

     5. Check if the notification interface of the registry center is open:

        netstat -anp | grep 9600

        

Troubleshoot service throttling issues

The service throttling configuration is sent from the DsrConsole microservice to MOSN through DRM. Follow these steps:

1. At the product layer, check the application's push configurations and records.

2. Check DsrConsole (middleware tenant).

   Go to the /home/admin/logs/ directory and check for error logs.

3. Check MOSN:

   1. Find the log file /home/admin/logs/mosn/default.log.

   2. Search the log file for keywords such as the application name, method name, or serviceRateLimitConfig. Check if the content in the value field matches the content that you entered on the frontend.

      find *.log | grep -rn "Alipay.cloudMeshTestJavaServer:name=com.alipay.guardian.client.sofa.drm.GuardianDrm.serviceRateLimitConfig,version=3.0@DRM"

      

Troubleshoot service routing issues

The DsrConsole microservice sends the routing configuration to the default namespace of Kubernetes (K8s) through OpenAPI. Pilot then uses ListAndWatch to monitor changes in the corresponding Custom Resource (CR) and notifies MOSN. Follow these steps:

1. Check DsrConsole:

   1. Go to the /home/admin/logs/ directory.

   2. Search for the configuration using keywords.

      Using SigmaRouterRuleDispatcher or dataId, check if the content of the body matches your configuration.

2. Check Cloud Mesh OpenAPI:

   • If the return value in step 1 is not 200, check the specific error message and take the appropriate action. For example, a 404 cluster not found error indicates that the K8s cluster information cannot be found. In this case, submit a ticket.

   • If the token or endpoint is incorrect, submit a ticket.

3. Check K8s:

   Check the content of the CRs. The CRs are in the default namespace of your K8s cluster. The routing-related CRs are virtualservices and destinationrules. Verify that their content matches what you entered.

4. Check Pilot:

   1. Check the logs to verify that the address specified by PILOT_SERVER_ADDRESS is reachable. If it is not, submit a ticket. The following log is an example of an error:

      2019-10-1117:01:24,236[ERROR][normal] fail to create stream client: rpc error: code =Unavailable desc = all SubConns are inTransientFailure, latest connection error: connection error: desc ="transport: Error while dialing dial tcp xxx.xxx.xxx.xxx:xxx: i/o timeout"

   2. Check the logs to verify that Pilot authorization was successful. If it was not, submit a ticket and check the InstanceId, AccessKey ID, and AccessKey secret. The following log is an example of an error:

      2019-10-1117:07:12,518[WARN][xds][ads client]get resp timeout: rpc error: code =Unknown desc = missing auth part of service node "xxx",retry after 1s

   3. Check if MOSN received the xDS content.

      Run curl [podIP]:34901/api/v1/config_dump to retrieve the content. If the content is empty or missing, submit a ticket.

5. Check MOSN:

   Check if the routing configuration was sent to the service consumer by following these steps:

   1. Enter the MOSN container of the service consumer. Use the dataId or xds keyword to check if the version number and weight are correct.

   2. Run the find default.log | grep -rn "xds" command.

      The following is an example:

Troubleshoot MOSN Sidecar injection failures

Follow these steps:

1. In the console, check if a cluster is created and the Sidecar injection feature is enabled.

   Enable Mesh for the cluster if it is not already enabled.

   Note

   In the Sidecar injection configuration, the Sidecar name must be mosn. Currently, only the mosn Sidecar is supported.

2. Check if a default template or an injection configuration rule is enabled.

   The following is an example of an enabled default template:

   Note

   The highlighted part is the default template. For each Sidecar type (container or virtual machine), only one default template can be enabled.

   The following is an example of an enabled injection configuration rule:

   Note

   Each time an injection rule is enabled, a ConfigMap is generated. During Sidecar injection, the conditions in the rule are matched.

   If only an injection rule is enabled, check if the matching conditions in the rule match the content of the pod that requires the injection.

3. Check if the environment variables specified for the operator match the annotations in the pod.

   Check if the application pod's YAML file contains an annotation with the key ake.cafe.sofastack.io/mosn-inject=true.

   

   Note

   The figure above shows that the Sidecar feature is enabled for the corresponding pod.

4. When creating a pod, check if K8s calls the operator's webhook.

   1. Enter the operator container.

      kubectl exec -it  sidecar-operator-7564cbf97b-jbpzz  -n hxz-smmp bash

      Modify the namespace based on your actual environment.

   2. Monitor the sidecar-operator.log file.

      tail -f /home/admin/logs/sidecar-operator.log

   3. Delete the pod and then restart it. Check if sidecar-operator.log contains any log output.

      kubectl delete pod crpc-client-5584848c67-nhzsp

      The figure below shows log output from the operator. The red box indicates that the sidecar-mosn-container-template-170 template was injected.

      • If there is log output but no template is matched, check if the mosn-template-ns startup parameter of the Sidecar operator matches the namespace of the template. If they do not match, modify the environment variable and restart the operator.

        The figure below shows the mosn-template-ns startup parameter of the Sidecar operator:

        The figure below shows the namespace of the template. The default namespace is cloudmesh. The namespace under tenant 000001 is also cloudmesh.

      • If there is no log output, check if the configured webhook address is correct.

        kubectl -n hxz-smmp get cm mesh-webhooks -oyaml