Enable distributed tracing in ASM

Description of the sample application

def getForwardHeaders(request):
    headers = {}
    # x-b3-*** headers can be populated using the opentracing span
    span = get_current_span()
    carrier = {}
    tracer.inject(
        span_context=span.context,
        format=Format.HTTP_HEADERS,
        carrier=carrier)
    headers.update(carrier)
    # ...
    incoming_headers = ['x-request-id']
    # ...
    for ihdr in incoming_headers:
        val = request.headers.get(ihdr)
        if val is not None:
            headers[ihdr] = val
    return headers

@GET
@Path("/reviews/{productId}")
public Response bookReviewsById(@PathParam("productId") int productId,
                            @HeaderParam("end-user") String user,
                            @HeaderParam("x-request-id") String xreq,
                            @HeaderParam("x-b3-traceid") String xtraceid,
                            @HeaderParam("x-b3-spanid") String xspanid,
                            @HeaderParam("x-b3-parentspanid") String xparentspanid,
                            @HeaderParam("x-b3-sampled") String xsampled,
                            @HeaderParam("x-b3-flags") String xflags,
                            @HeaderParam("x-ot-span-context") String xotspan) {
  if (ratings_enabled) {
    JsonObject ratingsResponse = getRatings(Integer.toString(productId), user, xreq, xtraceid, xspanid, xparentspanid, xsampled, xflags, xotspan);

Access example

Enter a URL in the format of http://{IP address of the ingress gateway}/productpage in the address bar of your browser, and then press the Enter key. You are navigated to the page of the Bookinfo application.

View applications

The Applications page of the Managed Service for OpenTelemetry console displays key metrics for all monitored applications, including the number of requests and errors on the current day and the health status. You can also configure labels for applications so that you can filter applications by label.

1. Log on to the Managed Service for OpenTelemetry console.

2. In the left-side navigation pane, click Applications. At the top of the page, select the target region.

View application details

The Application Details page shows key metrics, call topology, and traces for the application on each server where it is deployed.

1. Log on to the Managed Service for OpenTelemetry console.

2. In the left-side navigation pane, click Applications. At the top of the page, select the target region, and then click the name of the target application.

3. In the left-side navigation pane, click Application Details. In the machine list on the left, click All or a machine identified by its IP address.

   On the Overview tab, you can view the call topology and key performance metrics. On the Traces tab, you can view a list of traces for the application on the selected machine, sorted by duration in descending order. The page lists a maximum of 100 traces. Click the Traces tab to view the list of trace records. The left panel provides sorting metrics such as Response Time, Requests, and Errors, as well as instance filtering. The list on the right displays information for each span, such as Start Time, Span Name, Host/IP, Duration, Status, and traceid. You can filter the records by using the Duration greater than input box and the Exception checkbox. Click View logs to view the log details for the corresponding call.

View the waterfall chart of a trace

The waterfall chart of a trace shows the log generation time, status, IP address or server name, service name, and timeline.

1. On the Application Details page, click the Traces tab, and then click the Trace ID of the target trace.

2. On the Traces page that appears, view the waterfall chart for the trace.

   Note

   • The IP Address field can display an IP address or a machine name, depending on the display configuration on the Application Settings page. For more information, see Manage applications and tags.

   • You can move the pointer over a service name to view the information of the service, including the duration, start time, tags, and log events. For more information, see Application details.

   The top of the trace details page displays a summary, including the start time, total duration, number of applications, trace depth, and total number of spans. Below the summary, the span list uses indentation to show the call hierarchy between services. The columns include Span Name, Timeline (a blue bar that indicates the percentage of time consumed), Application Name, Start Time, IP Address, and Status.

FAQ

Why am I unable to view the trace information after I collect tracing data to Managed Service for OpenTelemetry in the ASM console?

1. View the trace push logs.

   1. Obtain the kubeconfig file of the cluster and use kubectl to connect to the cluster. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

   2. Run the following command to view the trace push logs of the pods related to the tracing-on-external-zipkin deployment in the istio-system namespace:

      kubectl logs "$(kubectl get pods -n istio-system  -l app=tracing -o jsonpath='{.items[0].metadata.name}')" -n istio-system -c  nginx

      If the status value in the output Nginx log is 406, the POST request to /api/v2/spans returned an HTTP 406 status code.

      remote_addr:192.168.26.98 time_local:17/Aug/2021:08:15:17 +0000 method:POST uri:/api/v2/spans host:zipkin  status:406   bytes_sent:1146 referer:- useragent:- forwardedfor:192.168.26.98  request_time:0.012  upstream_response_time:0.011
      remote_addr:192.168.26.98 time_local:17/Aug/2021:08:15:17 +0000 method:POST uri:/api/v2/spans host:zipkin  status:406   bytes_sent:1146 referer:- useragent:- forwardedfor:192.168.26.98  request_time:0.012  upstream_response_time:0.013

2. View the quota on the number of service requests per day and the actual number of service requests on the previous day.

   1. Log on to the Managed Service for OpenTelemetry console.

   2. In the left-side navigation pane, click Cluster Configurations to view the request quota and yesterday's request count.

      In the Quota configuration panel, the Configuration quota field displays the current quota value in units of 10,000 requests. The actual request count from yesterday is shown in parentheses in the panel title. If yesterday's request count exceeds the configured quota, you must increase the quota.

   If the actual request count from the previous day exceeds the configured daily quota, the reported data is discarded.

3. Modify the quota on the number of service requests per day.

   If the actual daily request count exceeds the daily quota, reported data is discarded and no trace information is displayed. Increase the daily quota to a value greater than the actual daily request count to resolve this issue.

   1. In the left-side navigation pane of the Managed Service for OpenTelemetry console, click Cluster Configurations.

   2. On the Cluster Configurations tab, in the Quota configuration area, modify the Configuration quota to a value greater than the service request count. Then, click Save.

   3. In the Prompt dialog box, click OK.

Why is the traceid changed after the request is processed by the ingress gateway or sidecar?

This issue occurs when the propagation headers in your application's request are incomplete, causing the sidecar proxy to regenerate them. To use a custom traceid, include at least x-b3-traceid and x-b3-spanid in B3 propagation format, or at least a traceparent header in W3C Trace Context propagation format.