Enable tracing for cluster data plane components

If you are new to tracing, see Basic Concepts to learn the basic concepts.

Prerequisites

• You have an ACK managed cluster or an ACK dedicated cluster that runs Kubernetes 1.28 or later. To upgrade your cluster, see Manually upgrade a cluster.

• You have Helm v3 or later installed. To upgrade Helm, see Migrate from Helm V2 to Helm V3.

  To install the OpenTelemetry Collector component required for this feature, you must use Helm 3.9 or later.

• You have activated Managed Service for OpenTelemetry and granted the required permissions. For more information, see Preparations.

  Managed Service for OpenTelemetry provides a comprehensive set of tools for distributed applications, including full lifecycle trace restoration, request volume statistics, topology maps, and application dependency analysis.

Follow this procedure to enable tracing for the data plane components in your cluster:

1. Retrieve the data Managed Service for OpenTelemetry and Managed Service for OpenTelemetry from Managed Service for OpenTelemetry, and then store the authentication token in a cluster secret. This allows the OpenTelemetry Collector to read the configuration and authentication information.

2. Deploy the OpenTelemetry Collector component to collect, process, and export observability data.

   The ACK Marketplace lets you quickly deploy the OpenTelemetry Collector in your cluster in DaemonSet mode.

3. In the Kubelet configuration, enable tracing and configure related parameters, such as the sampling rate. The OpenTelemetry Collector then forwards the collected data to Managed Service for OpenTelemetry.

Step 1: Obtain the endpoint and authentication token

1. Log on to the Managed Service for OpenTelemetry console. In the left-side navigation pane, click Integration Center.

2. In the Open Source Frameworks section, click the OpenTelemetry card.

3. Select a region for data reporting. Set Connection Method to Alibaba Cloud Internal Network Mode and Reporting Method to gRPC. Save the endpoint information for later use.

   When you use a region for the first time, its resources are automatically initialized. Follow the on-screen instructions to complete the process.

   

Step 2: Store the authentication token in a cluster secret

Store the authentication token in a secret within your ACK cluster. The OpenTelemetry Collector uses this authentication token by referencing it as a variable. The secret and the OpenTelemetry Collector that you will install in the next step must be in the same namespace.

By default, the OpenTelemetry Collector is deployed in the otel-collector namespace. You can create the otel-collector namespace manually or use a different namespace and select it when you install the OpenTelemetry Collector.

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Configurations > Secrets.

3. Click Create and follow the on-screen instructions to create the secret.

   Parameter	Value	Example
   Name	opentelemetry-exporter-config Important Use this exact name. Do not specify a custom name.
   Type	Opaque
   Configure the following parameters to add the data key:
   Name	grpc-token Important Use this exact name. Do not specify a custom name.
   Value	The authentication token that you obtained in Step 1.
   Encode Data Values Using Base64	Yes

Step 3: Install the OpenTelemetry Collector

The OpenTelemetry Collector is an observability data collector that provides unified capabilities for receiving, processing, and exporting data. It supports various open source observability data sources, such as Jaeger, Prometheus, and Fluent Bit, eliminating the need to deploy multiple collectors.

1. Log on to the ACK console. In the left navigation pane, click Marketplace > Marketplace.

2. Search for opentelemetry-collector and click the application card.

   This page provides a component description and parameter reference for the component's Helm chart. You can review the information on this page or follow the instructions below to complete the required configuration.

3. In the upper-right corner of the page, click Deploy. Select your cluster and namespace, and then follow the on-screen instructions to proceed to the next step.

   The namespace of the component must be the same as the namespace of the secret that you created in Step 2.

4. Select a chart version. We recommend using the default version. Set the chart parameters and follow the on-screen instructions to install the chart.

   Parameter	Description	Example
   mode	Set this parameter to daemonset to deploy the OpenTelemetry Collector as a DaemonSet. mode: "daemonset"
   enableXtraceTokenFromSecret	Set this parameter to true to reference the authentication token by using an environment variable. This method prevents the authentication token from being stored in plaintext in the OpenTelemetry Collector's configuration and avoids sensitive information leakage. enableXtraceTokenFromSecret: true Make sure that you have completed the operations in Step 2 to prevent chart deployment failures.
   exporters	Add an exporter named otlp to specify the address to which trace data is reported. Set endpoint to the intranet endpoint that you obtained in Step 1. exporters: otlp: endpoint: <your_intranet_endpoint> tls: insecure: true headers: Authentication: ${env:OPENTELEMETRY_GRPC_TOKEN}
   receivers	Add a receiver named otlp/kubelet that listens on local port 4317 to receive Kubelet trace data over OTLP/gRPC. receivers: otlp/kubelet: # Creates a receiver named otlp/kubelet. protocols: grpc: endpoint: 127.0.0.1:4317
   service.pipelines.traces	Configure the trace processing pipeline. Add the otlp exporter to service.pipelines.traces.exporters and the otlp/kubelet receiver to service.pipelines.traces.receivers. traces: exporters: - debug - otlp processors: - memory_limiter - batch - resource receivers: - otlp - jaeger - zipkin - otlp/kubelet
   hostNetwork	Set this parameter to true. This allows the OpenTelemetry Collector to directly use the host's IP address and port for communication. hostNetwork: true

   After the installation is successful, you are automatically redirected to the Helm page.

   Go to the Workloads > DaemonSets page. Select the namespace of the component. If a DaemonSet named opentelemetry-collector-agent exists and all its Pods are running, the installation is successful.

Step 4: Enable tracing for Kubelet

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Node Pools.

3. In the node pool list, find the target node pool and choose > Kubelet Configuration in the Actions column. Add custom parameters, review the notes, select the nodes to upgrade, and set a batch update policy (Maximum Parallel Nodes per Batch). Then, click Submit.

   • From the drop-down list, select tracing and then select endpoint. Set the value to localhost:4317. This sends Kubelet trace data to localhost:4317. This address corresponds to the endpoint of the otlp/kubelet receiver that you configured in Step 3.

   • From the drop-down list, select tracing again and then select samplingRatePerMillion. This parameter controls the sampling rate, which determines how many requests are sampled out of every one million requests. Depending on the overall configuration, tracing may affect the cluster's network and CPU performance. If you encounter performance issues, reduce the sampling rate based on the cluster load.

     For example, a value of 1000000 indicates a 100% sampling rate, which means all requests are sampled. A value of 100000 indicates a 10% sampling rate, and 10000 indicates a 1% sampling rate.

     

Step 5: View trace data in OpenTelemetry

1. Log on to the Managed Service for OpenTelemetry console. In the left-side navigation pane, click Applications.

2. At the top of the page, select a region, and then click the application name kubelet. Follow the console instructions to view trace information.

   This section describes only the main tabs. For more information about each tab, see Application details.

   • Trace Analysis tab: Displays information such as traces of Kubelet requests and the application topology.

     • In the search box in the upper-left corner, enter resources.k8s.cluster.name : "YourClusterId" to filter traces for a specific cluster.

       You can obtain the cluster ID from the cluster list on the ACK console.

     • In the trace list, click a Trace ID to view the details of a Kubelet request trace, including the topology view, the number of services involved, and the number of API calls.

   • Support Services tab: Displays the number of requests, number of errors, and average latency of the interfaces provided by Kubelet.

   • Dependent Services tab: Displays information about requests to services that Kubelet depends on, such as containerd. This information includes the number of requests, number of errors, and average latency.

Related documents

To monitor and record trace data for the cluster's API server component, see Enable tracing for cluster control plane components.