DocsICP FilingConsole
官方文档
首页

Connect to Jaeger

更新时间：
复制 MD 格式
产品详情
我的收藏

This topic describes how to use a Jaeger client to connect to Simple Log Service.

Background information

Containers and serverless programming make software delivery and deployment more efficient. As architectures evolve, the following changes occur.

  • Application architectures evolve from monolithic systems to microservices. Business logic is implemented through calls and requests between microservices.

  • From a resource perspective, physical servers are gradually replaced by virtual resources.

架构

These changes demonstrate that while architectures have become more elastic and standardized, they have also introduced more complex requirements for operations and maintenance (O&M) and diagnostics. To meet these needs, a variety of DevOps-oriented diagnostic and analysis systems have been created, including centralized logging systems (Logging), centralized metrics systems (Metrics), and distributed tracing systems (Tracing).

Alibaba Cloud also provides XTrace, which functions similarly to Jaeger and is used for distributed tracing. For more information, see XTrace.

Features of logging, metrics, and tracing

  • Logging records discrete events.

    For example, logs provide debug or error information for an application. This information is essential for diagnosing problems.

  • Metrics record aggregatable data.

    For example, the current depth of a queue can be a metric that is updated when an element is enqueued or dequeued. The number of HTTP requests can be a counter that is incremented when a new request arrives.

  • Tracing records request-scoped information.

    For example, tracing records the execution process and the execution time of a remote method call. Tracing is a powerful tool for troubleshooting system performance issues.

tedian

Note

This image is from the Peter Bourgon website.

You can classify existing systems based on the preceding information. For example, Zipkin focuses on tracing. Prometheus starts with a focus on metrics and may integrate more tracing features over time, but it is unlikely to provide in-depth logging capabilities. ELK and Alibaba Cloud Simple Log Service focus on logging but continuously integrate features from the other two domains. They are moving toward the center of the diagram.

For more information about the relationship between these three systems, see Peter Bourgon. The following sections focus on tracing.

Tracing

Popular tracing software includes:

  • Dapper (Google): The basis for all tracers

  • StackDriver Trace (Google)

  • Zipkin (Twitter)

  • Appdash (golang)

  • EagleEye (Taobao)

  • Diting (Pangu, the trace system used by Alibaba Cloud products)

  • Yuntu (Ant Group trace system)

  • sTrace (Shenma)

  • X-Ray (AWS)

Distributed tracing systems are evolving rapidly and are available in various types. They generally consist of three core steps:

  • Code instrumentation

  • Data storage

  • Query and visualization

The following figure shows an example of a distributed call. A client initiates a request. The request first reaches a load balancer. Then, it passes through an authentication service and a billing service. After that, it requests resources and finally returns a result. exampleAfter data is collected and stored, a distributed tracing system typically uses a time series chart with a timeline to visualize the call chain. However, during data ingestion, the logic for collecting trace data is deeply integrated with the user's code. In addition, the APIs of different systems are not compatible. This makes it difficult to switch tracing systems without major changes.timeline

OpenTracing

The OpenTracing specification was created to solve API incompatibility issues between different distributed tracing systems. OpenTracing is a lightweight, standardized layer between applications or libraries and the tracing or log analysis systems. For more information, see OpenTracing.

  • Benefits of OpenTracing:

    • OpenTracing is part of the Cloud Native Computing Foundation (CNCF). It provides uniform concepts and data standards for distributed tracing.

    • OpenTracing provides platform-neutral and vendor-neutral APIs. This allows developers to easily add or change tracing systems.

  • OpenTracing data model:

    In OpenTracing, a trace is implicitly defined by the spans that belong to it. A trace can be considered a directed acyclic graph (DAG) of spans. The relationships between spans are called References. In the following example, the call chain consists of eight spans. 

    Causal relationships between spans in a single trace
        [Span A]  <---(the root span)
            |
     +------+------+
     |             |
 [Span B]      [Span C] <---(Span C is a child of Span A, ChildOf)
     |             |
 [Span D]      +---+-------+
               |           |
           [Span E]    [Span F] >>> [Span G] >>> [Span H]
                                       ^
                                       |
                                       ^
                         (Span G is called after Span F, FollowsFrom)

    A time series chart based on a timeline can also provide a clearer visualization of a trace. The following is an example:

    Time relationships between spans in a single trace
--|---------|---------|---------|---------|---------|---------|---------|--> time
 [Span A..................................................................]
   [Span B..............................................................]
      [Span D..........................................................]
    [Span C........................................................]
         [Span E.......]        [Span F..] [Span G..] [Span H..]

    Each span contains the following information:

    • An operation name.

    • Start timestamp: Specifies the start time.

    • Finish timestamp: The time when a task is completed.

    • A set of key-value tags. The key must be a string, and the value can be a string, a Boolean, or a number.

    • A collection of log records for a span. Each log record includes a key-value pair and a timestamp. The key must be a string, and the value can be any type. Not all OpenTracing-compatible tracers must support all value types.

    • SpanContext: The context object for a span. Each SpanContext contains the following state:

      • In OpenTracing, implementations use a unique Span to transmit the state of the current call chain (such as the Trace and Span IDs) across process boundaries.

      • Baggage Items: A set of key-value pairs that travel with the trace and must also be transported across process boundaries.

    • References: Zero or more references to related spans. Spans are related to each other through their SpanContext objects.

    For more information about the OpenTracing data model, see OpenTracing data model.

For information about all OpenTracing implementations, see OpenTracing. Among these implementations, Jaeger and Zipkin are the most popular. For more information, see Jaeger and Zipkin.

Jaeger

Jaeger is an open source distributed tracing system from Uber. It is compatible with the OpenTracing API.Jaeger

Note

This image is from the JAEGER website.

As shown in the preceding figure, Jaeger consists of the following main components.

  • Jaeger Client: Implements OpenTracing-compliant software development kits (SDKs) for different languages. Applications use the API to report trace data. The client library sends trace information to the jaeger-agent according to the sampling policy specified by the application.

  • Agent: A network daemon process that listens on a UDP port to receive span data. It sends the data in batches to the collector. It is designed as a component to be deployed on all hosts. The agent decouples the client library from the collector and hides the collector's routing and discovery details from the client library.

  • Collector: Receives data from the jaeger-agent and writes the data to backend storage. The collector is a stateless component. You can run any number of jaeger-collectors at the same time.

  • Data Store: The backend storage is a pluggable component that supports writing data to Cassandra and Elasticsearch.

  • Query: Receives query requests, retrieves traces from the backend storage system, and displays them in the UI. The Query component is stateless. You can start multiple instances and deploy them behind an Nginx load balancer.

Jaeger on Alibaba Cloud Log Service

Jaeger on Alibaba Cloud Simple Log Service is a distributed tracing system developed based on Jaeger. It supports the persistence of collected trace data to Simple Log Service. You can query and display the data using the native Jaeger UI. For more information, see Jaeger on Alibaba Cloud Simple Log Service.log

Advantages of Jaeger:

  • Native Jaeger only supports data persistence to Cassandra and Elasticsearch. This requires you to maintain the backend storage system and manage its capacity. Jaeger on Alibaba Cloud Simple Log Service leverages the massive data processing capabilities of Simple Log Service. This lets you enjoy the benefits of Jaeger's distributed tracing without worrying about backend storage issues.

  • The Jaeger UI only provides features for querying and displaying traces. It has limited support for analyzing and troubleshooting problems. Using Jaeger on Alibaba Cloud Simple Log Service, you can use the powerful query and analysis features of Simple Log Service to identify system problems more quickly.

For more information about how to configure Jaeger on Alibaba Cloud Simple Log Service, see GitHub.

Jaeger configuration example

HotROD is an application that consists of multiple microservices. It uses the OpenTracing API to record trace information.

The following video shows how to use Jaeger on Alibaba Cloud Simple Log Service to diagnose problems in HotROD. The video covers the following topics:

  1. Configure Simple Log Service.

  2. Run Jaeger using docker-compose.

  3. Run HotROD.

  4. Retrieve specific traces using the Jaeger UI.

  5. View the details of traces using the Jaeger UI.

  6. Locate application performance bottlenecks using the Jaeger UI.

  7. Locate application performance bottlenecks using the Simple Log Service console.

  8. The application calls the OpenTracing API.

The following query statements are used in the example:

  • Calculate the average latency and the number of requests per minute for the HTTP GET /dispatch operation of the frontend service. 

    process.serviceName: "frontend" and operationName: "HTTP GET /dispatch" |
select from_unixtime( __time__ - __time__ % 60) as time,
truncate(avg(duration)/1000/1000) as avg_duration_ms,
count(1) as count
group by __time__ - __time__ % 60 order by time desc limit 60

  • Compare the execution time of each operation for two traces. 

    TraceID: "Trace1" or TraceID: "Trace2" |
select operationName,
(max(duration)-min(duration))/1000/1000 as duration_diff_ms
group by operationName
order by duration_diff_ms desc

  • Query the IP addresses of traces that have a latency greater than 1.5 s. 

    process.serviceName: "frontend" and operationName: "HTTP GET /dispatch" and duration > 1500000000 |
select "process.tags.ip" as IP,
truncate(avg(duration)/1000/1000) as avg_duration_ms,
count(1) as count
group by "process.tags.ip"
Previous：Console embed parametersNext：Connect to Simple Log Service by using JDBC