文档

使用OpenTelemetry对APISIX进行链路追踪

更新时间:

APISIX OpenTelemetry插件支持收集APISIX的调用链数据并上报至OpenTelemetry Collector,再由OpenTelemetry Collector转发上报至可观测链路 OpenTelemetry 版。APISIX OpenTelemetry插件目前仅支持通过HTTP方式上报调用链数据至OpenTelemetry Collector,不支持gRPC方式上报。

前提条件

  • APISIX版本 ≥ v2.13.0。

  • 获取接入点信息

    1. 登录ARMS控制台,在左侧导航栏单击接入中心

    2. 服务端应用区域单击OpenTelemetry卡片。

    3. 在弹出的OpenTelemetry面板中选择数据需要上报的地域。

      说明

      初次接入的地域将会自动进行资源初始化。

    4. 选择连接方式上报方式,然后复制接入点信息。

      • 连接方式:若您的服务部署在阿里云上,且所属地域与选择的接入地域一致,推荐使用阿里云内网方式,否则选择公网方式。

      • 上报方式:根据客户端支持的协议类型选择HTTP或gRPC协议上报数据。

      image.png

方案概览

image

使用OpenTelemetry对APISIX进行链路追踪,主要分为以下几步:

  1. 部署OpenTelemetry Collector:OpenTelemetry Collector是一个开源可观测数据采集器,负责接收、处理和导出来自各种数据源的可观测数据(例如调用链)。

  2. 在APISIX中启用OpenTelemetry插件:通过修改APISIX的配置文件启用OpenTelemetry插件,允许OpenTelemetry Collector收集APISIX的可观测数据。

  3. 设置APISIX OpenTelemetry插件生效范围:通过APISIX Admin API可以设置OpenTelemetry插件的生效范围,可以全局启用,也可以为指定的路由启用。

  4. 查看APISIX调用链:您可以在控制台查看由OpenTelemetry生成的APISIX调用链。

接入步骤

一. 部署OpenTelemetry Collector

下文以Docker方式为例,介绍如何部署OpenTelemetry Collector。更多部署方式,请参见下载并部署OpenTelemetry Collector

  1. 创建opentelemetry-config.yaml文件,并将下面的内容拷贝至文件。

    该文件用于定义和配置OpenTelemetry Collector的行为和功能,包括如何接收、处理和导出数据。

    说明

    请将${HTTP Endpoint}替换为前提条件中获取的HTTP接入点,如:http://tracing-analysis-dc-hz.aliyuncs.com/adapt_xxxxx/api/otlp/traces

    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            cors:
              allowed_origins:
              - http://*
              - https://*
            endpoint: 0.0.0.0:4318 # OTLP HTTP Receiver
    processors:
      batch:
    
    exporters:
      otlphttp:
        traces_endpoint: '${HTTP Endpoint}'
        tls:
          insecure: true
    
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlphttp]
  2. 启动OpenTelemetry Collector。

    docker run -v $(pwd)/opentelemetry-config.yaml:/etc/otelcol-contrib/config.yaml otel/opentelemetry-collector-contrib:0.105.0

二. 在APISIX中启用OpenTelemetry插件

您需要在APISIX的config.yaml配置文件中启用opentelemetry插件并修改Collector配置。

说明
  • 请将${Service Name}替换为应用名称,例如APISIX,该应用名称会显示在可观测链路 OpenTelemetry 版的应用列表中。

  • 请将${OpenTelemetry Collector Address}替换为Collector的访问地址,如127.0.0.1

  • 关于OpenTelemetry插件的更多上报配置,请参见APISIX Opentelemetry 插件上报配置

...
plugins:
  ... # 其他已经启用的插件
  - opentelemetry # 启用OpenTelemetry插件

plugin_attr:
  ...
  opentelemetry: # OpenTelemetry相关配置
    resource:
      service.name: ${Service Name} # 应用名称
    collector:
      address: ${OpenTelemetry Collector Address}:4318 # OpenTelemetry Collector OTLP HTTP Receiver地址
      request_timeout: 3
    batch_span_processor: # 配置批量处理
      drop_on_queue_full: false
      max_queue_size: 6
      batch_timeout: 2
      inactive_timeout: 1
      max_export_batch_size: 2

三. 设置APISIX OpenTelemetry插件生效范围

  • 全局启用OpenTelemetry插件。

    说明
    curl 'http://127.0.0.1:9080/apisix/admin/global_rules/1' \
    -H 'X-API-KEY:  edd1c9f034335f136f87ad84b625c8f1' \
    -X PUT -d '{
      "plugins": {
          "opentelemetry": {
              "sampler": {
                  "name": "always_on"
              }
          }
      }
    }'
  • 仅在指定路由启用OpenTelemetry插件。

    curl http://127.0.0.1:9080/apisix/admin/routes/1 \
    -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' \
    -X PUT -d '
    {
      "uri": "/get",
      "plugins": {
          "opentelemetry": {
              "sampler": {
                  "name": "always_on"
              }
          }
      },
      "upstream": {
          "type": "roundrobin",
          "nodes": {
              "httpbin.org:80": 1
          }
      }
    }'

四. 查看APISIX调用链

完成以上配置后,您可通过APISIX创建路由并进行访问。然后可以登录可观测链路 OpenTelemetry 版控制台,查看由OpenTelemetry生成的APISIX调用链。

  1. 在应用列表页查看APISIX应用。图片 1.png

  2. 在调用链分析页查看APISIX的调用链。image (3).png

操作示例

准备工作

操作步骤

  1. 下载APISIX官方Docker Compose Demo。

    git clone https://github.com/apache/apisix-docker.git
    cd apisix-docker/example
  2. 将OpenTelemetry Collector添加到APISIX Docker Compose Demo中。

    1. apisix-docker/example文件夹下创建ot_conf文件夹,并创建config.yaml文件。

      说明
      • 请将${HTTP Endpoint}替换为前提条件中获取的HTTP接入点,如:http://tracing-analysis-dc-hz.aliyuncs.com/adapt_xxxxx/api/otlp/traces

      receivers:
        otlp:
          protocols:
            grpc:
              endpoint: 0.0.0.0:4317
            http:
              cors:
                allowed_origins:
                - http://*
                - https://*
              endpoint: 0.0.0.0:4318
      processors:
        batch:
      
      exporters:
        otlphttp:
          traces_endpoint: '${HTTP Endpoint}'
          tls:
            insecure: true
      
      service:
        pipelines:
          traces:
            receivers: [otlp]
            processors: [batch]
            exporters: [otlphttp]
    2. 修改apisix-docker/example/docker-compose.yml文件,添加OpenTelemetry Collector服务。

      修改后的完整docker-compose.yml文件:

      #
      # Licensed to the Apache Software Foundation (ASF) under one or more
      # contributor license agreements.  See the NOTICE file distributed with
      # this work for additional information regarding copyright ownership.
      # The ASF licenses this file to You under the Apache License, Version 2.0
      # (the "License"); you may not use this file except in compliance with
      # the License.  You may obtain a copy of the License at
      #
      #     http://www.apache.org/licenses/LICENSE-2.0
      #
      # Unless required by applicable law or agreed to in writing, software
      # distributed under the License is distributed on an "AS IS" BASIS,
      # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
      # See the License for the specific language governing permissions and
      # limitations under the License.
      #
      
      version: "3"
      
      services:
        apisix:
          image: apache/apisix:${APISIX_IMAGE_TAG:-3.9.0-debian}
          restart: always
          volumes:
            - ./apisix_conf/config.yaml:/usr/local/apisix/conf/config.yaml:ro
          depends_on:
            - etcd
          ##network_mode: host
          ports:
            - "9180:9180/tcp"
            - "9080:9080/tcp"
            - "9091:9091/tcp"
            - "9443:9443/tcp"
            - "9092:9092/tcp"
          networks:
            apisix:
      
        etcd:
          image: bitnami/etcd:3.5.11
          restart: always
          volumes:
            - etcd_data:/bitnami/etcd
          environment:
            ETCD_ENABLE_V2: "true"
            ALLOW_NONE_AUTHENTICATION: "yes"
            ETCD_ADVERTISE_CLIENT_URLS: "http://etcd:2379"
            ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
          ports:
            - "2379:2379/tcp"
          networks:
            apisix:
      
        web1:
          image: nginx:1.19.0-alpine
          restart: always
          volumes:
            - ./upstream/web1.conf:/etc/nginx/nginx.conf
          ports:
            - "9081:80/tcp"
          environment:
            - NGINX_PORT=80
          networks:
            apisix:
      
        web2:
          image: nginx:1.19.0-alpine
          restart: always
          volumes:
            - ./upstream/web2.conf:/etc/nginx/nginx.conf
          ports:
            - "9082:80/tcp"
          environment:
            - NGINX_PORT=80
          networks:
            apisix:
      
        prometheus:
          image: prom/prometheus:v2.25.0
          restart: always
          volumes:
            - ./prometheus_conf/prometheus.yml:/etc/prometheus/prometheus.yml
          ports:
            - "9090:9090"
          networks:
            apisix:
      
        grafana:
          image: grafana/grafana:7.3.7
          restart: always
          ports:
            - "3000:3000"
          volumes:
            - "./grafana_conf/provisioning:/etc/grafana/provisioning"
            - "./grafana_conf/dashboards:/var/lib/grafana/dashboards"
            - "./grafana_conf/config/grafana.ini:/etc/grafana/grafana.ini"
          networks:
            apisix:
      
        otel-collector:
          image: otel/opentelemetry-collector-contrib:0.105.0
          volumes:
            - ./ot_conf/config.yaml:/etc/otelcol-contrib/config.yaml # 挂载 OpenTelemetry Collector 配置文件
          ports:
            - 4317:4317 # OTLP gRPC receiver
            - 4318:4318 # OTLP http receiver
          networks:
            apisix:
      
      networks:
        apisix:
          driver: bridge
      
      volumes:
        etcd_data:
          driver: local
  3. 在APISIX中启用OpenTelemetry插件。

    修改APISIX配置文件apisix-docker/example/apisix_conf/config.yaml,在文件末尾追加以下内容。

    plugins:
      - opentelemetry
    
    plugin_attr:
      prometheus:
        export_addr:
          ip: "0.0.0.0"
          port: 9091
      opentelemetry:
        resource:
          service.name: APISIX
        collector:
          address: docker-apisix-otel-collector-1:4318 # OTLP HTTP Receiver address
          request_timeout: 3
        batch_span_processor:
          drop_on_queue_full: false
          max_queue_size: 6
          batch_timeout: 2
          inactive_timeout: 1
          max_export_batch_size: 2
  4. 启动APISIX Docker Compose Demo。

    请在apisix-docker/example目录下执行。

    docker compose -p docker-apisix up -d
  5. 全局启用OpenTelemetry插件。

    通过APISIX Admin API设置OpenTelemetry插件在全局生效。

    curl 'http://127.0.0.1:9180/apisix/admin/global_rules/1' \
    -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' \
    -X PUT -d '{
        "plugins": {
            "opentelemetry": {
                "sampler": {
                    "name": "always_on"
                }
            }
        }
    }'
    
  6. 创建APISIX路由并测试调用链上报。

    1. 使用APISIX Admin API创建路由。

      curl "http://127.0.0.1:9180/apisix/admin/routes/1" -H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" -X PUT -d '
      {
        "methods": ["GET"],
        "host": "example.com",
        "uri": "/anything/*",
        "upstream": {
          "type": "roundrobin",
          "nodes": {
            "httpbin.org:80": 1
          }
        }
      }'
    2. 访问以下地址,APISIX OpenTelemetry插件会为本次请求生成调用链并上报至可观测链路 OpenTelemetry 版。

      curl -i -X GET "http://127.0.0.1:9080/anything/foo?arg=10" -H "Host: example.com"

      预期输出:

      # curl -i -X GET "http://127.0.0.1:9080/anything/foo?arg=10" -H "Host: example.com"
      HTTP/1.1 200 OK
      Content-Type: application/json
      Content-Length: 501
      Connection: keep-alive
      Date: Wed, 24 Jul 2024 03:26:11 GMT
      Access-Control-Allow-Origin: *
      Access-Control-Allow-Credentials: true
      Server: APISIX/3.9.0
      
      {
        "args": {
          "arg": "10"
        }, 
        "data": "", 
        "files": {}, 
        "form": {}, 
        "headers": {
          "Accept": "*/*", 
          "Host": "example.com", 
          "Traceparent": "00-xxxxxx-xxxx-01", 
          "User-Agent": "curl/7.61.1", 
          "X-Amzn-Trace-Id": "Root=1-xxx-xxxx", 
          "X-Forwarded-Host": "example.com"
        }, 
        "json": null, 
        "method": "GET", 
        "origin": "x.x.x.x, x.x.x.x", 
        "url": "http://example.com/anything/foo?arg=10"
      }

  7. 登录可观测链路 OpenTelemetry 版控制台,查看由OpenTelemetry生成的APISIX调用链。

    1. 在应用列表页查看APISIX应用。图片 1.png

    2. 在调用链分析页查看APISIX的调用链。image (3).png

相关信息

APISIX是一款云原生API网关,由Apache APISIX社区维护。它具有动态、实时、高性能等特点,提供了负载均衡、灰度发布(金丝雀发布)等丰富的流量管理功能。更多信息,请参见Apache APISIX