使用OpenTelemetry对APISIX进行链路追踪

前提条件

• APISIX版本 ≥ v2.13.0。

• 获取接入点信息

  1. 登录ARMS控制台，在左侧导航栏单击接入中心。

  2. 在服务端应用区域单击OpenTelemetry卡片。

  3. 在弹出的OpenTelemetry面板中选择数据需要上报的地域。

     说明

     初次接入的地域将会自动进行资源初始化。

  4. 选择连接方式和上报方式，然后复制接入点信息。

     • 连接方式：若您的服务部署在阿里云上，且所属地域与选择的接入地域一致，推荐使用阿里云内网方式，否则选择公网方式。

     • 上报方式：根据客户端支持的协议类型选择HTTP或gRPC协议上报数据。

     

方案概览

使用OpenTelemetry对APISIX进行链路追踪，主要分为以下几步：

1. 部署OpenTelemetry Collector：OpenTelemetry Collector是一个开源可观测数据采集器，负责接收、处理和导出来自各种数据源的可观测数据（例如调用链）。

2. 在APISIX中启用OpenTelemetry插件：通过修改APISIX的配置文件启用OpenTelemetry插件，允许OpenTelemetry Collector收集APISIX的可观测数据。

3. 设置APISIX OpenTelemetry插件生效范围：通过APISIX Admin API可以设置OpenTelemetry插件的生效范围，可以全局启用，也可以为指定的路由启用。

4. 查看APISIX调用链：您可以在控制台查看由OpenTelemetry生成的APISIX调用链。

接入步骤

...
plugins:
  ... # 其他已经启用的插件
  - opentelemetry # 启用OpenTelemetry插件

plugin_attr:
  ...
  opentelemetry: # OpenTelemetry相关配置
    resource:
      service.name: ${Service Name} # 应用名称
    collector:
      address: ${OpenTelemetry Collector Address}:4318 # OpenTelemetry Collector OTLP HTTP Receiver地址
      request_timeout: 3
    batch_span_processor: # 配置批量处理
      drop_on_queue_full: false
      max_queue_size: 6
      batch_timeout: 2
      inactive_timeout: 1
      max_export_batch_size: 2

操作示例

准备工作

• 确认已安装Git、Docker和Docker Compose。

• 确认APISIX版本 ≥ v2.13.0。

• 已获取接入点信息。

操作步骤

1. 下载APISIX官方Docker Compose Demo。

   git clone https://github.com/apache/apisix-docker.git
   cd apisix-docker/example

2. 将OpenTelemetry Collector添加到APISIX Docker Compose Demo中。

   1. 在apisix-docker/example文件夹下创建ot_conf文件夹，并创建config.yaml文件。

      说明

      • 请将${HTTP Endpoint}替换为前提条件中获取的HTTP接入点，如：http://tracing-analysis-dc-hz.aliyuncs.com/adapt_xxxxx/api/otlp/traces。

      receivers:
        otlp:
          protocols:
            grpc:
              endpoint: 0.0.0.0:4317
            http:
              cors:
                allowed_origins:
                - http://*
                - https://*
              endpoint: 0.0.0.0:4318
      processors:
        batch:
      
      exporters:
        otlphttp:
          traces_endpoint: '${HTTP Endpoint}'
          tls:
            insecure: true
      
      service:
        pipelines:
          traces:
            receivers: [otlp]
            processors: [batch]
            exporters: [otlphttp]

   2. 修改apisix-docker/example/docker-compose.yml文件，添加OpenTelemetry Collector服务。

      修改后的完整docker-compose.yml文件：

      #
      # Licensed to the Apache Software Foundation (ASF) under one or more
      # contributor license agreements.  See the NOTICE file distributed with
      # this work for additional information regarding copyright ownership.
      # The ASF licenses this file to You under the Apache License, Version 2.0
      # (the "License"); you may not use this file except in compliance with
      # the License.  You may obtain a copy of the License at
      #
      #     http://www.apache.org/licenses/LICENSE-2.0
      #
      # Unless required by applicable law or agreed to in writing, software
      # distributed under the License is distributed on an "AS IS" BASIS,
      # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
      # See the License for the specific language governing permissions and
      # limitations under the License.
      #
      
      version: "3"
      
      services:
        apisix:
          image: apache/apisix:${APISIX_IMAGE_TAG:-3.9.0-debian}
          restart: always
          volumes:
            - ./apisix_conf/config.yaml:/usr/local/apisix/conf/config.yaml:ro
          depends_on:
            - etcd
          ##network_mode: host
          ports:
            - "9180:9180/tcp"
            - "9080:9080/tcp"
            - "9091:9091/tcp"
            - "9443:9443/tcp"
            - "9092:9092/tcp"
          networks:
            apisix:
      
        etcd:
          image: bitnami/etcd:3.5.11
          restart: always
          volumes:
            - etcd_data:/bitnami/etcd
          environment:
            ETCD_ENABLE_V2: "true"
            ALLOW_NONE_AUTHENTICATION: "yes"
            ETCD_ADVERTISE_CLIENT_URLS: "http://etcd:2379"
            ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
          ports:
            - "2379:2379/tcp"
          networks:
            apisix:
      
        web1:
          image: nginx:1.19.0-alpine
          restart: always
          volumes:
            - ./upstream/web1.conf:/etc/nginx/nginx.conf
          ports:
            - "9081:80/tcp"
          environment:
            - NGINX_PORT=80
          networks:
            apisix:
      
        web2:
          image: nginx:1.19.0-alpine
          restart: always
          volumes:
            - ./upstream/web2.conf:/etc/nginx/nginx.conf
          ports:
            - "9082:80/tcp"
          environment:
            - NGINX_PORT=80
          networks:
            apisix:
      
        prometheus:
          image: prom/prometheus:v2.25.0
          restart: always
          volumes:
            - ./prometheus_conf/prometheus.yml:/etc/prometheus/prometheus.yml
          ports:
            - "9090:9090"
          networks:
            apisix:
      
        grafana:
          image: grafana/grafana:7.3.7
          restart: always
          ports:
            - "3000:3000"
          volumes:
            - "./grafana_conf/provisioning:/etc/grafana/provisioning"
            - "./grafana_conf/dashboards:/var/lib/grafana/dashboards"
            - "./grafana_conf/config/grafana.ini:/etc/grafana/grafana.ini"
          networks:
            apisix:
      
        otel-collector:
          image: otel/opentelemetry-collector-contrib:0.105.0
          volumes:
            - ./ot_conf/config.yaml:/etc/otelcol-contrib/config.yaml # 挂载 OpenTelemetry Collector 配置文件
          ports:
            - 4317:4317 # OTLP gRPC receiver
            - 4318:4318 # OTLP http receiver
          networks:
            apisix:
      
      networks:
        apisix:
          driver: bridge
      
      volumes:
        etcd_data:
          driver: local

3. 在APISIX中启用OpenTelemetry插件。

   修改APISIX配置文件apisix-docker/example/apisix_conf/config.yaml，在文件末尾追加以下内容。

   plugins:
     - opentelemetry
   
   plugin_attr:
     prometheus:
       export_addr:
         ip: "0.0.0.0"
         port: 9091
     opentelemetry:
       resource:
         service.name: APISIX
       collector:
         address: docker-apisix-otel-collector-1:4318 # OTLP HTTP Receiver address
         request_timeout: 3
       batch_span_processor:
         drop_on_queue_full: false
         max_queue_size: 6
         batch_timeout: 2
         inactive_timeout: 1
         max_export_batch_size: 2

4. 启动APISIX Docker Compose Demo。

   请在apisix-docker/example目录下执行。

   docker compose -p docker-apisix up -d

5. 全局启用OpenTelemetry插件。

   通过APISIX Admin API设置OpenTelemetry插件在全局生效。

   curl 'http://127.0.0.1:9180/apisix/admin/global_rules/1' \
   -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' \
   -X PUT -d '{
       "plugins": {
           "opentelemetry": {
               "sampler": {
                   "name": "always_on"
               }
           }
       }
   }'
   

6. 创建APISIX路由并测试调用链上报。

   1. 使用APISIX Admin API创建路由。

      curl "http://127.0.0.1:9180/apisix/admin/routes/1" -H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" -X PUT -d '
      {
        "methods": ["GET"],
        "host": "example.com",
        "uri": "/anything/*",
        "upstream": {
          "type": "roundrobin",
          "nodes": {
            "httpbin.org:80": 1
          }
        }
      }'

   2. 访问以下地址，APISIX OpenTelemetry插件会为本次请求生成调用链并上报至可观测链路 OpenTelemetry 版。

      curl -i -X GET "http://127.0.0.1:9080/anything/foo?arg=10" -H "Host: example.com"

      预期输出：

      # curl -i -X GET "http://127.0.0.1:9080/anything/foo?arg=10" -H "Host: example.com"
      HTTP/1.1 200 OK
      Content-Type: application/json
      Content-Length: 501
      Connection: keep-alive
      Date: Wed, 24 Jul 2024 03:26:11 GMT
      Access-Control-Allow-Origin: *
      Access-Control-Allow-Credentials: true
      Server: APISIX/3.9.0
      
      {
        "args": {
          "arg": "10"
        }, 
        "data": "", 
        "files": {}, 
        "form": {}, 
        "headers": {
          "Accept": "*/*", 
          "Host": "example.com", 
          "Traceparent": "00-xxxxxx-xxxx-01", 
          "User-Agent": "curl/7.61.1", 
          "X-Amzn-Trace-Id": "Root=1-xxx-xxxx", 
          "X-Forwarded-Host": "example.com"
        }, 
        "json": null, 
        "method": "GET", 
        "origin": "x.x.x.x, x.x.x.x", 
        "url": "http://example.com/anything/foo?arg=10"
      }

7. 登录可观测链路 OpenTelemetry 版控制台，查看由OpenTelemetry生成的APISIX调用链。

   1. 在应用列表页查看APISIX应用。

   2. 在调用链分析页查看APISIX的调用链。

相关信息

APISIX是一款云原生API网关，由Apache APISIX社区维护。它具有动态、实时、高性能等特点，提供了负载均衡、灰度发布（金丝雀发布）等丰富的流量管理功能。更多信息，请参见Apache APISIX。