使用Prometheus Exporter监控Kafka客户端

更新时间:
复制 MD 格式

Kafka客户端默认提供了丰富的监控指标,可以帮助您及时发现一些非预期的问题,比如网络连接、发送延迟、消费堆积等问题。本文为您介绍如何将JMX Exporter采集到的数据接入到阿里云可观测监控 Prometheus 版,并在可观测可视化 Grafana 版进行可视化查看。

前提条件

操作步骤

步骤一:配置客户端

  1. 登录ECS实例后,执行以下命令,下载JMX Exporter。

    wget https://arms-public.oss-cn-shanghai.aliyuncs.com/Prom/agent/kafka-jmx_prometheus_javaagent-1.18.1.jar
  2. 执行以下命令,启动客户端。

    说明

    命令中的路径和jar包需要按照实际情况进行替换。

    java -javaagent:/path/to/kafka-jmx_prometheus_javaagent-1.18.1.jar=5556 -jar your-kafka-client.jar
  3. 执行以下命令,查看导出的指标。

    curl http://localhost:5556/metrics
    [root@xxx ~]# curl http://localhost:5556/metrics
    # HELP jvm_threads_current Current thread count of a JVM
    # TYPE jvm_threads_current gauge
    jvm_threads_current 26.0
    # HELP jvm_threads_daemon Daemon thread count of a JVM
    # TYPE jvm_threads_daemon gauge
    jvm_threads_daemon 21.0
    # HELP jvm_threads_peak Peak thread count of a JVM
    # TYPE jvm_threads_peak gauge
    jvm_threads_peak 26.0
    # HELP jvm_threads_started_total Started thread count of a JVM
    # TYPE jvm_threads_started_total counter
    jvm_threads_started_total 30.0
    # HELP jvm_threads_deadlocked Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers
    # TYPE jvm_threads_deadlocked gauge
    jvm_threads_deadlocked 0.0
    # HELP jvm_threads_deadlocked_monitor Cycles of JVM-threads that are in deadlock waiting to acquire object monitors
    # TYPE jvm_threads_deadlocked_monitor gauge
    jvm_threads_deadlocked_monitor 0.0
    # HELP jvm_threads_state Current count of threads by state
    # TYPE jvm_threads_state gauge
    jvm_threads_state{state="NEW",} 0.0
    jvm_threads_state{state="TERMINATED",} 0.0
    jvm_threads_state{state="RUNNABLE",} 8.0
    jvm_threads_state{state="BLOCKED",} 0.0
    jvm_threads_state{state="WAITING",} 13.0
    jvm_threads_state{state="TIMED_WAITING",} 5.0
    jvm_threads_state{state="UNKNOWN",} 0.0
    # HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
    # TYPE process_cpu_seconds_total counter
    process_cpu_seconds_total 69.13
    # HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
    # TYPE process_start_time_seconds gauge
    process_start_time_seconds 1.733732603482E9
    # HELP process_open_fds Number of open file descriptors.
    # TYPE process_open_fds gauge
    process_open_fds 44.0
    # HELP process_max_fds Maximum number of open file descriptors.
    # TYPE process_max_fds gauge
    process_max_fds 65535.0
    # HELP process_virtual_memory_bytes Virtual memory size in bytes.
    # TYPE process_virtual_memory_bytes gauge
    process_virtual_memory_bytes 5.792423936E9
    # HELP process_resident_memory_bytes Resident memory size in bytes.
    # TYPE process_resident_memory_bytes gauge
    process_resident_memory_bytes 9.24934144E8
    # HELP jmx_config_reload_success_total Number of times configuration have successfully been reloaded.
    # TYPE jmx_config_reload_success_total counter
    jmx_config_reload_success_total 0.0
    # HELP jvm_gc_collection_seconds Time spent in a given JVM garbage collector in seconds.
    # TYPE jvm_gc_collection_seconds summary
    jvm_gc_collection_seconds_count{gc="PS Scavenge",} 25.0
    jvm_gc_collection_seconds_sum{gc="PS Scavenge",} 0.294
    jvm_gc_collection_seconds_count{gc="PS MarkSweep",} 2.0
    jvm_gc_collection_seconds_sum{gc="PS MarkSweep",} 0.107
    # HELP jvm_memory_pool_allocated_bytes_total Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
    

步骤二:配置Prometheus

  1. 登录ARMS控制台,在左侧导航栏单击接入中心

  2. 接入中心页面,单击基础设施页签,在基础设施区域选择自定义指标采集

  3. 自定义指标采集面板的开始接入页签,填写相应的信息,然后单击确定

    配置信息如下:

    1. 选择所属环境类型为ECS(VPC)(监控目标在 ECS 主机上运行)。

    2. 选择 VPC。

    3. 配置信息中,服务发现方式选择自动发现,主机服务发现方式选择实例ID,填写 ECS 实例 ID,服务端口填写5556,指标采集路径为/metrics,采集间隔为 15 秒。

    4. 单击确定

  4. 接入完成后,在左侧导航栏单击接入管理

  5. 接入管理页面的已接入环境页签中,选择ECS环境

  6. ECS环境列表中,单击目标环境名称进入ECS环境详情页面。

  7. 自监控页签,即可查看是否成功接入(数据有延迟,需要稍等一会)。

    自监控页签下,左侧导航选择Targets,主区域显示采集任务(如custom-job-kafka-prometheus)的 Target 状态。当 State 列显示UP时,表示数据采集正常,接入成功。

步骤三:配置Grafana大盘

  1. 登录ARMS控制台,在左侧导航栏选择Grafana服务 > 工作区管理

  2. 工作区管理页面,单击已创建好的工作区ID,在工作区信息页面查看详细信息。

    说明

    云服务集成区域中列表里的目标服务为未集成状态,则需要单击其操作列的集成,进行集成操作。

  3. 基本信息区域,单击公网地址后的链接进入Grafana。

    说明

    Grafana登录页面,您可以使用Admin账号和创建工作区时设置的Admin密码登录Grafana,也可以单击Sign in with Alibaba Cloud直接使用当前阿里云账号登录Grafana。

  4. Grafana首页,单击页面左上角的image图标。

  5. Grafana左侧导航栏,单击仪表板

  6. 仪表板页面选择新建 > 导入,在Import dashboard页面使用Grafana仪表板配置.json导入仪表盘。

  7. 导入成功后,监控数据大盘展示如下:

    image

指标信息