Beats安装失败的排查与解决方法

更新时间:
复制 MD 格式

当您安装采集器(Beats)时,遇到安装失败、心跳异常的情况时,可参考本文的方法进行排查解决。

操作步骤

  1. 排查安装Beats服务的ECS的操作系统是否是Alibaba Cloud Linux (Alinux)、RedHatCentOS。

  2. 排查安装Beats服务的ECS是否与ElasticsearchLogstash实例处于同一专有网络下。

  3. 排查安装Beats服务的ECS是否安装了云助手和Docker。

    您可以连接ECS实例,通过以下命令进行验证。

    • 查看云助手服务状态

      systemctl status aliyun.service

      正常情况下,返回结果如下。

      [root@VM01 ~]# systemctl status aliyun.service
      ● aliyun.service - aliyun-assist
         Loaded: loaded (/etc/systemd/system/aliyun.service; enabled; vendor preset: disabled)
         Active: active (running) since Sat 2020-08-15 15:38:25 CST; 4 days ago
       Main PID: 20311 (aliyun-service)
         CGroup: /system.slice/aliyun.service
                 └─20311 /usr/sbin/aliyun-service
      
      Aug 15 15:38:25 VM01 systemd[1]: Stopped aliyun-assist.
      Aug 15 15:38:25 VM01 systemd[1]: Started aliyun-assist.

      如果未安装云助手,可参见安装云助手客户端进行安装。

    • 查看Docker状态

      systemctl status docker

      正常情况下,返回如下结果。

      [root@VM01 ~]# systemctl status docker
      ● docker.service - Docker Application Container Engine
         Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
         Active: active (running) since Wed 2020-08-19 16:45:21 CST; 1min 39s ago
           Docs: http://docs.docker.com
       Main PID: 14625 (dockerd-current)
         CGroup: /system.slice/docker.service
                 ├─14625 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runti...
                 └─14632 /usr/bin/docker-containerd-current -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metric...
      
      Aug 19 16:45:20 VM01 dockerd-current[14625]: time="2020-08-19T16:45:20.973350772+08:00" level=warning msg="Docker could...yste...
      Aug 19 16:45:21 VM01 dockerd-current[14625]: time="2020-08-19T16:45:21.005411644+08:00" level=info msg="Graph migration...cond...
      Aug 19 16:45:21 VM01 dockerd-current[14625]: time="2020-08-19T16:45:21.006055965+08:00" level=info msg="Loading contain...tar...
      Aug 19 16:45:21 VM01 dockerd-current[14625]: time="2020-08-19T16:45:21.136526271+08:00" level=info msg="Firewalld runni...fals...
      Aug 19 16:45:21 VM01 dockerd-current[14625]: time="2020-08-19T16:45:21.239151591+08:00" level=info msg="Default bridge ...dres...
      Aug 19 16:45:21 VM01 dockerd-current[14625]: time="2020-08-19T16:45:21.280253025+08:00" level=info msg="Loading contain...done...
      Aug 19 16:45:21 VM01 dockerd-current[14625]: time="2020-08-19T16:45:21.442183479+08:00" level=info msg="Daemon has comp...atio...
      Aug 19 16:45:21 VM01 dockerd-current[14625]: time="2020-08-19T16:45:21.442217221+08:00" level=info msg="Docker daemon" ...1.13...
      Aug 19 16:45:21 VM01 systemd[1]: Started Docker Application Container Engine.
      Aug 19 16:45:21 VM01 dockerd-current[14625]: time="2020-08-19T16:45:21.450179119+08:00" level=info msg="API listen on /...soc...
      Hint: Some lines were ellipsized, use -l to show in full.

      如果未安装Docker,可参见部署并使用Docker(Alibaba Cloud Linux 2)进行安装。

  4. 检查采集器的YML配置,确认是否已设置如下信息。

    - type: log
      # Change to true to enable this input configuration.
      enabled: true              
      # Paths that should be crawled and fetched. Glob based paths.
      paths:
        - /var/log/*.log

    参数

    说明

    enabled

    默认为false,使用时一定要设置为true。

    paths

    指定日志文件路径,可以采用模糊匹配,例如*.log

    重要
    • paths与配置页面填写的Filebeat文件目录存在区别。Filebeat文件目录是Docker映射的目录,只有映射到采集目录下才能采集到paths指定的文件。建议二者保持一致。

    • 在配置页面指定采集器Output后,YML配置下不能重新指定Output,否则会提示安装错误。

    • 对于采集器YML中默认已经注释掉(#)的参数,需谨慎修改,比如与X-Pack相关的参数设置,否则会导致安装失败。

  5. 连接ECS实例,查看/opt/aliyunbeats/目录下,是否生成了对应的Beats实例,并确认是否存在conf、data、logs目录。

    [root@PVM1 ~]# cd /opt/aliyumbeats
    [root@PVM1 aliyumbeats]# ls
    ct-cn-77uqof2s7rgxxx
    [root@PVM1 ct-cn-77uqof2s7rgxxx]# ls
    filebeat
    [root@PVM1 filebeat]# ls
    conf  data  logs

    您也可以在logs目录下查看Beats日志,方便定位问题。

    [root@VM01 logs]# tail -n 2 filebeat
    2020-08-19T09:25:52.871Z	INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s	{"monitoring":
    "metrics": {"beat":{"cpu":{"system":{"ticks":130,"time":{"ms":1}},"total":{"ticks":390,"time":{"ms":5},"value":390},"user":{"ti
    cks":260,"time":{"ms":4}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":6},"info":{"ephemeral_id":"f960431f-2849-
    e-9dc9-f17xxx","uptime":{"ms":15900027}},"memstats":{"gc_next":4285312,"memory_alloc":2157688,"memory_total":367547441},
    libbeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"even
    t":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.17,"15":0.11,"5":0.09,"norm":{"1":0.17,"15":0.11
    ":0.09}}}}}}
    2020-08-19T09:26:22.872Z	INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s	{"monitoring":
    "metrics": {"beat":{"cpu":{"system":{"ticks":130,"time":{"ms":3}},"total":{"ticks":390,"time":{"ms":4},"value":390},"user":{"ti
    cks":260,"time":{"ms":1}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":6},"info":{"ephemeral_id":"f960431f-2849-
    e-9dc9-f17xxx","uptime":{"ms":16200027}},"memstats":{"gc_next":4285312,"memory_alloc":2449992,"memory_total":370470483},
    libbeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"even
    t":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.11,"15":0.11,"5":0.08,"norm":{"1":0.11,"15":0.11
    ":0.08}}}}}}
  6. 查看Beats所在容器的运行状态,并分析日志定位问题。

    1. 查看Docker容器中的运行状态。

      docker ps -a | grep filebeat

      正常情况下,返回结果如下。

      [root@VM01 logs]# docker ps -a | grep filebeat
      922d7f8xxx   registry-vpc.cn-hangzhou.aliyuncs.com/elasticsearch/aliyun-elasticsearch-beats:filebeat-6.8.5   "/usr/local/bin/do..."   33 minutes ago   Up 33 minutes   ct-cn-77uqof2s7rg   ___FILEBEAT
    2. 当容器处于exited状态时,查看容器输出日志。

      docker logs -f 容器id