当您安装采集器(Beats)时,遇到安装失败、心跳异常的情况时,可参考本文的方法进行排查解决。
操作步骤
排查安装Beats服务的ECS的操作系统是否是Alibaba Cloud Linux (Alinux)、RedHat或CentOS。
排查安装Beats服务的ECS是否与Elasticsearch或Logstash实例处于同一专有网络下。
排查安装Beats服务的ECS是否安装了云助手和Docker。
您可以连接ECS实例,通过以下命令进行验证。
查看云助手服务状态
systemctl status aliyun.service正常情况下,返回结果如下。
[root@VM01 ~]# systemctl status aliyun.service ● aliyun.service - aliyun-assist Loaded: loaded (/etc/systemd/system/aliyun.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2020-08-15 15:38:25 CST; 4 days ago Main PID: 20311 (aliyun-service) CGroup: /system.slice/aliyun.service └─20311 /usr/sbin/aliyun-service Aug 15 15:38:25 VM01 systemd[1]: Stopped aliyun-assist. Aug 15 15:38:25 VM01 systemd[1]: Started aliyun-assist.如果未安装云助手,可参见安装云助手客户端进行安装。
查看Docker状态
systemctl status docker正常情况下,返回如下结果。
[root@VM01 ~]# systemctl status docker ● docker.service - Docker Application Container Engine Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled) Active: active (running) since Wed 2020-08-19 16:45:21 CST; 1min 39s ago Docs: http://docs.docker.com Main PID: 14625 (dockerd-current) CGroup: /system.slice/docker.service ├─14625 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runti... └─14632 /usr/bin/docker-containerd-current -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metric... Aug 19 16:45:20 VM01 dockerd-current[14625]: time="2020-08-19T16:45:20.973350772+08:00" level=warning msg="Docker could...yste... Aug 19 16:45:21 VM01 dockerd-current[14625]: time="2020-08-19T16:45:21.005411644+08:00" level=info msg="Graph migration...cond... Aug 19 16:45:21 VM01 dockerd-current[14625]: time="2020-08-19T16:45:21.006055965+08:00" level=info msg="Loading contain...tar... Aug 19 16:45:21 VM01 dockerd-current[14625]: time="2020-08-19T16:45:21.136526271+08:00" level=info msg="Firewalld runni...fals... Aug 19 16:45:21 VM01 dockerd-current[14625]: time="2020-08-19T16:45:21.239151591+08:00" level=info msg="Default bridge ...dres... Aug 19 16:45:21 VM01 dockerd-current[14625]: time="2020-08-19T16:45:21.280253025+08:00" level=info msg="Loading contain...done... Aug 19 16:45:21 VM01 dockerd-current[14625]: time="2020-08-19T16:45:21.442183479+08:00" level=info msg="Daemon has comp...atio... Aug 19 16:45:21 VM01 dockerd-current[14625]: time="2020-08-19T16:45:21.442217221+08:00" level=info msg="Docker daemon" ...1.13... Aug 19 16:45:21 VM01 systemd[1]: Started Docker Application Container Engine. Aug 19 16:45:21 VM01 dockerd-current[14625]: time="2020-08-19T16:45:21.450179119+08:00" level=info msg="API listen on /...soc... Hint: Some lines were ellipsized, use -l to show in full.如果未安装Docker,可参见部署并使用Docker(Alibaba Cloud Linux 2)进行安装。
检查采集器的YML配置,确认是否已设置如下信息。
- type: log # Change to true to enable this input configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - /var/log/*.log参数
说明
enabled
默认为false,使用时一定要设置为true。
paths
指定日志文件路径,可以采用模糊匹配,例如*.log。
重要paths与配置页面填写的Filebeat文件目录存在区别。Filebeat文件目录是Docker映射的目录,只有映射到采集目录下才能采集到paths指定的文件。建议二者保持一致。
在配置页面指定采集器Output后,YML配置下不能重新指定Output,否则会提示安装错误。
对于采集器YML中默认已经注释掉(#)的参数,需谨慎修改,比如与X-Pack相关的参数设置,否则会导致安装失败。
连接ECS实例,查看/opt/aliyunbeats/目录下,是否生成了对应的Beats实例,并确认是否存在conf、data、logs目录。
[root@PVM1 ~]# cd /opt/aliyumbeats [root@PVM1 aliyumbeats]# ls ct-cn-77uqof2s7rgxxx [root@PVM1 ct-cn-77uqof2s7rgxxx]# ls filebeat [root@PVM1 filebeat]# ls conf data logs您也可以在logs目录下查看Beats日志,方便定位问题。
[root@VM01 logs]# tail -n 2 filebeat 2020-08-19T09:25:52.871Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": "metrics": {"beat":{"cpu":{"system":{"ticks":130,"time":{"ms":1}},"total":{"ticks":390,"time":{"ms":5},"value":390},"user":{"ti cks":260,"time":{"ms":4}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":6},"info":{"ephemeral_id":"f960431f-2849- e-9dc9-f17xxx","uptime":{"ms":15900027}},"memstats":{"gc_next":4285312,"memory_alloc":2157688,"memory_total":367547441}, libbeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"even t":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.17,"15":0.11,"5":0.09,"norm":{"1":0.17,"15":0.11 ":0.09}}}}}} 2020-08-19T09:26:22.872Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": "metrics": {"beat":{"cpu":{"system":{"ticks":130,"time":{"ms":3}},"total":{"ticks":390,"time":{"ms":4},"value":390},"user":{"ti cks":260,"time":{"ms":1}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":6},"info":{"ephemeral_id":"f960431f-2849- e-9dc9-f17xxx","uptime":{"ms":16200027}},"memstats":{"gc_next":4285312,"memory_alloc":2449992,"memory_total":370470483}, libbeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"even t":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.11,"15":0.11,"5":0.08,"norm":{"1":0.11,"15":0.11 ":0.08}}}}}}查看Beats所在容器的运行状态,并分析日志定位问题。
查看Docker容器中的运行状态。
docker ps -a | grep filebeat正常情况下,返回结果如下。
[root@VM01 logs]# docker ps -a | grep filebeat 922d7f8xxx registry-vpc.cn-hangzhou.aliyuncs.com/elasticsearch/aliyun-elasticsearch-beats:filebeat-6.8.5 "/usr/local/bin/do..." 33 minutes ago Up 33 minutes ct-cn-77uqof2s7rg ___FILEBEAT当容器处于exited状态时,查看容器输出日志。
docker logs -f 容器id