部署在容器计算服务 ACS(Container Compute Service)的应用除了使用阿里云日志服务(SLS)采集日志之外,还可以构建自己的日志采集系统,比如:EFK(ElasticSearch,FileBeat,Kibana)。由于ACS不支持部署DaemonSet组件,因此日志采集Agent FileBeat需要使用Sidecar的形态进行日志采集,本文主要介绍如何在ACS上部署自定义的日志采集Agent。
前提条件
已创建ACS集群。具体操作,请参见创建ACS集群。
背景信息
常见的日志采集方案都会有一个Agent用于采集业务日志,即在每个Pod实例中,除业务容器外,运行一个Sidecar容器作为日志Agent(例如:FileBeat),用于采集业务容器产生的日志。
Sidecar模式日志采集Agent必须和业务容器共享日志目录,业务容器将日志写入到共享目录中,Logtail通过监控共享目录中日志文件的变化并采集日志。
采集的日志可以分为以下两种:
标准输出
采集标准输出依赖于Medium为Stdout的EmptyDir卷。创建Pod时可以将该卷挂载到Sidecar容器上,Sidecar可以直接以文件的方式访问ACS基础组件收集的标准输出日志。
说明每个Pod最多可以声明1个Medium为Stdout的EmptyDir卷,并需要挂载到至少1个容器内。
采集标准输出会带来一定的性能开销和存储成本。
标准输出日志已配置自动轮转,具体规则参考日志轮转。其中,每个日志文件到达10MiB之后将启动轮转,并且每个容器允许的最大文件数为5。
文本文件
采集文本文件可以通过Pod内共享的volume实现。Pod内同一个volume可以挂载到多个容器,Sidecar可以直接收集业务容器输出到对应volume内的文本文件。
功能介绍
容器启动和退出顺序和Job类Pod
Sidecar容器常需要如下两个诉求:
Sidecar容器需要在业务容器前启动,在业务容器后推出。
对于Job类Pod而言,Sidecar容器需要在业务容器退出后主动退出。
在ACS场景下,你可以在Sidecar容器上面设置环境变量__IS_SIDECAR__="true"
来实现上述诉求。更多信息,请参见配置Sidecar容器启停顺序。
Sidecar模式采集容器文件日志
Sidecar容器如果是采集业务容器的文件日志,可以通过共享volume的方式。更多信息,请参见使用Sidecar模式采集业务容器文件日志。
Sidecar模式采集容器标准输出
在ACS场景下,可以通过配置volumes.emptyDir.medium=Stdout
来实现采集容器标准输出的效果。详细配置,请参见场景一。
场景一:采集应用文件日志和标准输出
使用以下内容,创建一个包含Filebeat Sidecar容器的Deployment。
您可以用相同的方式配置StatefulSet。
执行以下命令,获取Pod信息。
kubectl get pods -l app=nginx
NAME READY STATUS RESTARTS AGE nginx-85776xxxxx-xxxxx 2/2 Running 0 13m
查看日志。
执行以下命令,进入到filebeat容器。
kubectl exec -it deploy/nginx -c filebeat -- /bin/bash
在filebeat容器中查看应用日志。
cat /var/log/error.log
预期输出:
2024/10/30 10:38:28 [notice] 1#1: using the "epoll" event method 2024/10/30 10:38:28 [notice] 1#1: nginx/1.21.5 2024/10/30 10:38:28 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6) 2024/10/30 10:38:28 [notice] 1#1: OS: Linux 5.10.134-17.0.2.lifsea8.x86_64 2024/10/30 10:38:28 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576 2024/10/30 10:38:28 [notice] 1#1: start worker processes 2024/10/30 10:38:28 [notice] 1#1: start worker process 32
在filebeat容器中查看标准输出。
cat /stdout/nginx/0.log
预期输出:
2024-10-30T18:38:28.37305498+08:00 stdout F /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration 2024-10-30T18:38:28.373072927+08:00 stdout F /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/ 2024-10-30T18:38:28.379999685+08:00 stdout F /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh 2024-10-30T18:38:28.391562633+08:00 stdout F 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf 2024-10-30T18:38:28.405538758+08:00 stdout F 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf 2024-10-30T18:38:28.405657957+08:00 stdout F /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh 2024-10-30T18:38:28.409014952+08:00 stdout F /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh 2024-10-30T18:38:28.410397615+08:00 stdout F /docker-entrypoint.sh: Configuration complete; ready for start up
场景二:采集任务型应用日志
使用以下内容,部署基于busybox构建打印标准输出的任务。此任务执行100s之后会自动退出。
以下两种YAML分别展示了不同的Sidecar容器配置方式,部署后的任务是完全相同的。
K8s 1.28及以下版本
使用以下内容部署任务busybox-stdout。
进入到filebeat容器,可以看到已经共享了nginx容器的文件日志和标准输出。
% kubectl exec -it busybox-stdout-4fwnx -c filebeat -- /bin/bash $ cat /stdout/busybox/0.log 2024-11-07T18:46:06.77002301+08:00 stdout F Log message 1 2024-11-07T18:46:07.77143368+08:00 stdout F Log message 2 2024-11-07T18:46:08.772840307+08:00 stdout F Log message 3 2024-11-07T18:46:09.774047629+08:00 stdout F Log message 4 2024-11-07T18:46:10.776122173+08:00 stdout F Log message 5 2024-11-07T18:46:11.778271688+08:00 stdout F Log message 6
K8s 1.29及以上版本
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
data:
filebeat.yml: |
filebeat.inputs:
- type: log
paths:
- /stdout/*
output.elasticsearch:
host: '${NODE_NAME}'
hosts: '["https://${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}"]'
username: '${ELASTICSEARCH_USERNAME}'
password: '${ELASTICSEARCH_PASSWORD}'
protocol: https
ssl.certificate_authorities: [ "/usr/share/filebeat/certs/ca.crt" ]
---
apiVersion: batch/v1
kind: Job
metadata:
name: busybox-stdout
spec:
template:
spec:
initContainers:
- args:
- -e
- -E
- http.enabled=true
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: ELASTICSEARCH_USERNAME
valueFrom:
secretKeyRef:
key: username
name: elasticsearch-master-credentials
- name: ELASTICSEARCH_PASSWORD
valueFrom:
secretKeyRef:
key: password
name: elasticsearch-master-credentials
image: mirrors-ssl.aliyuncs.com/docker.elastic.co/beats/filebeat:8.5.1
imagePullPolicy: Always
restartPolicy: Always # 声明此容器为sidecar
name: filebeat
resources:
limits:
cpu: 250m
ephemeral-storage: 30Gi
memory: 512Mi
requests:
cpu: 250m
ephemeral-storage: 30Gi
memory: 512Mi
volumeMounts:
- mountPath: /stdout
name: stdout-log
readOnly: true
- mountPath: /usr/share/filebeat/certs/
name: elasticsearch-master-certs
- mountPath: /usr/share/filebeat/filebeat.yml
name: filebeat-config
readOnly: true
subPath: filebeat.yml
containers:
- name: busybox
image: mirrors-ssl.aliyuncs.com/busybox:latest
command: [ "sh", "-c" ]
args:
- |
for i in $(seq 1 100); do
echo "Log message $i"
sleep 1
done
volumes:
## 采集标准输出日志
- emptyDir:
medium: Stdout
name: stdout-log
- name: elasticsearch-master-certs
secret:
secretName: elasticsearch-master-certs
- configMap:
name: filebeat-config
name: filebeat-config
restartPolicy: OnFailure
# 下面的配置只是单纯为了本文档的示例
# 在真实环境中需要替换为,部署EFK的时候自动生成的配置
---
apiVersion: v1
data:
ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURJakNDQWdxZ0F3SUJBZ0lSQUl0SDZxR2YzRG9VNFBuVWRJOUdITlV3RFFZSktvWklodmNOQVFFTEJRQXcKR3pFWk1CY0dBMVVFQXhNUVpXeGhjM1JwWTNObFlYSmphQzFqWVRBZUZ3MHlOREV3TWprd056RTNNamxhRncweQpOVEV3TWprd056RTNNamxhTUJzeEdUQVhCZ05WQkFNVEVHVnNZWE4wYVdOelpXRnlZMmd0WTJFd2dnRWlNQTBHCkNTcUdTSWIzRFFFQkFRVUFBNElCRHdBd2dnRUtBb0lCQVFET1Zxc0svVXAvWTVNRVY5T3hzTUk2TTZMTFhYMGsKNFBGSjE0SklVNStUVnBRVVNhVmx3N0REeGtJaUQ3RDVHZ3I3Snh4WHV0cFNjVlo0QWN0UzNrNFJvV3lqdzg0cApoYW4wY3JZL2VaQmJlWjVFUUhCSXprU0ZhMWd4bkpUcVErSSsvR3lKSlNHNkQyR21UVHRqRXZ2R2pWL1loSDNHCk1DMnRadVNXN1hPYWZBKzFqWUNkVFpIZkNpeDdBZURVNU0zcVplNzR4MjhTeitDNkM4WUFCQ0ZSTnJsVGNFQW8KaGQ5WGNnellPUGdJY2VZSUJWb25DTDdzVWFPZGVKa1hrbmdBR2ZzWjB6RnJhMm1qZGZtcHVIaWZFM21LbUZ1agpydGhXVFZTdE9oZGtIUnZTck52NDZaSFdtYlErNXZCb1RiODllTFZuNTNwbzhmSkJIWWpHZ24zdEFnTUJBQUdqCllUQmZNQTRHQTFVZER3RUIvd1FFQXdJQ3BEQWRCZ05WSFNVRUZqQVVCZ2dyQmdFRkJRY0RBUVlJS3dZQkJRVUgKQXdJd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVTU2dmVtcDRMem9QdVJiOUY3bjlmcU9JNQp2blF3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUs2alBULzc2RnV2K0RLSmsxNG15b0ZzNThnRjRqbjlLWEUwCmFqOEMrZ1BUd2o2dUpUTjRLcWFmcnV0VGxlZWM5cXhabVZjQTgzanJhTEkySzlNN2ZyVE9pVE1vSnhmNnFrU1AKZ1ozazF2OG40Z0JGbzhsczZpc2YrankvL1dpMiswUVdWOElIU1lRbDlucklCT0lpb25rS1ljbDVQY2tKWVo4RQpkYVJUYW1xbi8zRTFGODFGaXFDT3dPc0NGRk5IRHhPRDRRb25NbU5ReFFvb1I3Mks1V2R6TmQrTG5BbjE4eHZlCjE2b3gybzZNQ3hjQS9RWDE0d3dDVi9lb1Y4KzIwWlJRY01LT1U5Y0djSjZNYm9TSW1odzJzS0NpNkpYcUQwQWMKSFZscWFzTGh4cHZBc3lJdXY3TjB5VnVhVVdZVDVMV1oyOFBGUVozcmwvYTIvNHZHNStJPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURpRENDQW5DZ0F3SUJBZ0lSQUp3R2cyNVR5YUNEVmlxNWJEbDliREF3RFFZSktvWklodmNOQVFFTEJRQXcKR3pFWk1CY0dBMVVFQXhNUVpXeGhjM1JwWTNObFlYSmphQzFqWVRBZUZ3MHlOREV3TWprd056RTNNamxhRncweQpOVEV3TWprd056RTNNamxhTUI4eEhUQWJCZ05WQkFNVEZHVnNZWE4wYVdOelpXRnlZMmd0YldGemRHVnlNSUlCCklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUE0L1QzMUp5VzJTSCtHZjhVTHJXVmhYZmwKdHhJRytMcGFGK0l6Q2lnSGh6QW5ZZnoyM0luQTZLOG9WWWtsejZLZzRzbEZjeXRHbStxd01ta3c3QzZXeHRVRwpSWlZiUXloZDZ6L1laRGhGUVpCQXZMWDVRVnIwWmIyRlN0NjdQM3dNa3Z1RE16TVZnRXoxZ2x0TmRyOVZiOXQyCjRUTXBka09GeFpPV24rZ0IyM0l3YnlNb0ZIMVNDZ1ZtcC9EQTNHZU1ENGErWURUcGowd1dSUFRRemdNcXh4YkUKd3FFdGN5R01yLzF6Sm4ycDZ6SWdmV3E0K1pwM2lRU2VOdjFUWWpHVm5xYzdWWUhDd29nb3pSRDI5TldTbC9BMApVSTVsbld3SDI3aU51QU1pVWFQRmx4eWtNbTlFbSs4SUcrT2VsRks4aDlBSEl2TDVYSFJjT3VOQzk2SjU1d0lECkFRQUJvNEhDTUlHL01BNEdBMVVkRHdFQi93UUVBd0lGb0RBZEJnTlZIU1VFRmpBVUJnZ3JCZ0VGQlFjREFRWUkKS3dZQkJRVUhBd0l3REFZRFZSMFRBUUgvQkFJd0FEQWZCZ05WSFNNRUdEQVdnQlRucTk2YW5ndk9nKzVGdjBYdQpmMStvNGptK2REQmZCZ05WSFJFRVdEQldnaFJsYkdGemRHbGpjMlZoY21Ob0xXMWhjM1JsY29JY1pXeGhjM1JwClkzTmxZWEpqYUMxdFlYTjBaWEl1WkdWbVlYVnNkSUlnWld4aGMzUnBZM05sWVhKamFDMXRZWE4wWlhJdVpHVm0KWVhWc2RDNXpkbU13RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQURwMVlXOFBmMm5YMldqeU5YZlVTSVJiamN0YQo2RTVpQzEwdENlbUxFZVpPZlpSRGtKOUVSaUNFQ2tUVUNCNy9QemFrTlE1UGNYQzVmcmYyWCtucGZ3RFFyREN4ClM0WkpEMWZFeHN0Yis3L29yQmgrWXNYcHJiUUJMbDJ6M0w0dm5tZ1kyb3V5bjdyT2NOdWQveENvWUdBVUd4a0YKdmVvUDNld0NUVzlaUVhGVWF0WUUzMno1bHRXTlRTOE5RU1hQRUtoSUlqYWNOL29SQ2ZhY1pRaTFoOUhTczdzQQpOcmludkRCTnE4bDl0b3g5NFZadCtXN3NmUXZvVU5hTTV1OXk0UU5Ib25rdUZ2enZMdkpGeEtvbWE0RmtFOHl5CmphR2RpUXh2NVFXdW1sTlBzZ3VOUUpSMnp3QzJEUkVVZUR1WC96Zk9xdDBucDFOZWpoWU11VTIyVk5zPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBNC9UMzFKeVcyU0grR2Y4VUxyV1ZoWGZsdHhJRytMcGFGK0l6Q2lnSGh6QW5ZZnoyCjNJbkE2SzhvVllrbHo2S2c0c2xGY3l0R20rcXdNbWt3N0M2V3h0VUdSWlZiUXloZDZ6L1laRGhGUVpCQXZMWDUKUVZyMFpiMkZTdDY3UDN3TWt2dURNek1WZ0V6MWdsdE5kcjlWYjl0MjRUTXBka09GeFpPV24rZ0IyM0l3YnlNbwpGSDFTQ2dWbXAvREEzR2VNRDRhK1lEVHBqMHdXUlBUUXpnTXF4eGJFd3FFdGN5R01yLzF6Sm4ycDZ6SWdmV3E0CitacDNpUVNlTnYxVFlqR1ZucWM3VllIQ3dvZ296UkQyOU5XU2wvQTBVSTVsbld3SDI3aU51QU1pVWFQRmx4eWsKTW05RW0rOElHK09lbEZLOGg5QUhJdkw1WEhSY091TkM5Nko1NXdJREFRQUJBb0lCQUhSK3VEaDdYY3ZSUjE1WgpzU0s5d1kvWDJobFlxUjlyZktjLy9mMXV2NG9pM2IyQjNWYVBQM3FxS042dG5Ca2tienYyeC9zM1hucEgwWXV4Cm5rTFUvRkRZaE1BQ3VBVDJHQ2tsRTUwRDlNQ3d5NlNsQ3FDUHJ0NWZvRUxHMk1KMHpxZyt5S25kclZ0SCtSK0oKTVdsQ0ZwTjNnS1ZOMUI2UUcwa0JSN1NvaUdwd1ZhSEFOMEc2Q0NKTnR0eDByZHo5UVRpU3BCMmlzYmhaSVpRVgp2U1NwemY3R2ZzaU5pN1VaOTN0WkZ5S21PRE1CUW5yZm5xWkphVXdVeWhHYk02Y2h4YWpmY0dka1dadWJ6azZJCnUrTWFDcE5VOWsvSXpDUkhnNjVyZGhITlozeVdDYzh6UXRDdFNkaWFPRmFwclZ4aHhSeFU5WW9FT2hsaVlVS0wKNGhIN1VPRUNnWUVBNkkrL3MyMDN5WGx1R0tqb3VIRXc5Z29aNU9YNHk3WWFGb2dzTm9jMTBZdUxRSURjTEozTQpLWjc4aTZ1dlV3d2lVMVpKTmRYbHFhcFFDNFRBN2N0bktnTVpNRnhzYzFPQzZsTnJ6b3FXM3FST0NDNW9DRUJOCmVSQlR2V1FjOVhXUUxrK2wwcHBEZ3BaVFVjc0tYS2NJSldlakhVb0xuajMrcmo2ZjFsOHlhU01DZ1lFQSt1NXEKd1ZtU2NkQ1pqdXQzbktsZ3V6ZGlpbWpvZ0Jkc1VnRUg4WXhkK2QzTXgveVR0eGthdGNwMnYvTHNPQ3BMV1YzSQpNVTdCTUtVR3BpZGpOMzdYQ3FMREZHM3UzdHZhMHBjditsOENaYk45NWdVN3RNeHB2WjZyZjRwVFlKdG1Ya2RsCkVmUkNxMkhocU11UEpSUlhjMllOaDU2TjNSUW5CWnJRVmxLdjBtMENnWUVBb0Fvc2RpRjIvcU1kN01Kd1JGMUEKd0ZCN09WWTVQSmI0cFFEWXpEMkgvOGZ6OEZPOU1NYjJ0TDNBTmEzVVhXWkFTUEZjT0R3V2JBZlVSZGo1bTZzYQpONE1pVm5HRUFHazc4bDJ1RnRpd3NrNkhsSUc2L2RLaWZlbUtkdzdxRHREMGc2bzBCeFk1MXlmejlwbXZhOHRXCmc4Y3FMUUhEdFFZY3VYUkhNcE1ZY2RrQ2dZRUFrZHdxbys5OEo3cDR1Rkg1T2xCZWtSVFZxOXpsWVNlOGFFSi8KS3BKTVFpVUNsekVqY0NnZ2xaRjF5NGZhZFo5b0l5OVhZZ29FVkZGbzl3WW9MeWNFdXdMM1lKV3laMHJtL01pegpNOWNzWG8raVhDV29taVRFUmx2SUZxQUNiVUtIazcvdWFTeFI0S3RKNzhNN2x2TW5Ea1pCRVJkQ0lVTklsNEp4CkhleDhsVlVDZ1lCdlVMQUU3WkMyTkFnZUpYUmpRalFRd2laNU1jS09rTTBvVjlsYUlaR0xZUUVTOHVtYXA0THYKMVc5clN4UElxMTg3Q1haejBuOTdPN210aUpwYXhERDg1QVdZMEg5MGhFNitvZUltWmlPN3ZYbW5RTVRNZjNtWgp6dUcvNk84ckpsNWcyNnY4NTVBYjVUbC9ZQTNRcW1tOVdKdUt5eGw1bWxvMGkxNU14cWNia1E9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
kind: Secret
metadata:
name: elasticsearch-master-certs
type: kubernetes.io/tls
---
apiVersion: v1
data:
password: Z3BVMTFFZXZNWWFmMkVCUw==
username: ZWxhc3RpYw==
kind: Secret
metadata:
name: elasticsearch-master-credentials
type: Opaque
场景三:基于OpenKruise SidecarSet自动注入Sidecar容器
上述提供的方法需要Sidecar容器配置到每个工作负载当中,在工作负载比较多的情况下,会比较大的增加用户的工作量。OpenKruise的SidecarSet提供了一种对Sidecar容器的管理能力,能够自动注入和独立升级Sidecar容器,进而做到一次配置自动注入的效果。详细信息,请参见向虚拟节点中的Pod注入Sidecar容器。