通过Sidecar-CRD方式采集容器日志_采集容器日志_Logtail采集_数据采集

技术原理

通过Sidecar模式采集日志，依赖于Logtail和业务容器共享的日志目录，业务容器将日志写入到共享目录中，Logtail通过监控共享目录中日志文件的变化并采集日志。详细信息请参见官方文档Sidecar日志采集介绍和Sidecar模式示例。

步骤一：安装Sidecar

Sidecar模式的配置模板如下所示。

apiVersion: batch/v1
kind: Job
metadata:
  name: nginx-log-sidecar-demo
  namespace: default
spec:
  template:
    metadata:
      name: nginx-log-sidecar-demo
    spec:
      restartPolicy: Never
      containers:
      - name: nginx-log-demo
        image: registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest
        command: ["/bin/mock_log"]
        args: ["--log-type=nginx", "--stdout=false", "--stderr=true", "--path=/var/log/nginx/access.log", "--total-count=1000000000", "--logs-per-sec=100"]
        volumeMounts:
        - name: nginx-log
          mountPath: /var/log/nginx
      ##### logtail sidecar container
      - name: logtail
        # more info: https://cr.console.aliyun.com/repository/cn-hangzhou/log-service/logtail/detail
        # this images is released for every region
        image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:latest
        # when recevie sigterm, logtail will delay 10 seconds and then stop
        command:
        - sh
        - -c
        - /usr/local/ilogtail/run_logtail.sh 10
        livenessProbe:
          exec:
            command:
            - /etc/init.d/ilogtaild
            - status
          initialDelaySeconds: 30
          periodSeconds: 30
        resources:
          limits:
            memory: 512Mi
          requests:
            cpu: 10m
            memory: 30Mi
        env:
          ##### base config
          # user id
          - name: "ALIYUN_LOGTAIL_USER_ID"
            value: "${your_aliyun_user_id}"
          # user defined id
          - name: "ALIYUN_LOGTAIL_USER_DEFINED_ID"
            value: "${your_machine_group_user_defined_id}"
          # config file path in logtail's container
          - name: "ALIYUN_LOGTAIL_CONFIG"
            value: "/etc/ilogtail/conf/${your_region_config}/ilogtail_config.json"
          ##### env tags config
          - name: "ALIYUN_LOG_ENV_TAGS"
            value: "_pod_name_|_pod_ip_|_namespace_|_node_name_|_node_ip_"
          - name: "_pod_name_"
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: "_pod_ip_"
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: "_namespace_"
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: "_node_name_"
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: "_node_ip_"
            valueFrom:
              fieldRef:
                fieldPath: status.hostIP
        volumeMounts:
        - name: nginx-log
          mountPath: /var/log/nginx
      ##### share this volume
      volumes:
      - name: nginx-log
        emptyDir: {}

登录您的Kubernetes集群，详情请参见SSH密钥对访问Kubernetes集群。

配置基础运行参数。

##### base config
          # user id
          - name: "ALIYUN_LOGTAIL_USER_ID"
            value: "${your_aliyun_user_id}"
          # user defined id
          - name: "ALIYUN_LOGTAIL_USER_DEFINED_ID"
            value: "${your_machine_group_user_defined_id}"
          # config file path in logtail's container
          - name: "ALIYUN_LOGTAIL_CONFIG"
            value: "/etc/ilogtail/conf/${your_region_config}/ilogtail_config.json"


参数	说明
`${your_region_config}`	请根据日志服务Project所在地及网络类型填写，其中，地域信息请参见表 1。如果为公网，格式为：`region-internet`，例如：华东 1（杭州）为cn-hangzhou-Internet。如果为阿里云内网，格式为`region`。例如：华东 1（杭州）为cn-hangzhou。
`${your_aliyun_user_id}`	您的阿里云主账号ID，详情请参见配置用户标识。
`${your_machine_group_user_defined_id}`	您机器组的自定义标识，请确保该标识在您的Project所在地域内唯一，详情请参见创建用户自定义标识机器组。

配置挂载路径。
```
volumeMounts:
- name: nginx-log
mountPath: /var/log/nginx
```
- 请确保Logtail容器和业务容器挂载相同的目录。
- 建议使用emptyDir的挂载方式。
延迟停止采集的时间。
通常情况下，延迟停止采集的时间为10秒，即Logtail容器在接收到外部停止信号后会等待10秒再退出，防止有部分数据没有采集完毕。
```
command:        
- sh        
- -c        
- /usr/local/ilogtail/run_logtail.sh 10
```

步骤二：创建采集配置

您只需要定义AliyunLogConfig CRD即可创建采集配置，CRD配置格式如下所示。如果您要删除对应的采集配置只需删除对应的CRD资源即可。

apiVersion: log.alibabacloud.com/v1alpha1      ## 默认值，无需修改
kind: AliyunLogConfig                          ## 默认值，无需修改
metadata:
  name: simple-stdout-example                  ## 资源名，在集群内唯一
spec:
  project: k8s-my-project                      ## Project名称，如果不设置，默认采集日志到日志服务组件安装时的Project
  logstore: k8s-stdout                         ## Logstore名称，不存在时自动创建
  machineGroups:- nginx-log-sidecar            ## 机器组名称，如果不设置，默认应用到DaemonSet所在的机器组。
  shardCount: 2                                ## [可选]Shard数量，默认为2，支持1-10
  lifeCycle: 90                                ## [可选]Logstore中数据的存储时间，默认为90，支持1-7300，7300天为永久存储
  logtailConfig:                               ## 详细配置
    inputType: plugin                          ## 采集的数据源类型，file（文本文件），只支持采集文本文件
    configName: simple-stdout-example          ## 采集配置的名称，与资源名(metadata.name)保持一致
    inputDetail:                               ## 采集配置的详细信息，具体请参见本文下方的示例
      ...

Sidecar只支持文本文件采集，文本文件采集模式中，需把dockerFile选项设置为false。

logtailConfig字段的详细说明请参见Logtail配置。

创建完成后，Logtail容器自动将产生的日志采集到日志服务，您可登录日志服务控制台查看。

示例

在线下IDC上自建Kubernetes集群安装Sidecar。其中，日志服务所在地域为华东1（杭州），使用的是公网方式采集，待挂载的卷名称为nginx-log ，类型为emptyDir，分别挂载到nginx-log-demo容器和Logtail容器的/var/log/nginx目录下。

安装Sidecar。

apiVersion: batch/v1
kind: Job
metadata:
  name: nginx-log-sidecar-demo
  namespace: default
spec:
  template:
    metadata:
      name: nginx-log-sidecar-demo
    spec:
      restartPolicy: Never
      containers:
      - name: nginx-log-demo
        image: registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest
        command: ["/bin/mock_log"]
        args: ["--log-type=nginx", "--stdout=false", "--stderr=true", "--path=/var/log/nginx/access.log", "--total-count=1000000000", "--logs-per-sec=100"]
        volumeMounts:
        - name: nginx-log
          mountPath: /var/log/nginx
      ##### logtail sidecar container
      - name: logtail
        # more info: ttps://cr.console.aliyun.com/repository/cn-hangzhou/log-service/logtail/detail
        # this images is released for every region
        image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:latest
        # when recevie sigterm, logtail will delay 10 seconds and then stop
        command:
        - sh
        - -c
        - /usr/local/ilogtail/run_logtail.sh 10
        livenessProbe:
          exec:
            command:
            - /etc/init.d/ilogtaild
            - status
          initialDelaySeconds: 30
          periodSeconds: 30
        env:
          ##### base config
          # user id
          - name: "ALIYUN_LOGTAIL_USER_ID"
            value: "xxxxxxxxxx"
          # user defined id
          - name: "ALIYUN_LOGTAIL_USER_DEFINED_ID"
            value: "nginx-log-sidecar"
          # config file path in logtail's container
          - name: "ALIYUN_LOGTAIL_CONFIG"
            value: "/etc/ilogtail/conf/cn-hangzhou-internet/ilogtail_config.json"
          ##### env tags config
          - name: "ALIYUN_LOG_ENV_TAGS"
            value: "_pod_name_|_pod_ip_|_namespace_|_node_name_|_node_ip_"
          - name: "_pod_name_"
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: "_pod_ip_"
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: "_namespace_"
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: "_node_name_"
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: "_node_ip_"
            valueFrom:
              fieldRef:
                fieldPath: status.hostIP
        volumeMounts:
        - name: nginx-log
          mountPath: /var/log/nginx
      ##### share this volume
      volumes:
      - name: nginx-log
        emptyDir: {}

创建采集配置。

配置采集的访问日志为/var/log/nginx/access.log ，采集的目的Logstore为nginx-access。

# config for access log
apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  # your config name, must be unique in you k8s cluster
  name: nginx-log-access-example
spec:
  # project name to upload log
  project: k8s-nginx-sidecar-demo
  # logstore name to upload log
  logstore: nginx-access
  # machine group list to apply config, should be same with your sidecar' [ALIYUN_LOGTAIL_USER_DEFINED_ID]
  machineGroups:
  - nginx-log-sidecar
  # logtail config detail
  logtailConfig:
    # log file's input type is 'file'
    inputType: file
    # logtail config name, should be same with [metadata.name]
    configName: nginx-log-access-example
    inputDetail:
      # 极简模式日志，logType设置为"common_reg_log"
      logType: common_reg_log
      # 日志文件夹
      logPath: /var/log/nginx
      # 文件名, 支持通配符，例如log_*.log
      filePattern: access.log
      # sidecar模式，dockerFile设置为false
      dockerFile: false
      # 行首正则表达式，如果为单行模式，设置成 .*
      logBeginRegex: '.*'
      # 解析正则
      regex: '(\S+)\s(\S+)\s\S+\s\S+\s"(\S+)\s(\S+)\s+([^"]+)"\s+(\S+)\s(\S+)\s(\d+)\s(\d+)\s(\S+)\s"([^"]+)"\s.*'
      # 提取出的key列表
      key : ["time", "ip", "method", "url", "protocol", "latency", "payload", "status", "response-size",ser-agent"]
# config for error log

配置采集的错误日志为/var/log/nginx/error.log ，采集的目的Logstore为nginx-error。

# config for error log
apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  # your config name, must be unique in you k8s cluster
  name: nginx-log-error-example
spec:
  # project name to upload log
  project: k8s-nginx-sidecar-demo
  # logstore name to upload log
  logstore: nginx-error
  # machine group list to apply config, should be same with your sidecar' [ALIYUN_LOGTAIL_USER_DEFINED_ID]
  machineGroups:
  - nginx-log-sidecar
  # logtail config detail
  logtailConfig:
    # log file's input type is 'file'
    inputType: file
    # logtail config name, should be same with [metadata.name]
    configName: nginx-log-error-example
    inputDetail:
      # 极简模式日志，logType设置为"common_reg_log"
      logType: common_reg_log
      # 日志文件夹
      logPath: /var/log/nginx
      # 文件名, 支持通配符，例如log_*.log
      filePattern: error.log
      # sidecar模式，dockerFile设置为false
      dockerFile: false

通过Sidecar-CRD方式采集容器日志

前提条件

技术原理

步骤一：安装Sidecar

步骤二：创建采集配置

示例