本文介绍如何安装Sidecar及使用CRD方式创建采集配置,完成容器日志的采集。

前提条件

已安装alibaba-log-controller Helm,详情请参见安装Logtail日志组件

技术原理

通过Sidecar模式采集日志,依赖于Logtail和业务容器共享的日志目录,业务容器将日志写入到共享目录中,Logtail通过监控共享目录中日志文件的变化并采集日志。详细信息请参见官方文档Sidecar日志采集介绍Sidecar模式示例

步骤一:安装Sidecar

Sidecar模式的配置模板如下所示。
apiVersion: batch/v1
kind: Job
metadata:
  name: nginx-log-sidecar-demo
  namespace: default
spec:
  template:
    metadata:
      name: nginx-log-sidecar-demo
    spec:
      restartPolicy: Never
      containers:
      - name: nginx-log-demo
        image: registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest
        command: ["/bin/mock_log"]
        args: ["--log-type=nginx", "--stdout=false", "--stderr=true", "--path=/var/log/nginx/access.log", "--total-count=1000000000", "--logs-per-sec=100"]
        volumeMounts:
        - name: nginx-log
          mountPath: /var/log/nginx
      ##### logtail sidecar container
      - name: logtail
        # more info: https://cr.console.aliyun.com/repository/cn-hangzhou/log-service/logtail/detail
        # this images is released for every region
        image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:latest
        # when recevie sigterm, logtail will delay 10 seconds and then stop
        command:
        - sh
        - -c
        - /usr/local/ilogtail/run_logtail.sh 10
        livenessProbe:
          exec:
            command:
            - /etc/init.d/ilogtaild
            - status
          initialDelaySeconds: 30
          periodSeconds: 30
        resources:
          limits:
            memory: 512Mi
          requests:
            cpu: 10m
            memory: 30Mi
        env:
          ##### base config
          # user id
          - name: "ALIYUN_LOGTAIL_USER_ID"
            value: "${your_aliyun_user_id}"
          # user defined id
          - name: "ALIYUN_LOGTAIL_USER_DEFINED_ID"
            value: "${your_machine_group_user_defined_id}"
          # config file path in logtail's container
          - name: "ALIYUN_LOGTAIL_CONFIG"
            value: "/etc/ilogtail/conf/${your_region_config}/ilogtail_config.json"
          ##### env tags config
          - name: "ALIYUN_LOG_ENV_TAGS"
            value: "_pod_name_|_pod_ip_|_namespace_|_node_name_|_node_ip_"
          - name: "_pod_name_"
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: "_pod_ip_"
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: "_namespace_"
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: "_node_name_"
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: "_node_ip_"
            valueFrom:
              fieldRef:
                fieldPath: status.hostIP
        volumeMounts:
        - name: nginx-log
          mountPath: /var/log/nginx
      ##### share this volume
      volumes:
      - name: nginx-log
        emptyDir: {}
  1. 登录您的Kubernetes集群,详情请参见SSH密钥对访问Kubernetes集群
  2. 配置基础运行参数。
    ##### base config
              # user id
              - name: "ALIYUN_LOGTAIL_USER_ID"
                value: "${your_aliyun_user_id}"
              # user defined id
              - name: "ALIYUN_LOGTAIL_USER_DEFINED_ID"
                value: "${your_machine_group_user_defined_id}"
              # config file path in logtail's container
              - name: "ALIYUN_LOGTAIL_CONFIG"
                value: "/etc/ilogtail/conf/${your_region_config}/ilogtail_config.json"
    参数 说明
    ${your_region_config} 请根据日志服务Project所在地及网络类型填写,其中,地域信息请参见表 1
    • 如果为公网,格式为:region-internet,例如:华东 1(杭州)cn-hangzhou-Internet
    • 如果为阿里云内网,格式为region。例如:华东 1(杭州)cn-hangzhou
    ${your_aliyun_user_id} 您的阿里云主账号ID,详情请参见图 1
    ${your_machine_group_user_defined_id} 您机器组的自定义标识,请确保该标识在您的Project所在地域内唯一,详情请参见创建用户自定义标识机器组
  3. 配置挂载路径。
    volumeMounts:
    - name: nginx-log
    mountPath: /var/log/nginx
    • 请确保Logtail容器和业务容器挂载相同的目录。
    • 建议使用emptyDir的挂载方式。
  4. 延迟停止采集的时间。
    默认延迟停止采集的时间为10秒,即Logtail容器在接收到外部停止信号后会等待10秒再退出,防止有部分数据没有采集完毕。
    command:        
    - sh        
    - -c        
    - /usr/local/ilogtail/run_logtail.sh 10

步骤二:创建采集配置

您只需要定义AliyunLogConfig CRD即可创建采集配置,CRD配置格式如下所示。如果您要删除对应的采集配置只需删除对应的CRD资源即可。
apiVersion: log.alibabacloud.com/v1alpha1      ## 默认值,无需修改
kind: AliyunLogConfig                          ## 默认值,无需修改
metadata:
  name: simple-stdout-example                  ## 资源名,在集群内唯一
spec:
  project: k8s-my-project                      ## Project名称,如果不设置,默认采集日志到日志服务组件安装时的Project
  logstore: k8s-stdout                         ## Logstore名称,不存在时自动创建
  machineGroups:- nginx-log-sidecar            ## 机器组名称,如果不设置,默认应用到DaemonSet所在的机器组。
  shardCount: 2                                ## [可选]Shard数量,默认为2,支持1-10
  lifeCycle: 90                                ## [可选]Logstore中数据的存储时间,默认为90,支持1-7300,7300天为永久存储
  logtailConfig:                               ## 详细配置
    inputType: plugin                          ## 采集的数据源类型,file(文本文件),只支持采集文本文件
    configName: simple-stdout-example          ## 采集配置的名称,与资源名(metadata.name)保持一致
    inputDetail:                               ## 采集配置的详细信息,具体请参见本文下方的示例
      ...

Sidecar只支持文本文件采集,文本文件采集模式中,需把dockerFile选项设置为false。

logtailConfig字段的详细说明请参见Logtail配置

创建完成后,Logtail容器自动将产生的日志采集到日志服务,您可登录日志服务控制台查看。

示例

在线下IDC上自建Kubernetes集群安装Sidecar。其中,日志服务所在地域为华东1(杭州),使用的是公网方式采集,待挂载的卷名称为nginx-log ,类型为emptyDir,分别挂载到nginx-log-demo容器和Logtail容器的/var/log/nginx目录下。

  1. 安装Sidecar。
    apiVersion: batch/v1
    kind: Job
    metadata:
      name: nginx-log-sidecar-demo
      namespace: default
    spec:
      template:
        metadata:
          name: nginx-log-sidecar-demo
        spec:
          restartPolicy: Never
          containers:
          - name: nginx-log-demo
            image: registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest
            command: ["/bin/mock_log"]
            args: ["--log-type=nginx", "--stdout=false", "--stderr=true", "--path=/var/log/nginx/access.log", "--total-count=1000000000", "--logs-per-sec=100"]
            volumeMounts:
            - name: nginx-log
              mountPath: /var/log/nginx
          ##### logtail sidecar container
          - name: logtail
            # more info: ttps://cr.console.aliyun.com/repository/cn-hangzhou/log-service/logtail/detail
            # this images is released for every region
            image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:latest
            # when recevie sigterm, logtail will delay 10 seconds and then stop
            command:
            - sh
            - -c
            - /usr/local/ilogtail/run_logtail.sh 10
            livenessProbe:
              exec:
                command:
                - /etc/init.d/ilogtaild
                - status
              initialDelaySeconds: 30
              periodSeconds: 30
            env:
              ##### base config
              # user id
              - name: "ALIYUN_LOGTAIL_USER_ID"
                value: "xxxxxxxxxx"
              # user defined id
              - name: "ALIYUN_LOGTAIL_USER_DEFINED_ID"
                value: "nginx-log-sidecar"
              # config file path in logtail's container
              - name: "ALIYUN_LOGTAIL_CONFIG"
                value: "/etc/ilogtail/conf/cn-hangzhou-internet/ilogtail_config.json"
              ##### env tags config
              - name: "ALIYUN_LOG_ENV_TAGS"
                value: "_pod_name_|_pod_ip_|_namespace_|_node_name_|_node_ip_"
              - name: "_pod_name_"
                valueFrom:
                  fieldRef:
                    fieldPath: metadata.name
              - name: "_pod_ip_"
                valueFrom:
                  fieldRef:
                    fieldPath: status.podIP
              - name: "_namespace_"
                valueFrom:
                  fieldRef:
                    fieldPath: metadata.namespace
              - name: "_node_name_"
                valueFrom:
                  fieldRef:
                    fieldPath: spec.nodeName
              - name: "_node_ip_"
                valueFrom:
                  fieldRef:
                    fieldPath: status.hostIP
            volumeMounts:
            - name: nginx-log
              mountPath: /var/log/nginx
          ##### share this volume
          volumes:
          - name: nginx-log
            emptyDir: {}
  2. 创建采集配置。
    • 配置采集的访问日志为/var/log/nginx/access.log ,采集的目的Logstore为nginx-access
      # config for access log
      apiVersion: log.alibabacloud.com/v1alpha1
      kind: AliyunLogConfig
      metadata:
        # your config name, must be unique in you k8s cluster
        name: nginx-log-access-example
      spec:
        # project name to upload log
        project: k8s-nginx-sidecar-demo
        # logstore name to upload log
        logstore: nginx-access
        # machine group list to apply config, should be same with your sidecar' [ALIYUN_LOGTAIL_USER_DEFINED_ID]
        machineGroups:
        - nginx-log-sidecar
        # logtail config detail
        logtailConfig:
          # log file's input type is 'file'
          inputType: file
          # logtail config name, should be same with [metadata.name]
          configName: nginx-log-access-example
          inputDetail:
            # 极简模式日志,logType设置为"common_reg_log"
            logType: common_reg_log
            # 日志文件夹
            logPath: /var/log/nginx
            # 文件名, 支持通配符,例如log_*.log
            filePattern: access.log
            # sidecar模式,dockerFile设置为false
            dockerFile: false
            # 行首正则表达式,如果为单行模式,设置成 .*
            logBeginRegex: '.*'
            # 解析正则
            regex: '(\S+)\s(\S+)\s\S+\s\S+\s"(\S+)\s(\S+)\s+([^"]+)"\s+(\S+)\s(\S+)\s(\d+)\s(\d+)\s(\S+)\s"([^"]+)"\s.*'
            # 提取出的key列表
            key : ["time", "ip", "method", "url", "protocol", "latency", "payload", "status", "response-size",ser-agent"]
      # config for error log
    • 配置采集的错误日志为/var/log/nginx/error.log ,采集的目的Logstore为nginx-error
      # config for error log
      apiVersion: log.alibabacloud.com/v1alpha1
      kind: AliyunLogConfig
      metadata:
        # your config name, must be unique in you k8s cluster
        name: nginx-log-error-example
      spec:
        # project name to upload log
        project: k8s-nginx-sidecar-demo
        # logstore name to upload log
        logstore: nginx-error
        # machine group list to apply config, should be same with your sidecar' [ALIYUN_LOGTAIL_USER_DEFINED_ID]
        machineGroups:
        - nginx-log-sidecar
        # logtail config detail
        logtailConfig:
          # log file's input type is 'file'
          inputType: file
          # logtail config name, should be same with [metadata.name]
          configName: nginx-log-error-example
          inputDetail:
            # 极简模式日志,logType设置为"common_reg_log"
            logType: common_reg_log
            # 日志文件夹
            logPath: /var/log/nginx
            # 文件名, 支持通配符,例如log_*.log
            filePattern: error.log
            # sidecar模式,dockerFile设置为false
            dockerFile: false