Collect Kubernetes events
This document describes how to use eventer to collect events from Kubernetes and send them to Simple Log Service.
The Kubernetes architecture is based on a state machine. Transitions between states generate corresponding events. Transitions between normal states generate Normal-level events, and transitions between normal and abnormal states generate Warning-level events.
Container Service for Kubernetes (ACK) provides an out-of-the-box event monitoring solution for container scenarios. It uses the ACK-maintained NPD and the included kube-eventer to provide container event monitoring.
Node Problem Detector (NPD) is a diagnostic tool for Kubernetes nodes. It converts node anomalies, such as Docker Engine hangs, Linux kernel hangs, network egress issues, or file descriptor anomalies, into node events. When combined with kube-eventer, NPD enables a closed-loop system for node event alerting.
kube-eventer is an open source tool maintained by ACK for the offline processing of Kubernetes events. It pushes cluster events to external systems, such as DingTalk, SLS, and EventBridge. It also provides filter conditions for different event levels to enable real-time collection, targeted alerting, and asynchronous archiving.
Node Problem Detector (NPD) is a diagnostic tool for Kubernetes nodes. It converts node anomalies, such as Docker Engine hangs, Linux kernel hangs, network egress issues, or file descriptor anomalies, into node events. When combined with kube-eventer, NPD enables a closed-loop system for node event alerting.
kube-eventer is an open source tool maintained by ACK for the offline processing of Kubernetes events. It pushes cluster events to external systems, such as DingTalk, SLS, and EventBridge. It also provides filter conditions for different event levels to enable real-time collection, targeted alerting, and asynchronous archiving.
Prerequisites
A Kubernetes cluster is created, such as an Alibaba Cloud ACK cluster or an ACK serverless cluster.
Billing
K8s event center is free of charge if the following conditions are met:
The data retention period for the Logstore associated with K8s event center is 90 days (the default value).
The volume of data written to K8s event center is less than 256 MB per day, which is approximately 250,000 events.
Examples:
If you use the default data retention period of 90 days and your Kubernetes cluster generates 1,000 events per day, K8s event center is free of charge.
If you set the data retention period to 105 days and your Kubernetes cluster generates 1,000 events per day, you are charged for Logstore storage after 90 days. The billable item is Log Storage. For more information about the Log Storage billable item, see Billable items in pay-by-feature mode.
Step 1: Deploy kube-eventer and node-problem-detector
ACK
For an ACK cluster, the ack-node-problem-detector component integrates the features of kube-eventer and node-problem-detector. You only need to deploy this component. For an ACK serverless cluster, you need to deploy the kube-eventer component.
NPD detects node problems or faults based on configurations and third-party plug-ins and then generates corresponding cluster events. The Kubernetes cluster also generates various events due to state transitions, such as pod evictions and image pull failures. The Kubernetes Event Center of Simple Log Service (SLS) aggregates all Kubernetes events in real time and provides storage, query, analysis, visualization, and alerting capabilities. The following steps describe how to send cluster events to the Kubernetes Event Center of SLS:
If you selected Install Node-problem-detector (NPD) and create an Event Center when you created the cluster, proceed to Step 2 to view the Kubernetes Event Center.
If you did not select Install Node-problem-detector and Create an Event Center when you created the cluster, you can install it manually.
Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of the target cluster. In the left navigation pane, choose .
On the Log and Monitoring tab, find and install ack-node-problem-detector.
Self-managed Kubernetes
Deploy
kube-eventer.Install kubectl. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.
Create a configuration file named eventer.yaml based on the following template.
apiVersion: apps/v1 kind: Deployment metadata: labels: name: kube-eventer name: kube-eventer namespace: kube-system spec: replicas: 1 selector: matchLabels: app: kube-eventer template: metadata: labels: app: kube-eventer annotations: scheduler.alpha.kubernetes.io/critical-pod: '' spec: dnsPolicy: ClusterFirstWithHostNet serviceAccount: kube-eventer containers: - image: registry.cn-hangzhou.aliyuncs.com/acs/kube-eventer:v1.2.5-cc7ec54-aliyun name: kube-eventer command: - "/kube-eventer" - "--source=kubernetes:https://kubernetes.default" ## .send to sls ## --sink=sls:https://{endpoint}?project={project}&logStore=k8s-event®ionId={region-id}&internal=false&accessKeyId={accessKeyId}&accessKeySecret={accessKeySecret} - --sink=sls:https://cn-beijing.log.aliyuncs.com?project=k8s-xxxx&logStore=k8s-event®ionId=cn-beijing&internal=false&accessKeyId=xxx&accessKeySecret=xxx env: # If TZ is assigned, set the TZ value as the time zone - name: TZ value: "Asia/Shanghai" volumeMounts: - name: localtime mountPath: /etc/localtime readOnly: true - name: zoneinfo mountPath: /usr/share/zoneinfo readOnly: true resources: requests: cpu: 10m memory: 50Mi limits: cpu: 500m memory: 250Mi volumes: - name: localtime hostPath: path: /etc/localtime - name: zoneinfo hostPath: path: /usr/share/zoneinfo --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: kube-eventer rules: - apiGroups: - "" resources: - events verbs: - get - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kube-eventer roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kube-eventer subjects: - kind: ServiceAccount name: kube-eventer namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: kube-eventer namespace: kube-systemParameter
Type
Required
Description
endpoint
string
Yes
The endpoint of Simple Log Service. For more information, see Endpoints.
project
string
Yes
The Simple Log Service project.
logStore
string
Yes
The Simple Log Service Logstore.
internal
string
Required for self-managed Kubernetes clusters.
For a self-managed Kubernetes cluster, you must set this parameter to false.
regionId
string
Required for self-managed Kubernetes clusters.
The ID of the region where Simple Log Service resides. For more information, see Endpoints.
accessKeyId
string
Required for self-managed Kubernetes clusters.
The AccessKey ID. We recommend that you use the AccessKey pair of a Resource Access Management (RAM) user. For more information, see AccessKey pair.
accessKeySecret
string
Required for self-managed Kubernetes clusters.
The AccessKey secret. We recommend that you use the AccessKey pair of a Resource Access Management (RAM) user. For more information, see AccessKey pair.
Run the following command to apply the
eventer.yamlconfiguration to the cluster.kubectl apply -f eventer.yamlExpected output:
deployment.apps/kube-eventer created clusterrole.rbac.authorization.k8s.io/kube-eventer created clusterrolebinding.rbac.authorization.k8s.io/kube-eventer created serviceaccount/kube-eventer created
Deploy
node-problem-detector.
Step 2: Create a K8s event center instance
After you create a K8s event center instance, Simple Log Service automatically creates a Logstore named k8s-event and related dashboards in the destination project.
Log on to the Simple Log Service console.
In the Log Application section, click the Intelligent O&M tab, and then click K8s Event Center.
On the Event Center Management page, click Add in the upper-right corner.
In the Create Event Center panel, configure the parameters and click Next.
If you select Select Existing Project, select an existing project from the Project drop-down list to manage the resources of the K8s event center instance, such as the Logstore and dashboards.
If you select Select Kubernetes Cluster from Container Service for Kubernetes, select an existing K8s cluster from the K8s Cluster drop-down list. If you use this method, Simple Log Service automatically creates a project named
k8s-log-{cluster-id}to manage the resources of the K8s event center instance, such as the Logstore and dashboards.
Step 3: Use the K8s event center instance
After you create a K8s event center instance and deploy kube-eventer and node-problem-detector, you can start using the event center. You can view the event overview, query event details, view the pod lifecycle, view node events, view core component events, configure alerts, run custom queries, and update the version.
On the K8s Event Center page, find the target K8s event center instance and click the
icon to perform the following operations.
Actions | Description |
View event overview | The Event Overview page displays summary statistics of core events. The statistics include the total number of events, a comparison of error events between today and yesterday, alert statistics, error event trends, and pod OOM details. Note Pod OOM information does not identify the specific pod. You can locate only the node, process name, and process ID where the event occurred. You can run a custom query to find pod restart events that occur around the time of the pod OOM to help identify the specific pod. |
Query event details | The Event Detail Query page displays the details of events filtered by various dimensions, such as event type, event object, host, namespace, and name. |
View pod lifecycle | The Pod Lifecycle page displays a graphical timeline of events in a pod's lifecycle. You can also filter important pod events by event level. |
View node events | The Node Events page displays node event details, such as the node lifecycle and an event list. |
View core component events | The Core Component Events page displays details of core component events, such as ECS restart failures and unimplemented URL modes. |
Set up alerts | On the Alert Configuration page, you can configure alerts for K8s event center. For more information, see Set up alerts. |
Run custom queries | On the Custom Query page, you can customize query statements and analytic statements. All events in the K8s event center are stored in a Logstore, so you can use all Logstore features, such as custom queries, event consumption, custom reports, and custom alerts. For more information, see Quick start for query and analysis. To access the project associated with the K8s event center instance, you can find the project name in one of the following ways:
|
Update version | On the Version Update page, you can upgrade K8s event center. |
Sample log
The following code shows a sample log.
hostname: cn-hangzhou.i-***********"
level: Normal
pod_id: 2a360760-****
pod_name: logtail-ds-blkkr
event_id: {
"metadata":{
"name":"logtail-ds-blkkr.157b7cc90de7e192",
"namespace":"kube-system",
"selfLink":"/api/v1/namespaces/kube-system/events/logtail-ds-blkkr.157b7cc90de7e192",
"uid":"2aaf75ab-****",
"resourceVersion":"6129169",
"creationTimestamp":"2019-01-20T07:08:19Z"
},
"involvedObject":{
"kind":"Pod",
"namespace":"kube-system",
"name":"logtail-ds-blkkr",
"uid":"2a360760-****",
"apiVersion":"v1",
"resourceVersion":"6129161",
"fieldPath":"spec.containers{logtail}"
},
"reason":"Started",
"message":"Started container",
"source":{
"component":"kubelet",
"host":"cn-hangzhou.i-***********"
},
"firstTimestamp":"2019-01-20T07:08:19Z",
"lastTimestamp":"2019-01-20T07:08:19Z",
"count":1,
"type":"Normal",
"eventTime":null,
"reportingComponent":"",
"reportingInstance":""
}log field | Type | Description |
hostname | string | The hostname of the host on which the event occurred. |
level | string | The log level. Valid values: Normal and Warning. |
pod_id | string | The unique ID of the pod. This field is included only if the event is related to a pod. |
pod_name | string | The name of the pod. This field is included only if the event is related to a pod. |
event_id | json | The details of the event. The value of this field is a JSON string. |
FAQ
A K8s event center instance has no data
After you deploy a K8s event center instance, new events are automatically collected. You can search for them on the Custom Query page. We recommend that you set the time range in the upper-right corner to 1 Day. If no data is found, there are usually two causes:
No new events are generated in the Kubernetes cluster since you deployed K8s event center.
You can run the
kubectl get events --all-namespacescommand to check whether new events are generated in the cluster.The parameters are incorrectly configured in your deployment of
kube-eventerandnode-problem-detector.If you use a Container Service for Kubernetes (ACK) cluster, perform these steps:
Log on to the ACK console.
On the Clusters page, click the name of the target cluster.
In the left-side navigation pane, choose .
On the Helm page, find ack-node-problem-detector and click Update.
Check and modify the parameter settings.
If you use a self-managed Kubernetes cluster, see Collect Kubernetes events for the parameter settings.
How do I view the container logs for an event?
If you use a Container Service for Kubernetes (ACK) cluster, perform these steps:
Log on to the ACK console.
On the Clusters page, click the name of the target cluster.
In the left-side navigation pane, choose .
Set Namespace to kube-system.
In the Pods list, find the target pod and click Log.
If you use a self-managed Kubernetes cluster, check the logs of the Pods with the filename prefix
eventer-slsin the kube-system namespace.