Applications, containers, and infrastructure components in a cluster continuously generate large volumes of scattered log data, which makes log collection and management complex. Container Service for Kubernetes (ACK) integrates with Simple Log Service (SLS) to provide solutions that simplify the log management process.
For more information about the features and terms of SLS, see What is Simple Log Service and Glossary.
Introduction to log collection components
Log collection components are deeply integrated with SLS and use automatic discovery to simplify container log management. These components support various formats, such as text logs, binary logging, and HTTP data. Log data can be stored directly in SLS for persistent storage. You can use the real-time analysis and query features of SLS to quickly identify and locate issues, which improves troubleshooting efficiency and reduces O&M costs. The following log collection components are supported.
LoongCollector (formerly Logtail): Logtail is a log collection agent provided by Simple Log Service. You can use Logtail to collect logs from servers such as Alibaba Cloud ECS instances, self-managed IDCs, or servers from other cloud providers. Logtail collects logs from files without requiring application code modifications, so your applications run without interruption. LoongCollector is Simple Log Service's next-generation collection agent. As an upgrade to Logtail, it is backward-compatible and delivers enhanced performance.
Logtail-based data collection: Logtail is a log collection agent that is provided by Simple Log Service. You can use Logtail to collect logs from multiple data sources, such as Alibaba Cloud Elastic Compute Service (ECS) instances, servers in data centers, and servers from third-party cloud service providers. Logtail supports non-intrusive log collection based on log files. You do not need to modify your application code, and log collection does not affect the operation of your applications.
Log collection principles
This section describes two log collection methods and their procedures.
Collection procedure
In an ACK cluster, you can deploy a log collection component to securely transmit various types of log data to SLS for storage and analysis. This method simplifies configuration and management and improves efficiency. The basic procedure is as follows:
The entire procedure includes:
Deploy a log collection component.
When you install a log collection component for your cluster, you create an AliyunLogConfig CRD and deploy resources based on the log collection component. This associates the AliyunLogConfig in the ACK cluster with the SLS collection configuration.
LoongCollector component: loongcollector-ds (DaemonSet), loongcollector-operator (Deployment), alibaba-log-configuration (ConfigMap)
Logtail component: logtail-ds (DaemonSet), alibaba-log-controller (Deployment), alibaba-log-configuration (ConfigMap)
Configure data collection. Create a collection configuration. The log collection component works based on this configuration.
You can configure log collection using the console, a CRD configuration, or environment variables. For more information, see Collect container logs from ACK clusters, Collect stdout and stderr from Kubernetes containers in DaemonSet mode (old version), and Collect text logs from Kubernetes containers in Sidecar mode.
Report logs to SLS. The log collection component uploads log data to SLS in real time.
Collection method
In a Kubernetes cluster, there are two log collection methods: Sidecar and DaemonSet. These methods have different procedures and are suited for different scenarios.
DaemonSet: The log collection component is deployed in the
kube-systemnamespace and runs as a DaemonSet. The log collection pod on each node collects data from all pods on that node, including standard output and files. For more information, see Collect container logs from ACK clusters.Sidecar
Each application pod runs a log collection container that shares the log volume with the main container. For more information, see Collect text logs from Kubernetes containers in Sidecar mode.
Classification | DaemonSet method | Sidecar method |
Scenarios | Suited for clusters with clearly classified logs and relatively simple functions.
| Suited for large, hybrid clusters. This method is useful when you need different log collection configurations for each pod. However, this method can generate a large volume of logs per pod and consume significant resources. |
Resource usage | One log collection pod runs on each node. | One log collection container runs in each pod. |
Deployment and O&M | You only need to maintain the DaemonSet. | A log collection container must be deployed for each pod that requires log collection. |
Log type collected | Standard output and text logs. | Text logs. |
Supported log types
In an ACK cluster, you can collect log data from different sources and securely transmit it to SLS for storage and analysis.
Log type | Description | References |
Application (container) logs | You can quickly collect container logs from ACK clusters, including the container's standard output and text files from within the container. |
|
Control plane component logs | You can collect logs from control plane components, such as API Server, kube-scheduler, kube-controller-manager, cloud-controller-manager, Controlplane-Events, and the ALB Ingress Controller. The collected logs are sent to a specified Logstore. This facilitates centralized management and analysis of the running status of control plane components. It also helps you quickly troubleshoot issues, monitor component health, and verify the effects of custom parameter configurations. Each SLS Logstore corresponds to a Kubernetes control plane component. | Collect control plane component logs of ACK managed clusters |
Network component logs |
| |
Data plane component logs | You can collect logs for plugins, such as storage plugins, Terway networking, Flannel networking, and Auto Scaling. |
Limits
Limitations | Description |
Cluster version |
|
| |
Actual usage depends on the collection rate, the number of monitored directories and files, and the degree of sending blockage. Ensure that the actual usage is less than 80% of the limit. For more information about the limits, see Logtail limits. |
Billing
The costs incurred when you use SLS in an ACK cluster are the sum of cluster management fees and SLS fees.
Cluster management fees: These fees are charged by ACK. Only ACK Pro clusters incur these fees. ACK Basic clusters and ACK dedicated clusters are free of charge. For more information, see Billing overview.
SLS fees: These fees are billed on a pay-as-you-go basis and are charged by SLS. For more information, see Billing overview.
FAQ
How do I copy logs from a project to another project?
To copy the access logs from logstore-a in project-a to the newly created logstore-b in project-b for unified query and analysis in project-b, see Replicate Logstore data.
What do I do if the log collection status is abnormal?
If the collection status is abnormal when you use Logtail to collect logs from containers, such as standard or Kubernetes containers, you can follow the instructions in How to troubleshoot abnormal container log collection to troubleshoot the issue, check the running status, and perform other O&M operations.
FAQ about log loss, log deletion, and reducing log storage costs
For information about how to change the log retention period and how to disable log collection, see Logstore-related issues.
Logtail-related issues
For information about Logtail log collection latency, how to collect historical logs, and how long it takes for Logtail configuration changes to take effect, see Logtail basic issues.