Centralized log aggregation for container clusters across multiple accounts
Solution overview
In a multi-account environment, you may want to collect logs from container clusters in different Alibaba Cloud accounts into a single log archive account. This allows for centralized log management and security auditing. The logs can include cluster operation logs, cluster audit logs, and service logs. This solution demonstrates how to use Simple Log Service (SLS) to configure centralized aggregation and auditing for container cluster logs across multiple accounts.
Solution advantages
Centralized logging
This solution provides a unified method for collecting logs from multiple container clusters. It explains how to enable and centrally collect cluster audit logs, operation logs, and service logs. Centralized log collection enables you to analyze logs, create visualizations, and perform security audits from a single, customizable, and user-friendly console.
Flexible log query and visualization
Simple Log Service supports SQL for custom queries. SLS also provides rich reporting features. You can use these features to display query results across different dimensions.
Customer scenario
Log collection and analysis for multiple clusters across multiple accounts
Scenario description
All services run in ACK managed clusters on the public cloud. You need to manage the logs from these clusters in a central log archive account.
Applicable customers
Public cloud customers
Cluster logs from each account are consolidated into a single log archive account.
Solution architecture
Architecture description:
There are three key types of container logs to focus on: service logs, cluster operation logs, and API server audit logs.
Service logs: You can use Logtail-DS to collect service logs from specified paths.
Operation logs: You can use ActionTrail to collect cluster-level management events, such as adding a node or disabling deletion protection.
Cluster audit logs: Enable API server audit logging to centrally collect API server operation events.
Products, pricing, and terms
Product pricing
Product Name |
Description |
Pricing |
Container Service for Kubernetes (ACK) |
Container Service for Kubernetes (ACK) is one of the first service platforms to pass the Kubernetes Conformance Certification program. ACK provides high-performance management services for containerized applications and supports lifecycle management for enterprise-level Kubernetes containerized applications. ACK helps you run Kubernetes containerized applications on the cloud with high efficiency. |
Charged. For more information, see Pricing. |
Simple Log Service |
Simple Log Service (SLS) is a cloud-native observability platform that provides large-scale, low-cost, and real-time services for data such as logs, metrics, and traces. SLS offers one-stop services for data collection, transformation, querying, analysis, visualization, alerting, and delivery. It helps you improve your digital capabilities in development, O&M, operations, and security. |
Charged. For more information, see Pricing. |
Resource Directory (RD) |
Resource Directory (RD) is a service that Alibaba Cloud provides for enterprise customers to manage relationships between multiple accounts and resources. |
Free |
ActionTrail |
ActionTrail is an Alibaba Cloud service that lets you query and deliver operation records of your cloud account resources. You can use ActionTrail for security analytics, resource change tracking, and compliance auditing. |
ActionTrail is currently free of charge. However, logs are delivered to Simple Log Service (SLS), which is billed separately based on SLS pricing. |
Glossary
Name |
Description |
Logtail-DS |
Logtail-DS is a log collection agent provided by Simple Log Service. It collects logs from clusters that run on Alibaba Cloud ECS instances, in on-premises data centers, or on other cloud platforms. |
Security
Simple Log Service security
Simple Log Service provides extensive security features. These include server-side encryption, fine-grained access control, and detailed service logs. It also offers a log audit feature that provides real-time, automated, and centralized log collection and auditing for major cloud products across multiple accounts. Simple Log Service provides rich data insights while meeting your enterprise security and compliance requirements. For more information, see Security and compliance.
Notes
Product limits
Simple Log Service has some usage limits. For more information, see Limits.
SLS storage costs
When you collect cluster audit logs across accounts, a copy of the log data is also saved in the SLS project of the source account. To minimize storage costs, use the Logstore in the source account only as a data tunnel. Set the data retention period for the Logstore to one day and disable indexing. This approach significantly reduces the cost of storing audit logs in the source account.
Service log planning
Logtail-DS uses different Logstore configurations based on the directory paths on different hosts. Plan your application log directories in advance.
Automatic Storage Tiering
Simple Log Service provides a cold storage feature to reduce your long-term storage costs. This feature does not affect log query, analysis, visualization, alerting, collection, or transformation capabilities. Cold storage is billed based on the amount of storage space used. There is no charge for moving data between hot and cold storage. For more information, see the product documentation.
Notes on cluster audit logs
The following operations are not audited: redundant watch requests from kube-proxy, get requests for nodes from kubelet and system:nodes, operations on endpoints by kube-system components, and get requests for namespaces from the API server.
Read-only URLs such as /healthz*, /version*, and /swagger* are not audited.
For operations on Secrets, ConfigMaps, and tokenreviews, which may contain sensitive information or binary files, the log level is set to Metadata. This level records only the user, timestamp, resource, and action for the request event. It does not include the request body or response body.
For sensitive APIs related to authentication, RBAC, certificates, autoscaling, and storage, the corresponding request and response bodies are recorded for both read and write operations.
Implementation steps
Scenario planning
This topic describes the implementation steps for three scenarios:
Scenario 1: Collect service logs from multiple clusters into a log archive account
Scenario 2: Collect cluster audit logs from multiple clusters into a log archive account
Scenario 3: Collect cluster operation logs into a log archive account
Preparations
Ensure that Container Service for Kubernetes (ACK) is activated for all relevant Alibaba Cloud accounts. For more information, see the official documentation.
Ensure that Simple Log Service (SLS) is activated for all relevant Alibaba Cloud accounts. For more information, see the official documentation.
Ensure that Resource Directory is enabled. For more information, see Enable a resource directory.
Ensure that a multi-account trail is created in ActionTrail to collect events from all regions into Simple Log Service (SLS). For more information, see Create a multi-account trail.
Implementation duration
After you complete the preparations, the implementation takes about 30 minutes.
Procedure
Scenario 1: Collect service logs from multiple clusters into a log archive account
Demo architecture
Install the Logtail component in the ACK cluster of the business account
Scenario 1: Install the Logtail component for an existing Kubernetes cluster
Log on to the ACK console.
In the navigation pane on the left, click Clusters.
On the Clusters page, click the target cluster.
In the navigation pane on the left, choose Operations > Components.
On the Logs and Monitoring tab, find Logtail-DS and click Install. After the installation, Simple Log Service automatically creates a Project named k8s-log-${your_k8s_cluster_id} (you can also specify a custom name). It also creates a machine group named k8s-group-${your_k8s_cluster_id} and a Logstore named config-operation-log in the Project.
Scenario 2: Install the Logtail component when you create a Kubernetes cluster
Log on to the ACK console.
In the navigation pane on the left, click Clusters.
On the Clusters page, click Create Cluster.
On the Component Configurations page, select Use Log Service.
After you select Use Log Service, a prompt to create a Project appears. For more information about how Simple Log Service organizes logs, see the documentation for a Project. You can create a Project in one of two ways.
Use Existing Project: You can select an existing Project to manage the collected container logs.
Create Project: Simple Log Service automatically creates a Project named k8s-log-{ClusterID} to manage the collected container logs. {ClusterID} is the unique ID of your new Kubernetes cluster.
After the installation, a machine group named k8s-group-${your_k8s_cluster_id} and a Logstore named config-operation-log are automatically created in the Project.
Configure cross-account collection of ACK service logs in Simple Log Service
Step 1: Set the Alibaba Cloud account ID as a custom identifier
Log on to the ACK console using your business account.
Set the log archive account as a custom identifier.
In the navigation pane on the left, click Clusters.
On the Clusters page, click the target cluster.
In the left navigation pane, select Configuration Management > Configuration Item.
Set Namespace to kube-system, and then click Edit in the alibaba-log-configuration row.
In the Edit panel, perform the following operations and then click OK. In the log-ali-uid configuration item, add the ID of the log archive account. Then, record the value of the log-machine-group configuration item, such as k8s-group-cc47****54428. You will set the custom identifier to this value when you create the machine group. Separate multiple account IDs with commas (,), for example, 17****397, 12****456.
Restart Logtail-DS to apply the configuration. On the Logtail-DS details page, confirm that the status of each pod is Running and that the creation time is after you updated the configuration.
In the navigation pane on the left, choose Workloads > DaemonSets.
In the DaemonSet list, click Edit in the Logtail-DS row.
In the Environment Variables section, click Add.
Add a custom variable with any content, for example, random_id: 439157431651471905349.
Click Update.
Step 2: Create a machine group
Log on to the Simple Log Service console using your log archive account.
In the Projects section, click the target Project.
In the navigation pane on the left, choose Resources > Machine Groups.
To the right of Machine Groups, click the
icon and choose Create Machine Group.In the Create Machine Group dialog box, configure the parameters and click OK. Set Custom Identifier to the machine group identifier that you recorded in Step 1, such as k8s-group-cc47****54428. For more information about the other parameters, see Create a custom identifier-based machine group.
Verify that the heartbeat status of the servers in the machine group is OK.
In the machine group list, click the target machine group.
On the Machine Group Settings page, verify the heartbeat status of the container nodes (ECS instances). A Heartbeat status of OK indicates that the container nodes are connected to Simple Log Service. If the status is FAIL, see Troubleshoot Logtail heartbeat failures.
Step 3: Create a Logtail collection configuration
Log on to the Simple Log Service console using your log archive account.
In the Import Data section, click Kubernetes - File.
Select the target Project and Logstore, and then click Next.
Click Use Existing Machine Groups.
Select the machine group that you created in Step 2, move it from Source Machine Groups to Applied Machine Groups, and then click Next.
Configure the Logtail collection settings and click Next. For more information about the parameters, see Collect container text logs in DaemonSet mode using the console. In this step, pay attention to the log path.
Preview the data, configure indexes, and then click Next. By default, Simple Log Service enables the full-text index feature. You can also manually or automatically create field indexes based on the collected logs. For more information, see Create indexes.
Scenario 2: Collect cluster audit logs from multiple clusters into a log archive account
Demo architecture
Enable audit logging for the ACK cluster
This section describes how to enable audit logging for an ACK cluster. Log on to your business account and perform the following steps:
Log on to the ACK console.
In the navigation pane on the left, click Clusters.
On the Clusters page, click the name of the target cluster, or click Details in the Actions column of the target cluster.
In the navigation pane on the left, choose Security > Cluster Audit.
Click Enable Cluster Audit, select Use Existing Project or Create New Project, and then click OK.
Create a Project and a Logstore in the log archive account to receive cluster logs from other accounts
This section describes how to create a Project and a Logstore to receive cluster logs from other accounts. Log on to your log archive account and perform the following steps:
Create a Project
Log on to the Simple Log Service console.
In the Projects section, click Create Project.
In the Create Project panel, configure the parameters and click OK. For more information about the parameters, see the user guide.
Create a Logstore
Log on to the Simple Log Service console.
In the Projects section, click the target Project.
On the Logstores tab, click the + icon.
On the Create Logstore page, configure the parameters. For more information about the parameters, see the user guide.
Aggregate cluster audit logs from multiple accounts into a single account
This section describes how to aggregate cluster audit logs from multiple accounts into a single account. Log on to the business account that contains the source audit logs and perform the following steps:
Create an SLS data transformation job to aggregate cluster audit logs from other accounts into a single account:
Log on to the Simple Log Service console.
Click the target project in the Project list area.
On the Logstores page, click the target Logstore. The name of the cluster audit Logstore is
audit-{cluster_id}.Create a data transformation job.
Leave the transformation script empty and click Save.
Configure the storage destination. For example, assume you want to aggregate logs into a Logstore named collect-cluster-audit-logs in a Project named collect-logs in a different account. Click OK to submit the data transformation job. For more information, see Aggregate data from multiple Logstores.
Note: You can use a custom role or an AccessKey for authorization. For more information, see the official documentation.
After a few moments, the aggregated cluster logs appear in the destination Logstore of the target account:
Troubleshooting
If you encounter issues when you use Logtail to collect service logs, see the following topics:
Possible causes and solutions for failures when collecting cluster audit logs
An invalid AccessKey ID or AccessKey secret is configured for the source Logstore.
Error log:
{ "errorCode": "Unauthorized", "errorMessage": "AccessKeyId not found: LTAI****************" } { "errorCode": "SignatureNotMatch", "errorMessage": "signature uJf****************= not match" }Solution: Verify that the AccessKey ID and AccessKey secret that you configured for the source Logstore are valid and correct.