Collect logs from ACK clusters

1. Prerequisites

Before you begin, ensure you have the following resources and configurations in place.

1. Prepare an ACK cluster and an ES instance

   • A Container Service for Kubernetes (ACK) cluster.

   • An Alibaba Cloud ES instance. We recommend enabling the auto index creation feature so the service can automatically create indices based on the data it receives.

     Go to the Basic Information page of the ES instance, click Configuration and Management > Cluster Configuration, click Modify, and select Enable.

2. Ensure network connectivity

   • Ensure that the ACK cluster and the ES instance can communicate with each other. If they are not in the same Virtual Private Cloud (VPC), you must first establish a connection. We recommend using aVPC peering connection to enable network communication.

3. Prepare the container image

   This solution uses the official Elastic Agent image. To ensure deployment stability and accessibility, we recommend pulling the required image and pushing it to your own Alibaba Cloud Container Registry (ACR) repository.

   # Pull the official image
   docker pull elastic/elastic-agent:9.1.5

   This solution has been tested. The elastic/elastic-agent:9.x version is compatible with ES version 8.17.

2. Configure the OTel Collector in ACK

You will create this configuration as a Kubernetes ConfigMap named otel-config. This ConfigMap configures the OpenTelemetry Collector (OTel Collector) to automatically collect, process, and export log data to your Alibaba Cloud ES cluster.

1. Log on to the ACK console and click Configurations > ConfigMaps.

2. Create a ConfigMap named otel-config. Set the key to otel.yml and the value to the content of your OTel configuration file.

   The following example shows the content of otel.yml. Modify it based on your actual configuration. For more details on OTel configuration, see openTelemetryReceivers.

   receivers:
     # Configure the filelog receiver
     filelog:
       include: [/path/to/logs/*.log]  # Path to the log files
       exclude: []                    # Files to exclude
       start_at: end                  # Start reading from the end of the file
       multiline:
         line_start_pattern: ^\d{4}-
   exporters:
     # Configure the Elasticsearch exporter
     elasticsearch/logs:
       endpoints: ["http://es-cn-xxxxxxpxi00xxxxxx.elasticsearch.aliyuncs.com:9200"]  # ES cluster endpoint
       user: "elastic"                          
       password: "your_password"
       tls:
         insecure_skip_verify: false                   
       retry:
         enabled: true
         initial_interval: 5s
         max_interval: 30s
   service:
     pipelines:
       logs:
         receivers: [filelog]
         exporters: [elasticsearch/logs]

   Parameter	Description
   Log collection (receivers)	Data source: Collects logs in real time from all .log files in the /path/to/logs/*.log directory (for example, application logs). Data source: Collects logs in real time from all .log files at the path /path/to/logs/*.log (for example, application logs). Key settings: start_at: end: Starts reading from the end of the file to avoid re-collecting historical logs and only process new logs. multiline: Identifies multi-line logs, such as Java stack traces, by using a pattern like ^\d{4}- as the start of a new log entry. This ensures complete entries are parsed correctly.
   Log export (exporters)	Target system: Securely transmits processed logs to the Alibaba Cloud ES instance. exporters.elasticsearch/logs.endpoints: The access endpoint for Elasticsearch. You can find the endpoint and port number on the Basic Information page of your ES instance. exporters.elasticsearch/logs.user: The username for accessing Elasticsearch. Security and reliability: TLS encryption: Specifies whether to enable certificate verification. Automatic retry: Retries failed attempts with an interval from 5s to 30s to prevent data loss.
   Data pipeline (service)	Full pipeline: The filelog receiver ingests logs, which are then processed and exported to ES. (In other words: log collection → standardized processing → secure transmission → storage in ES). Full workflow: filelog ingests logs → Adds resource and environment metadata → Processes data in batches → Exports data to ES. (That is: Log collection → Standardized processing → Secure transmission → Storage in ES)

3. Deploy the application and sidecar container

Add the log collection sidecar container definition to the Pod template of your existing Deployment or StatefulSet. This ensures that any new Pod created for your application will automatically include the sidecar container. The following example uses a Deployment, but the same configuration applies to a StatefulSet.

1. In the Deployment list, click the name of the target Deployment to open its details page.

2. Click Edit.

3. On the Edit page, find the Volume section and click Add Local Storage to add a shared log volume for the application container.

   On the Edit page, in the Volume section, click Add Local Storage to add a shared log volume for the business container.

   Important

   This volume acts as shared storage between the application container and the sidecar, ensuring the sidecar can read the log files. The Container Path must exactly match the actual log path of the application container; otherwise, the sidecar cannot collect data.

4. On the Edit page, click Add Container to add the log collection sidecar container.

   On the Edit page, click Add Container to add a log collection Sidecar container.

   • Image Name: Select the elastic/elastic-agent:9.1.5 image you prepared in the prerequisites.

   • Environment Variables: Add ELASTIC_AGENT_OTEL with the value true to enable OpenTelemetry collection mode.

5. Configure volume mounts for the sidecar container. Add two mount points:

   • Mount 1: Shared log volume

     • Volume Name: Select app-log (the volume shared with the application container).

     • Container path: /path/to/logs (Exactly the same as the mount path of the application container).

   • Mount 2: OTel configuration file

     • Volume type: Select ConfigMap.

     • Volume name: otel-config (The name of the ConfigMap that must be created in advance).

     • Mount path: /usr/share/elastic-agent/otel.yml (This must strictly match the official image path).

     • Subpath: otel.yml (Important: Mount only a single file to avoid overwriting the directory).

6. After you confirm that the configuration is correct, click Update.

   Kubernetes will automatically perform a rolling update of the Pods. The new Pods will contain both the application container and the sidecar container.

4. Verify log collection

1. Check container logs

   Get the Pod name and view the sidecar container's logs to confirm that it has started and is processing data correctly.

   If you see these logs, the container has started successfully.

2. Verify data in Elasticsearch

   Log on to the Kibana console associated with the ES instance and execute a query in Dev Tools to confirm that the log data is successfully written. The index name is typically generated automatically based on the filelog receiver, in a format such as logs-filelog-default.

   GET /logs-filelog-default/_search
   {
     "query": {
       "match_all": {}
     }
   }

   If the query returns log records from your application, the data pipeline is working correctly.