Collect Kubernetes Pod text logs with OTel-Elasticsearch(ES)-阿里云帮助中心

Architecture

This solution uses a sidecar deployment architecture. The core workflow is as follows:

Log sharing: The application container writes log files to a Kubernetes Volume (of type emptyDir) that is shared with the sidecar container.
Log collection: Running in the same Pod as the application container, the Elastic Agent sidecar container reads log files by mounting the shared Kubernetes Volume.
Mode switching: Elastic Agent switches to EDOT by setting the ELASTIC_AGENT_OTEL=true environment variable. In this mode, its behavior and configuration follow the OpenTelemetry Collector specification.
Data export: Based on its OpenTelemetry configuration file, Elastic Agent exports the collected logs directly to the specified Alibaba Cloud Elasticsearch instance.
Configuration management: A ConfigMap manages and mounts the collection configuration for Elastic Agent (otel.yml).

Prerequisites

Create an Alibaba Cloud Elasticsearch instance and enable the automatic index creation feature.

For detailed instructions, see Create an Alibaba Cloud Elasticsearch instance and Configure YML parameters.
Set up a native Kubernetes cluster or use an existing one.
Ensure that your Kubernetes cluster can connect to your Elasticsearch instance. If they are in different VPCs, establish a connection with a VPC Peering Connection.

Step 1: Prepare OTel Collector image

Pull the official Elastic Agent image. To ensure deployment stability and accessibility, we recommend pushing the image to your own container image repository.

docker pull elastic/elastic-agent:9.1.5

The elastic/elastic-agent:9.x image is compatible with Elasticsearch 8.17.

Step 2: Create OTel Collector configuration

Create a ConfigMap to store the OpenTelemetry Collector configuration.

Create an otel-config.yaml file with the following content:

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-config
  namespace: default
data:
  otel.yml: |-
    receivers:
      # Configure the filelog receiver.
      filelog:
        include: [/home/appuser/logs/*.log]  # Log file path, which must be the same as the application container's log path.

        exclude: [ ]

        start_at: end                        # Start reading from the end of the file to avoid collecting old logs.
        multiline:
          line_start_pattern: ^2026-         # Pattern that marks the start of a multi-line log entry. Adjust this pattern to match your log format.
    exporters:
      # Configure the Elasticsearch exporter.
      elasticsearch/logs:
        endpoints: ["http://es-cn-xxx.elasticsearch.aliyuncs.com:9200"]  # Replace with your actual ES cluster address.
        user: "{your username}"
        password: "{your password}"
        tls:
          insecure_skip_verify: false
        retry:
          enabled: true
          initial_interval: 5s
          max_interval: 30s
    processors:
      # Use batch processing to optimize performance.
      batch:
        send_batch_size: 1000
        timeout: 10s
      # Add resource metadata.
      resourcedetection:
        detectors: [system, env]
        system:
          hostname_sources: [os]
    service:
      pipelines:
        logs:
          receivers: [filelog]
          processors: [resourcedetection, batch]
          exporters: [elasticsearch/logs]

Configuration	Description
Log collection (receivers)	Collects `.log` files in real time from the specified path. `start_at: end` starts reading from the end of the file to collect only new logs. `multiline` identifies multi-line entries, such as Java exception stacks, by using a timestamp pattern to mark the start of a new log entry.
Log export (exporters)	Transfers logs to an Alibaba Cloud Elasticsearch instance. `endpoints` specifies the ES endpoint (you can find it on the Basic Information page of the ES instance). `user` specifies the username for accessing ES. This exporter supports TLS encryption and automatic retries (with backoff from `5s` to `30s`).
Data pipeline (service)	End-to-end flow: `filelog` receives logs -> adds resource metadata -> batches data -> exports to ES.

For more information about OTel configuration, see OpenTelemetry Receivers.

Run the following command to create the ConfigMap:

kubectl apply -f otel-config.yaml

Verify that the ConfigMap was created successfully:

kubectl get configmap otel-config

Step 3: Deploy application and sidecar

Add the log collection sidecar container definition to the Pod template of your existing Deployment or StatefulSet. The following example uses a Deployment, but the configuration is the same for a StatefulSet.

Define shared volumes.

In spec.template.spec.volumes, add two volumes at the same level as containers:
- app-log: an emptyDir volume for sharing log files between the application container and the sidecar container.
- otel-config: a ConfigMap volume that provides the collection configuration to the sidecar container.
```
volumes:
  - emptyDir: {}
    name: app-log
  - configMap:
      name: otel-config
    name: otel-config
```
Configure volume mounts for the application container.

Add the following mount to volumeMounts in the application container:
```
volumeMounts:
  - mountPath: /home/appuser/logs
    name: app-log
```
Important
The mountPath must match the application container's actual log output path. Otherwise, the sidecar cannot collect log data.

Inject the log collection sidecar container.

Add the following sidecar container definition under spec.template.spec.containers:

- name: otel-sidecar
  image: xxx.cn-hangzhou.cr.aliyuncs.com/otel/elastic-agent:9.1.5  # Replace with the actual image address.
  imagePullPolicy: IfNotPresent
  env:
    - name: ELASTIC_AGENT_OTEL
      value: 'true'
  resources:
    limits:
      cpu: '1'
      memory: 2Gi
    requests:
      cpu: '1'
      memory: 2Gi
  volumeMounts:
    - mountPath: /home/appuser/logs          # Shared log directory, which must be the same as the application container's log directory.
      name: app-log
    - mountPath: /usr/share/elastic-agent/otel.yml  # Mount path for the OTel configuration file.
      name: otel-config
      subPath: otel.yml

Setting the ELASTIC_AGENT_OTEL environment variable to true configures Elastic Agent to run in OpenTelemetry mode.

Step 4: Restart and verify

Restart the Deployment to apply the changes.

kubectl rollout restart deployment/<deployment-name>

Check the sidecar container logs to verify that the collector started correctly.
```
kubectl logs <pod-name> -c otel-sidecar
```
Log in to the Kibana console that is associated with your Elasticsearch instance. In Developer Tools, run the following query to verify that log data was written successfully. The index name is typically logs-filelog-default.
```
GET /logs-filelog-default/_search
{
  "query": {
    "match_all": {}
  }
}
```
If the query returns log records from your application, the data pipeline is working correctly.