Log collection and processing, ES log ingestion assistant-Elasticsearch(ES)-阿里云帮助中心

Log collection and processing is a managed service for Alibaba Cloud Elasticsearch (ES) that provides a server-side write capability. It resolves the performance bottlenecks and operational complexity of high-concurrency log ingestion in self-managed ELK stacks. After you create a task, the platform automatically generates a write endpoint. You can then configure this endpoint in your collector, such as Filebeat or an OpenTelemetry Collector, to send log data to Elasticsearch.

Advantages

Traditional log collection solutions typically require a self-managed Kafka + Logstash multi-tier pipeline between the collector and Elasticsearch to handle high-concurrency writes. The log collection and processing service consolidates these intermediate components into a fully managed server-side write channel, simplifying the four-tier pipeline into two tiers. The key advantages are as follows:

Dimension	Traditional architecture	Log collection and processing service
Architecture	Collector → Kafka → Logstash → ES: a four-tier serial pipeline with many components	Collector → Managed service → ES: a two-tier pipeline where you only need to configure a write endpoint
O&M overhead	You must deploy and maintain Kafka and Logstash, and manage broker scaling, topics, partitions, consumer groups, and more	Fully managed — no need to purchase or maintain additional middleware
Troubleshooting	Many potential faults across multiple components requiring step-by-step investigation	Shorter pipeline with a significantly narrower scope of investigation
Write capacity	Depends on a self-managed Kafka buffer that requires manual scaling	Managed high-concurrency writes that automatically handle traffic spikes and prevent data loss
Resource cost	You must reserve Kafka and Logstash cluster resources for peak traffic, which leads to waste during off-peak periods	Pay-as-you-go with no need to reserve dedicated cluster resources
Compatibility	Components have version compatibility requirements and upgrades require synchronized configuration changes	Supports both Filebeat and OpenTelemetry collection protocols, is compatible with existing collectors, and is fully managed and maintenance-free

Limitations

Supports ES versions 7.10 and 8.17.0.
The OpenTelemetry collection method is supported only for ES version 8.17.0. The Filebeat collection method is supported for ES versions 7.10 and 8.17.0.

Step 1: Create a log write task

Log on to the Alibaba Cloud Elasticsearch console.
Click the ID of the target instance to go to the Basic Information page. In the left-side navigation pane, click Data Processing and Sync.
On the Data Processing and Sync page, find the Log Collection Service card and click Get Started.

On the Log Collection Service page, click Create and configure the following parameters:

Parameter	Description
Task Name	A custom name to identify the log write task.
API key	Select an available API Key. If none are available, go to the Alibaba Cloud OpenSearch console to create one.
Username	The username for the target Elasticsearch instance, used for write authentication.
Password	The password for the target Elasticsearch instance.
VPC	Select the VPC where the log source is located. By default, this is the VPC of the Elasticsearch instance. Each log write task is bound to one VPC. Ensure that the log source and the bound VPC can communicate with each other.
vSwitch	Select a vSwitch from the drop-down list. By default, this is the vSwitch of the Elasticsearch instance. A vSwitch that does not meet availability zone requirements is grayed out and cannot be selected.

After confirming the parameters, click Finish. The task is then created.
After the task is created, the system automatically generates a write endpoint in the format http://t-**********.private.-hangzhou.es-log.aliyuncs.com to receive log data.

Step 2: Obtain the write endpoint

After the task is created, obtain the write endpoint from the task details page. The collector will use this endpoint to send log data.

On the log collection and processing service page, find the target task in the task list and click View.
On the task details page, obtain the write endpoint:
- Internal Endpoint: Enabled by default. This endpoint is for collectors within the same VPC.
- Public Endpoint: Disabled by default. To collect logs over the internet, enable the public endpoint and configure a public access whitelist.
At the bottom of the task details page, find the configuration file template that corresponds to your Elasticsearch instance version and collection method:
- Version 8.17.0: Supports OpenTelemetry Collector and Filebeat.
- Version 7.10: Supports only Filebeat.
Click Copy to get the complete configuration file, which includes the endpoint and authentication details.

Step 3: Deploy an OpenTelemetry collector

The OpenTelemetry (OTel) collection method is supported only for ES version 8.17.0. If you are using Filebeat, apply the configuration file from Step 2 and skip this step.

This example shows how to deploy Elastic Agent in OpenTelemetry-compatible mode on a Container Service for Kubernetes (ACK) cluster. The agent collects container logs and sends them to your ES instance through the managed write endpoint.

1. OTel configuration file

The OTel configuration file that you copied in Step 2 is automatically populated with the write endpoint and authentication details. The core structure is as follows:

receivers:
  # Log receiver: Collects log files from the specified path.
  filelog:
    include: [/path/to/logs/*.log]   # Log file path. Modify this based on your actual path.

    exclude: [ ]

    start_at: end                    # Starts reading from the end of the file to collect only new logs.
    multiline:
      line_start_pattern: ^2025-     # Start marker for multiline logs. Modify based on your actual log format.
exporters:
  # Log exporter: Sends logs to the managed write endpoint.
  elasticsearch/logs:
    endpoints: ["http://es-cn-xxxxxxpxi00xxxxxx.elasticsearch.aliyuncs.com:9200"]  # The write endpoint from the task details page.
    user: "elastic"
    password: "your_password"
    tls:
      insecure_skip_verify: false
    retry:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
service:
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [elasticsearch/logs]

Parameter	Description
`receivers.filelog.include`	Path to the log files to collect. Wildcards are supported. The path must match the actual log output path of your application container.
`receivers.filelog.start_at`	Set to `end` to start reading from the end of the file. This prevents the collection of historical logs.
`receivers.filelog.multiline`	Defines multiline log parsing rules. For example, for a Java stack trace that spans multiple lines, you must configure a regular expression pattern to match the starting line.
`exporters.elasticsearch/logs.endpoints`	The address of the write endpoint. This is automatically populated from the task details page.
`exporters.elasticsearch/logs.user`	The username for the Elasticsearch instance.
`exporters.elasticsearch/logs.password`	The password for the Elasticsearch instance.
`exporters.elasticsearch/logs.retry`	The retry strategy for failed requests. The interval increases from 5s to 30s to prevent data loss.

Modify the log path in receivers.filelog.include and the regular expression pattern in multiline.line_start_pattern to match your environment. For more information about the configuration parameters, see OpenTelemetry Collector Receivers.

2. Create a ConfigMap in ACK

Deploy the OpenTelemetry configuration file to your ACK cluster as a ConfigMap.

Log on to the Container Service for Kubernetes (ACK) console and select your cluster.
In the left-side navigation pane, choose Configuration Management > ConfigMaps.
Click Create to create a ConfigMap named otel-config. Add the following key-value pair:
- Key: otel.yml
- Value: Paste the content of the OTel configuration file that you copied in Step 2.

3. Deploy a sidecar container

In your application's Deployment or StatefulSet, add an Elastic Agent container in sidecar mode to share a log volume with your application container.

In the ACK console, go to Workloads > Stateless (or Stateful), find the target workload, and click Edit.
Add a shared log volume: in the Volumes area, click Add Local Storage, add an emptyDir volume (for example, named app-log), and mount it to the application container's log output path.
Click Add Container to configure the sidecar container:
- Image Name: elastic/elastic-agent:9.1.5 (We recommend pushing the image to Alibaba Cloud Container Registry (ACR) beforehand.)
- environment variable: Add ELASTIC_AGENT_OTEL with the value true to enable the OpenTelemetry collection mode.

Configure two volume mounts for the sidecar container:

Mount	Volume name	Container path	Description
Shared log volume	`app-log`	`/path/to/logs`	Must match the log path of the application container.
OTel configuration	`otel-config` (ConfigMap)	`/usr/share/elastic-agent/otel.yml` (subPath: `otel.yml`)	Mount a single file to avoid overwriting the directory.

Click Update to submit the changes. Kubernetes performs a rolling restart of the Pods. Each new Pod contains both the application and sidecar containers.

4. Verify log collection

In the ACK console, view the logs of the sidecar container to confirm that it has started correctly.
Log on to the Kibana console and run the following query to verify that logs are successfully written:
```
GET /logs-filelog-default/_search
{
  "query": {
    "match_all": {}
  }
}
```
If the query returns application log records, the ingestion pipeline is working correctly.

Manage log write tasks

In the task list on the log collection and processing service page, you can perform the following operations on existing tasks:

Actions	Description
View Details	View task details, including creation information, the write endpoint, and collector configurations.
Stop	Stop a running log write task. After the task is stopped, collectors cannot write data through the endpoint.
Delete	Delete a log write task. After a task is deleted, its endpoint becomes permanently invalid.

FAQ

If your log data is not appearing in Elasticsearch as expected, troubleshoot the issue by following these steps:

Check the task status
Log on to the Alibaba Cloud Elasticsearch console. On the log write task list page, confirm that the Status of the task is Running. If the task status is abnormal, check whether the VPC and vSwitch configurations are correct, and ensure the log source can communicate with the bound VPC.
View task error logs
In the task list, click View on the target task to open its details page and check the error log.
If the error log contains exceptions, troubleshoot based on the error message. For example, the following error indicates that Elasticsearch user authentication failed. Check whether the username, password, or API Key is correctly configured.
```
reason: unable to authenticate user [elastic] for REST request [/_bulk]
type: security_exception
status: 401
```
Submit a ticket
If the issue persists, submit a ticket for technical support. Provide a detailed problem description and include any relevant error logs to expedite diagnosis.

Alibaba Cloud log collection enhancement