Choose a collection method based on your deployment environment. To collect historical data instead of streaming logs, see data import.
Data collection methods
Simple Log Service transfers and stores your target data so you can query and analyze it, run data processing to transform its format and content, or consume and ship data to third-party systems.
All collection methods support only incremental data collection. To collect historical data, use data import.
Two primary collection agents are available:
LoongCollector (formerly Logtail): The next-generation log collection agent from Simple Log Service. Deploy it on servers, Kubernetes nodes, or Docker hosts to collect logs from files without modifying application code. It is backward-compatible with Logtail and delivers enhanced performance.
SDK-based collection: Use the Simple Log Service SDKs or APIs to write logs directly from your application code. This gives you the most flexibility for custom collection logic.
Collection solutions for Kubernetes
If your applications run in Kubernetes, choose one of the following methods:
-
Collect container logs from a Kubernetes cluster: Collect application logs from standard output or text files. Before you begin, read the Notes on collecting container logs from a Kubernetes cluster.
Collect container logs from a cluster by using the console (standard output/files): Configure log collection rules through the console. Suitable for a small number of clusters and test environments.
Collect container logs from a cluster by using Kubernetes CRDs (standard output/files): Use Custom Resource Definitions (CRDs) to configure log collection rules as code. Easy to template and version-control — recommended for production clusters and CI/CD pipelines.
Collect Kubernetes monitoring data (metrics): Deploy Prometheus on Kubernetes, collect monitoring data into a MetricStore in Simple Log Service, and connect the MetricStore to Grafana for visualization.
Collect Kubernetes events: Use Eventer with Simple Log Service to collect events from your Kubernetes cluster.
Collection solutions for hosts and servers
If your applications run on Alibaba Cloud ECS instances, servers in self-managed data centers, or other cloud servers, choose one of the following methods:
Continuously collect text logs from hosts: LoongCollector supports multiple log formats including Nginx, JSON, Apache, IIS, and delimiter-separated formats. Note the relationship between your servers and your Simple Log Service project, as it determines how the agent is installed.
Collect host monitoring data (metrics): Collect CPU, memory, load, disk, and network metrics from your hosts.
Collect Windows event logs: Simple Log Service uses Windows APIs to subscribe to Windows event channels and continuously collect event logs published by applications and the kernel.
Collection solutions for Docker
If your applications run in Docker containers, choose one of the following methods:
Collect Docker container logs (standard output/files): Collect standard output (stdout/stderr) and text log files from Docker containers.
Collect Docker events: Collect Docker events including container lifecycle, image operations, plug-in activity, network changes, and storage volume interactions.
Alibaba Cloud service collection
Alibaba Cloud service log collection: Simple Log Service can collect logs from Alibaba Cloud services across elastic computing, storage, security, and database categories. These logs include operational events, running status, and business activity records.
Collection solutions for middleware
Collect SQL query results: Collect query results from SQL Server, MySQL, and PostgreSQL.
-
Collect front-end and web service logs:
Use WebTracking to collect front-end logs: WebTracking collects user behavior data from browsers, mini programs, and mobile clients by adding a code snippet to your front-end page.
Collect HTTP data: Periodically send requests to a specified URL and upload the response body to Simple Log Service as a log entry.
Collect Unity3D logs: Use WebTracking to collect Unity3D logs.
Integration with third-party ecosystems
-
Integrate with other open-source agents and protocols to collect logs: Simple Log Service accepts logs from a range of open-source tools and protocols:
Use the Loggie sink configuration to upload collected logs to Simple Log Service.
Use the Log4j2 Appender to send logs directly from your Java application.
Use Syslog-ng to collect logs and forward them over the Syslog protocol.
Use tools such as the Kafka Producer SDK, the Beats family of shippers, Collectd, Fluentd, Logstash, Telegraf, and Vector to collect logs and upload them to Simple Log Service over the Kafka protocol.
Integrate Telegraf monitoring data: Write monitoring data collected by Telegraf (from MySQL, Redis, Elasticsearch, ClickHouse, Kafka, Tomcat, and others) to LoongCollector using the InfluxDB protocol. LoongCollector then uploads the data to Simple Log Service.
Collect Open-Falcon monitoring data: Configure Transfer to upload Open-Falcon data to Simple Log Service. Your Open-Falcon version must include the InfluxDB support feature.
Ingest Prometheus monitoring data by using the Remote Write protocol: Enable the Remote Write feature in Prometheus to send monitoring data directly to Simple Log Service.
SDK collection
SDK-based collection: Simple Log Service provides SDKs for Java, Python, PHP, Node.js, C, Go, iOS, Android, and C++. Use these SDKs to write logs or write time-series data.
Data import
Import existing data from other applications or historical files into Simple Log Service for analysis.
Simple Log Service collects only incremental logs. To collect historical logs, import historical log files.
Data masking and encryption
Data encryption: Simple Log Service supports data encryption at rest using Key Management Service (KMS) and encryption in transit over HTTPS.
-
Data processing: Apply data masking at different stages of the data pipeline:
Process data during collection (processor plug-ins): Requires LoongCollector. Masking runs on the local server before transmission, consuming local server resources.
Process data during ingestion (ingestion processors): Masking runs after transmission but before storage, consuming Simple Log Service resources. Data is stored only after processing completes.
Process data after ingestion (data processing): Mask data that is already stored in Simple Log Service before outputting it to another Logstore or a third-party system.
Advanced collection scenarios
Collect logs cross-region: Your application runs in region A and your Simple Log Service project is in region B. This solution explains how to send logs from an ECS instance in region A to the project in region B.
Collect logs cross-account: Centrally collect logs from applications running under separate accounts into a single account for unified management.
Collect IoT and embedded device logs: IoT devices such as smart routers, TV sticks, Tmall Genie, and cleaning robots are numerous, widely distributed, difficult to debug, and constrained by hardware. This solution explains how to handle log collection from these devices.
High-availability solutions for client-side data collection: Two active-active, geo-redundant solutions to reduce the risk of single-cluster failures: a dual-write solution and a data replication and write-switching solution.
Collect logs from servers on an internal network: If your servers have no public network access, use a proxy to collect logs and forward them to Simple Log Service.
For more solutions, see Data collection best practices.
FAQ
-
Which network type should I select if I use Express Connect for access?
Select the Alibaba Cloud internal network — either a classic network or a Virtual Private Cloud (VPC).
-
How do I select a network type and an endpoint?
Network type: LoongCollector network transport types.
Endpoint: Find the service endpoint for your region.
-
Can I collect public IP addresses when collecting data from the public network?
Automatically add public IP addresses and arrival timestamps to logs.
-
How do I troubleshoot collection issues?
For more questions, see Data collection FAQ.