Simple Log Service delivers logs to security information and event management (SIEM) systems, enabling you to import all regulatory, audit, and other related logs from Alibaba Cloud into your security operations center (SOC).
Terms
-
SIEM: Security information and event management system, such as Splunk or IBM QRadar.
-
Splunk HEC: Splunk HTTP Event Collector, an interface that receives logs over HTTP or HTTPS.
Deployment suggestions
-
OS: Linux, such as Ubuntu x64.
-
Hardware:
-
CPU: 2.0+ GHz × 8 cores.
-
Memory: 16 GB, 32 GB recommended.
-
Network: 1 Gbps network interface controller (NIC).
-
Disk space: 2 GB minimum, 10 GB+ recommended.
-
-
Network parameters:
Bandwidth from your environment to Alibaba Cloud must exceed the log generation rate. Otherwise, logs cannot be consumed in real time. Example: 1 TB of raw logs per day at a uniform rate with 2x peaks, compressed at 5:1, requires approximately 4 MB/s (32 Mbps).
-
Python: This guide uses Python. For Java, use Consume logs using consumer groups.
Python SDK
-
Use a standard CPython interpreter.
-
Install the SLS SDK for Python: python3 -m pip install aliyun-log-python-sdk -U.
-
Usage details: User Guide.
Consumer library
The consumer library is an advanced log consumption method that uses consumer groups to abstract and manage consumers. It handles load balancing and failovers automatically, so you can focus on business logic.
A Logstore contains multiple shards. The consumer library assigns shards to consumers based on these principles:
-
Each shard is assigned to only one consumer.
-
A single consumer can be assigned multiple shards.
When a new consumer joins a consumer group, shard assignments are rebalanced automatically. The assignment principles remain the same, and the process is transparent to the user.
The consumer library saves checkpoints, enabling programs to resume from the last position after a failure without duplicate consumption.
Spark Streaming, Storm, and Flink Connector are all built on the consumer library.
Delivery methods
Build a program that uses SLS consumer groups for real-time consumption and sends logs to your SIEM over HTTPS or Syslog.
-
HTTPS: Deliver logs to a SIEM.
-
Syslog: Deliver logs to a SIEM over Syslog.