Many applications require real-time data collection and processing from public network sources, such as mobile clients, web pages, PCs, servers, hardware devices, and cameras. A traditional architecture typically uses frontend servers with Kafka to handle this process. The LogHub feature of Simple Log Service can replace this architecture and provides a stable, low-cost, elastic, and secure solution.
Scenarios
Public network data collection involves gathering data from sources such as mobile clients, external servers, web pages, and other devices. After the data is collected, it is used for applications such as real-time computing and data warehousing.
Solution 1: Frontend servers and Kafka
Because Kafka does not provide a RESTful protocol and is mostly used within clusters, you typically need an Nginx server as a public network proxy. Then, you can use Logstash or an API to write data from the public network to message middleware, such as Kafka, through the Nginx server. The required services are as follows:
Facilities | Quantity | Configuration | Function | Price |
ECS server | 2 units | 1-core 2 GB | Both ECS instances are used as frontend servers to provide services simultaneously. | CNY 108 per instance per month |
Server Load Balancer | 1 unit | Standard | Pay-as-you-go instance | CNY 14.4/month (lease) + CNY 0.8/GB (traffic) |
Servers with Kafka and ZooKeeper installed | 3 units | 1-core 2 GB | Write and process data. | CNY 108 per server per month |
Solution 2: Use LogHub
You can write data directly to the LogHub endpoint using a mobile SDK, Logtail, or Web Tracking JS. The required service is as follows:
Facilities | Function | Price |
LogHub | Real-time data collection | Less than CNY 0.18/GB. For more information, see Billing Rules. |
Solution comparison
Scenario 1: Collect 10 GB of data from approximately one million write requests per day. (The 10 GB represents the size after compression. The original data size is typically 50 GB to 100 GB.)
Solution 1
Server Load Balancer lease fee: 0.02 × 24 × 30 = CNY 14.4
Server Load Balancer traffic fee: CNY 0 (Inbound traffic is free, and no outbound public network traffic is generated)
ECS fee: 108 × 2 = CNY 216 (Assuming disk fees are zero)
Kafka ECS fee: Free, assuming it is shared with other services
Total: CNY 484.8/month
Solution 2
LogHub traffic fee: 10 × 0.18 × 30 = CNY 54
LogHub request fee: 0.12 × 30 = CNY 3.6
Total: CNY 57.6/month
Scenario 2: Collect 1 TB of data from approximately 100 million write requests per day.
Solution 1
Server Load Balancer lease fee: 0.02 × 24 × 30 = CNY 14.4
Server Load Balancer traffic fee: CNY 0 (Inbound traffic is free, and no outbound public network traffic is generated)
Server Load Balancer specification fee: 0.63 × 24 × 30 = CNY 453.6 (slb.s2.medium)
ECS fee: 240 × 2 = CNY 480 (ecs.g6.large)
ECS disk fee: CNY 4,800 (Peak traffic is twice the average. A single replica requires a write speed of 50 MB/s. Data is stored for 3 days, which requires 6 TB of capacity. The monthly fee for a standard SSD of this size is CNY 4,800.)
Kafka ECS fee: Free, assuming it is shared with other services
Total: CNY 6,696/month
Solution 2
LogHub traffic fee: 1,000 × 0.15 × 30 = CNY 4,500 (Tiered pricing)
LogHub request fee: 0.12 × 100 × 30 = CNY 360
Total: CNY 4,860/month
Comparison results
As shown in these two scenarios, Solution 2 is more cost-effective for public network data collection than Solution 1. In addition, Solution 2 provides the following advantages:
Elastic scaling: Handles traffic from megabytes to petabytes per day.
Rich access control: Enables you to manage read and write permissions using an access control list (ACL).
HTTPS support: Supports encryption in transit.
Free log shipping: Integrates with data warehouses at no extra cost.
Rich monitoring data: Provides clear insights into your business status.
SDK-based integration with upstream and downstream systems: LogHub provides multiple software development kits (SDKs) that you can use to connect to upstream and downstream systems. This allows for deep integration with open source products and other Alibaba Cloud products.
To experience this service, visit the Simple Log Service homepage.