Data transformation organizes and enriches your log data, reducing time and labor costs. With proper configuration, you can further reduce storage costs by 12%–30%.
Typical configuration
Based on the transformation principles and performance guide, simplify your collection plan: ingest data into one or more Logstores, then use data transformation to distribute it. Configure the storage period and index for each destination Logstore as needed.
Cost factors
The Pay-by-feature billing items and show that cost depends on three factors:
-
Daily data ingestion volume.
-
Data retention period.
-
Index configuration.
The following two examples show how to optimize costs by adjusting storage structure and content.
Optimize storage structure
Baseline: an application writes 100 GB/day, stored for 30 days with a full-text index. Monthly cost: approximately CNY 2,248.
If only 20% of logs (such as operation and error logs) need 30-day retention while the rest only need 7 days, use the following transformation plan:
-
Create a source Logstore to store data for 3 days without an index.
-
Create destination Logstore 1 to store operation and error logs for 30 days with an index.
-
Create destination Logstore 2 to store general logs for 7 days with an index.
Monthly cost drops to approximately CNY 1,672, a saving of approximately 25%.
For a 60-day baseline, storing 20% of important logs for 60 days and the rest for 7 days saves 12% while doubling the retention of critical logs.
Optimize storage content
Using the same baseline (100 GB/day, 30-day retention, full-text index), the monthly cost is approximately CNY 2,248.
Example raw log (1021 bytes):
__source__: 192.0.2.0
__topic__: ddos_access_log
body_bytes_sent: 3866
cc_action: none
cc_blocks:
cc_phase:
content_type: text/x-flv
host: www.example.com
http_cookie: i1=w1;x2=q2
http_referer: http://www.example.com
http_user_agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.0.2.1 Safari/537.36
http_x_forwarded_for: 192.0.2.2
https: true
isp_line: BGP
matched_host: www.example.com
method: GET
real_client_ip: 192.0.2.3
remote_addr: 192.0.2.4
remote_port: 48196
request_length: 2946
request_method: GET
request_time_msec: 78920
request_uri: /request/nvwlvvkhw
server_name: www.example.com
status: 502
time: 2019-07-22T17:40:26+08:00
ua_browser: mozilla
ua_browser_family:
ua_browser_type:
ua_browser_version: 9.0
ua_device_type:
ua_os: windows_7
ua_os_family:
upstream_addr: 192.0.2.4:80
upstream_ip: 192.0.2.5
upstream_response_time: 0.858
upstream_status: 200
user_id: st0s2b5To keep only important fields indexed for 30 days and discard the rest after 3 days, use the following transformation plan:
-
Create a source Logstore with a 3-day data retention period and no index.
-
Create a destination Logstore with a 30-day data retention period and an index for the required fields.
If the processed log is about 60% of its original size, monthly cost drops to approximately CNY 1,579, a saving of approximately 30%.
After transformation, the log shrinks from 1021 bytes to 618 bytes:
__source__: 192.0.2.0
__topic__: ddos_access_log
body_bytes_sent: 3866
content_type: text/x-flv
host: www.example.com
http_referer: http://www.example.com
ua_browser: mozilla
ua_browser_family:
ua_browser_type:
ua_browser_version: 9.0
ua_device_type:
ua_os: windows_7
http_x_forwarded_for: 192.0.2.2
matched_host: www.example.com
method: GET
real_client_ip: 192.0.2.3
request_length: 2946
request_uri: /request/nvwlvvkhw
status: 502
upstream_addr: 192.0.2.4:80
upstream_ip: 192.0.2.5
upstream_response_time: 0.858
upstream_status: 200
user_id: st0s2b5