Accelerate queries with business key aggregation
Business key aggregated storage is a LogStore storage enhancement in Simple Log Service that writes the logs of one business key, such as a trace ID or a session ID, into the same ShardGroup. Otherwise, those logs spread across all shards, and each point query scans the entire LogStore, which gets slower as data grows. Enable the feature to prune irrelevant shards at query time, without changing your write method or query syntax.
Business key aggregated storage is currently released only in the ap-southeast-8 region. Confirm the region of your LogStore before you plan a business key policy.
Scenarios
Business key aggregated storage reduces the number of shards that a query scans when your query conditions consistently include equality conditions on the same business key fields. The following table lists typical scenarios and the recommended business key combinations.
Scenario | Recommended business key | Benefit |
Trace troubleshooting |
| Narrows the shard scan range when you query a single call chain, so trace logs return faster. |
Agent application observability |
| Stores the model calls, tool calls, retrievals, memory operations, and anomalies of one agent run together, which speeds up trace queries. |
User session analysis |
| Returns the complete behavior logs of one user or one session quickly. |
Order and transaction troubleshooting |
| Locates order exceptions, payment failures, and transaction chain issues quickly. |
Host and client access logs |
| Groups access logs from the same source, client, or domain name for troubleshooting. |
IoT and device logs |
| Returns the operation, alert, and connection logs of a single device quickly. |
Security audit and attack tracing |
| Tracks the behavior logs of a specific IP address, account, AccessKey, or user. |
Unsuitable scenarios
In the following scenarios, business key aggregated storage brings limited benefit or introduces write hotspots. Evaluate your query patterns and data distribution before you enable it.
Scenario | Reason |
Queries rarely contain a fixed business key | Simple Log Service cannot prune shards by business key, so the benefit is limited. |
The LogStore has too few shards | Each ShardGroup contains too few writable shards, which limits disaster recovery capability. |
Business key values are heavily skewed | Values of the business key fields are distributed unevenly, which causes write skew. For example, if a single |
If your LogStore matches one of these scenarios, check the writable shard configurations in Prerequisites and the value distribution and coverage of your candidate business key fields before you decide whether to enable the feature.
Prerequisites
Create a project and a LogStore. For instructions, see Manage projects and Manage LogStores.
Plan the business key fields, such as
trace_id,session_id,user_id, ororder_id. Choose fields that exist consistently in your logs, are frequently used as query filters, and have high selectivity.Check the number of writable shards in the target LogStore. 8 writable shards is the recommended configuration, because each ShardGroup then keeps about 2 writable shards and a reasonable disaster recovery capability. You can also enable the feature with 4 writable shards, but each ShardGroup keeps only about 1 writable shard and disaster recovery capability is limited. To view and adjust the number of shards, see Manage shards.
Limits
The following limits apply to business key aggregated storage:
Supported regions — Business key aggregated storage is currently released only in the ap-southeast-8 region. Other regions are not supported.
Supported resources — Only LogStores support business key aggregated storage.
Number of business key fields — You must configure at least one and at most two business key fields.
Field order — Field order participates in route computing. Changing the field order is treated as a policy change.
Missing or empty values — If a business key field does not exist or its value is empty, an empty string is used in the hash calculation.
Number of ShardGroups — The number of ShardGroups must be a power of 2, such as 4, 8, 16, 32, or 64.
Effect of a policy change — A new policy takes effect after a short delay and applies only to logs that are written after it takes effect. Simple Log Service does not redistribute historical data.
Configure business key aggregated storage
Turn on business key aggregated storage when you create a LogStore, or configure it on the properties page of an existing LogStore. Both entry points expose the same configuration items, and the business key fields and the number of ShardGroups have identical meanings in both.
Enable business key aggregated storage on a new LogStore
Log on to the Simple Log Service console.
In the Project list, click the target project.
On the Log Storage > Logstores tab, click the + icon.
On the Create Logstore page, find the Business key aggregated storage configuration item.
Turn on the Business key aggregated storage switch, and then configure Business key fields and Number of ShardGroups. To let the console fill in a suggested number of ShardGroups based on the current number of writable shards, click Use recommended value.
Click OK.
Change or turn off business key aggregated storage on an existing LogStore
Log on to the Simple Log Service console.
In the Project list, click the target project.
On the Log Storage > Logstores tab, move the pointer over the target LogStore and select Modify.
In the left-side navigation pane of the LogStore properties page, click Business key aggregated storage.
Configure the settings in one of the following ways:
To enable the feature or change its settings, turn on the Business key aggregated storage switch and configure Business key fields and Number of ShardGroups. Changing the business key fields, their order, or the number of ShardGroups is a policy change, and the new policy applies only to logs that are written after the change takes effect.
To turn off the feature, turn off the Business key aggregated storage switch. New logs are then no longer aggregated by business key, and business key pruning no longer applies to them.
Click Save.
After you turn off business key aggregated storage, the logs that are already written remain in their original shards. Simple Log Service does not migrate or redistribute them, so turning off the feature causes neither data loss nor data rewriting.
Parameters
The following table describes the business key aggregated storage configuration items on the LogStore creation page and on the LogStore properties page.
Parameter | Description |
Business key aggregated storage | Specifies whether to enable business key aggregated storage. After you turn on the switch, Simple Log Service writes the logs that share a business key into the same ShardGroup. |
Business key fields | The fields that determine which ShardGroup a log is written to. Configure one or two fields. |
Number of ShardGroups | The number of logical groups used for business key aggregation. The value must be a power of 2. (Recommended) Use the value that the console suggests. |
Use recommended value | Calculates the recommended number of ShardGroups based on the current number of writable shards. |
Shard data range balance status | Indicates whether the shard data ranges of the current LogStore are balanced, which helps you decide whether a data rerange is needed. |
Data rerange | Redistributes shard data ranges to fix unbalanced shard data ranges. Before you run a data rerange, check whether logs are written with a specified hash key, for example through the SDK. |
Choose the number of ShardGroups
The number of ShardGroups determines the granularity of query pruning: the more ShardGroups you configure, the fewer shards each query scans, but the fewer writable shards each group keeps, which lowers disaster recovery capability and increases hotspot risk. The console suggests a value based on the current number of writable shards in the LogStore, and the suggested value balances query pruning efficiency, disaster recovery capability, and hotspot risk.
Current writable shards | Recommended ShardGroups | Description |
4 | 4 | About 1 writable shard per group. Queries can be pruned, but disaster recovery capability is limited. |
8 | 4 | About 2 writable shards per group. Suitable when you start using the feature. |
16 | 4 or 8 | Choose the value based on your query acceleration requirements. |
32 | 8 or 16 | Suitable for high-frequency point queries. |
64 | 16 or 32 | Suitable for large-scale point queries, trace troubleshooting, and session analysis. |
Click Use recommended value to apply the suggested value. Within a range, the higher value gives finer query pruning, and the lower value keeps more writable shards in each ShardGroup.
Query hit and fallback conditions
After you enable business key aggregated storage, a query hits business key pruning when it contains equality conditions on all business key fields. Simple Log Service calculates the target ShardGroup from the business key values and scans only the shards of that group instead of all shards of the LogStore, which reduces the volume of data that participates in the computation.
For example, the business key fields are tenant_id and agent_run_id, and the query contains the following conditions:
tenant_id = 'tenant-a' AND agent_run_id = 'run-001'Simple Log Service resolves a single target ShardGroup from these values and scans only the shards of that group.
In the following cases, a query cannot hit business key aggregated storage and falls back to a full shard scan:
Missing business key field — The query does not contain one of the business key fields.
Non-equality match — A business key field is matched by a fuzzy match, a range match, or a regular expression instead of an equality condition.
Ambiguous business key value — The query expression does not resolve to a unique business key value.
Policy change within the time range — The query time range spans a policy change or covers logs that were written before the current policy took effect, so a single policy cannot be used for pruning.
Feature not enabled — Business key aggregated storage is not enabled for the LogStore.
Billing
Business key aggregated storage does not change the billing method of a LogStore. After you enable it, write, storage, query, and analysis fees are still charged based on the current billing method of the LogStore.
When a query hits business key pruning, the scanned data volume and the query duration decrease, and the related query and analysis fees can decrease as well. The actual benefit depends on whether the query contains the complete business key, the number of ShardGroups, the data distribution, and the query time range.
FAQ
Why are my queries not noticeably faster after I enable business key aggregated storage?
First check whether your queries meet the hit conditions described in Query hit and fallback conditions. If the queries do meet the hit conditions, check the following configuration causes:
The business key fields are missing from a large proportion of logs, so the aggregation effect is weak.
The number of ShardGroups is too small, so few shards are pruned.
What happens when a business key field does not exist?
Simple Log Service uses an empty string in the hash calculation, so all logs with a missing or empty value are written to the same ShardGroup. If the business key fields are missing from a large number of logs, write skew occurs. Check the coverage of the business key fields before you enable business key aggregated storage.
When do I need a data rerange?
Consider a data rerange when Shard data range balance status indicates that the shard data ranges are unbalanced. Before you run it, check whether logs are written with a specified hash key, for example through the SDK. If your application keeps writing with a specified hash key, hotspots may persist after the rerange.