Brute-force scan

Associated administration item: DG-C-44

A brute-force scan is caused by the improper use of partitions or filter conditions. This results in a full table scan or a scan of a very large data volume during computation, which wastes significant computing resources. The administration item for brute-force scans is designed to expose these code issues and guide logic optimization to reduce computing resource consumption.

Applicable data source

• MaxCompute

Rule Evaluation

A scan is identified as a brute-force scan if it meets either of the following conditions:

• More than 90 partitions in a table are queried.

• The total storage size of the queried partitions is greater than 90 GB.

Common scenarios that trigger a brute-force scan include the following:

• Reading a large partitioned table without adding partition constraints, which causes a full table scan.

• Using partition filter conditions, but the time span of the data access is too large.

• Using incorrect partition filter conditions, which invalidates the constraints.

Resolution guide

The following methods are commonly used to resolve brute-force scans:

• Add partition filter conditions to reduce the number of scanned partitions, or create a smaller intermediate table and then scan its historical partitions to reduce the amount of scanned data.

• Move the partition constraints up into a subquery to prevent the brute-force scan.

• For tables with many partitions that require daily scanning and computation for business purposes, create an intermediate table to perform a daily aggregation.

• Check the partition filter conditions and correct any encoding errors that invalidate the partition constraints.

Notes

Ensure that resolving a brute-force scan does not affect business operations. If business requirements require access to data over a large time span, add the operation to a whitelist or create an aggregate table for downstream access.