`split_size` hint

How it works

MaxCompute reads a table's data and divides it into splits of the specified size. The number of map tasks is approximately:

map tasks = table data size / split_size

For example, a 10 GB table with split_size(256) creates about 40 map tasks. Dropping to split_size(64) creates about 160 map tasks — four times the parallelism.

MaxCompute honors the split_size hint for most tables. One exception: if the query optimizer uses a clustered table's bucketing properties for optimization, it ignores the hint for that table.

Tune the split size

A mismatch between task count and available resources is one of the most common causes of slow SQL jobs:

• Too many tasks for available resources -- Tasks queue for execution slots, and scheduling overhead increases. The job spends more time waiting than computing.

• Too few tasks for available resources -- Resources sit idle while a small number of tasks process large data chunks sequentially. The job takes longer than necessary.

Set split_size in multiples of 256 MB (e.g., 256, 512, 1024) for optimal performance.

Examples

All examples place the hint immediately after the table name in the FROM clause.

Set the split size to 1 MB when reading src. The src table data is divided into 1 MB chunks, each processed by a separate map task. For a 10 GB table, this creates about 10,240 map tasks instead of the default 40:

SELECT a.key
FROM src a /*+split_size(1)*/
JOIN src2 b ON a.key = b.key;

Increase the split size to 512 MB to reduce task count on a large table. For a 10 GB table, this cuts the task count from 40 to about 20:

SELECT *
FROM large_table /*+split_size(512)*/
WHERE dt = '2024-01-01';

Behavior with clustered tables and multiple hints

• Clustered tables: The query optimizer ignores the split_size hint for a clustered table if it uses the table's bucketing properties for optimization.

• Multiple hints on the same table: If a query reads the same table multiple times with different split_size hints, MaxCompute uses the smallest value. For example:

  • split_size(1) and split_size(10) on the same table -- MaxCompute uses 1.

  • split_size(1) on one reference and no hint on another -- MaxCompute uses 1 (not the 256 MB default).

Reference