Reduce the time required to generate execution plans-MaxCompute(MaxCompute)-阿里云帮助中心

This topic explains why generating a physical execution plan can be time-consuming and provides solutions to optimize the process.

Symptoms

When a physical execution plan for a job is being generated, the status in SubStatusHistory in Logview is 1235 SQLTask is splitting data sources or 1240 SQLTask is generating execution plan.

Causes and solutions

The following table describes the possible causes of this issue and their corresponding solutions.

Cause	Description	Solution
Excessive number of partitions	When MaxCompute compiles a job, it processes the metadata for each partition read. It determines how to split the data and how much data to assign to each compute unit. This information is then written to the execution plan. A large number of partitions creates significant overhead, increasing compilation time.	Optimize your SQL query to reduce the number of partitions read. For example, use partition pruning to filter out unnecessary partitions, or split a large job into multiple smaller jobs.
Excessive number of small files	During job compilation, MaxCompute determines how to split data for each compute unit based on file sizes within a partition. A large number of small files increases the overhead of this splitting process, leading to longer compilation times. Small files are typically generated for two main reasons: Improperly using Tunnel to upload data, such as creating a new upload session for each record. Inserting data into a partitioned table creates a new file in the corresponding partition's storage directory.	Use the TunnelBufferedWriter interface to upload data. This simplifies the process and avoids creating small files. Run the `alter table tablename [partition] merge smallfiles;` command in your project. This instructs MaxCompute to automatically compact small files in the specified table or partition.

Note This performance impact is significant only with tens of thousands, or even hundreds of thousands, of partitions or small files. Dozens or hundreds of them typically do not cause noticeable delays.