Handle data expansion
Updated at:
Data expansion occurs when a Fuxi task produces far more output data than it receives as input, which degrades performance. Learn common causes and how to resolve them.
Problem description
In Logview, the output data volume of a Fuxi task may be significantly larger than its input. You can verify this by checking the I/O Record and I/O Bytes attributes of the Fuxi task.
For example, a 1 GB input can expand to 1 TB after processing. Processing 1 TB of data on a single instance significantly reduces performance.
Causes and measures
The following table lists possible causes and recommended measures.
| Cause | Description | Measure |
| Bug in code | The code is defective. Examples:
|
Fix bugs in the code. |
| Improper aggregation operations |
Standard aggregation operations rarely cause issues because they produce small amounts of intermediate data with low computational complexity. However, certain aggregation operations, such as
collect_list and median, must retain all intermediate data. When combined with other aggregation techniques, this can lead to data expansion. Examples:
|
Avoid aggregation operations that cause data expansion. |
Improper JOIN operations |
For example, the left table of a JOIN operation contains a large amount of population data, and the right table is a dimension table, which records hundreds of rows of data for each gender. If you perform the JOIN operation on the data based on genders, the size of data in the left table may expand to hundreds of times larger than the original size. |
To prevent data expansion, aggregate the right table before joining it with the left table. |
Is this page helpful?