DLF provides automatic storage optimization for Iceberg tables, including compaction and snapshot expiration. Once enabled, DLF schedules and runs optimization tasks in the background — no job maintenance is required. This helps reduce storage costs and improve query performance.
Prerequisites
An Iceberg table has been created in DLF. For more information, see Manage Iceberg tables.
Limitations
Iceberg V3 tables and orphan file cleanup are not supported for automatic optimization.
For these scenarios, you can manually run storage maintenance jobs using EMR Serverless Spark. For more information, see Iceberg table maintenance.
Enable storage optimization
Log on to the DLF console
In the left-side navigation pane, choose Catalogs, and then navigate to the target Catalog > Database > Table.
On the target Iceberg table, click the Storage Optimization tab.
Turn on the Intelligent Storage Optimization toggle.
Once enabled, DLF automatically schedules and runs compaction and snapshot expiration tasks. Storage optimization is disabled by default.
Compaction
Compaction merges small files into larger files close to the target size, reducing the total file count. This lowers metadata overhead and improves query performance.
Execution: After storage optimization is enabled, DLF automatically schedules and runs compaction in the background. No manual job maintenance is required.
Target file size: Controlled by the table property
write.target-file-size-bytes. The default value is 512 MB.
Snapshot Expire
Snapshot expiration runs automatically as part of storage optimization tasks. It removes historical snapshots that exceed the retention period to free up storage space.
The snapshot retention policy follows Iceberg standard table properties:
Description | Default value |
Snapshot retention period | 5 days |
Minimum number of snapshots to retain | 1 |
You can configure the snapshot retention period under Snapshot Retention Rule on the Snapshot Expire tab.