Automatic storage optimization for Lance tables
Data Lake Formation (DLF) provides automatic storage optimization for Lance tables, which runs data compaction and index maintenance on your behalf. After you enable these features, DLF checks for data changes at the configured interval, merges data fragments, processes soft-deleted records, and maintains indexes for new data. This topic describes how to configure automatic storage optimization in the DLF console and how to view the results.
Data compaction and index maintenance
Automatic storage optimization for Lance tables consists of two features:
| Feature | Purpose | Use cases |
| Automatic data compaction | Merges eligible data fragments to reduce the number of fragments, and processes soft-deleted records based on the policy. | Frequent small-batch writes, updates, or deletions |
| Automatic index maintenance | Builds vector, scalar, or full-text indexes, adds index coverage for new data, and merges index segments when the conditions are met. | Continuous writes that require efficient vector searches, conditional filtering, or full-text searches |
You can enable automatic data compaction and automatic index maintenance separately or together. The following behavior applies to both features:
-
Default state — Both features are disabled by default. Saving the check interval and maintenance parallelism does not turn on either switch.
-
Shared configuration — Both features use the table-level maintenance configuration of the Lance table.
-
Execution order — When both features are enabled, DLF completes data compaction first and then maintains indexes based on the compacted data.
Automatic storage optimization applies to the main dataset. It does not include expired historical version cleanup or orphan file cleanup. Completing data compaction does not mean that files referenced by historical versions are immediately released.
Console entry points
The following table lists the console page where you configure each setting:
| Configuration | Page location |
| Check interval and maintenance parallelism | Storage Optimization > Storage Optimization Management > Edit |
| Automatic data compaction switch, compaction parameters, and execution history | Storage Optimization > Compaction |
| Index creation, field parameters, and the table-level Automatic Index Building switch | Indexes |
| Task-level index building policy | Storage Optimization > Index > Edit |
Prerequisites
-
The Lance Catalog is running, and you have completed the first write to the target table by using a client. For instructions on creating and managing the Catalog, see Manage and use a Lance Catalog. For instructions on reading and writing Lance tables with Python or Daft, see Work with DLF Lance using Python or Work with DLF Lance tables using Daft.
-
The current user has the DLF administrator role and the permissions required to call maintenance APIs. If you use a RAM user, grant the user the DLF API and data permissions. For instructions, see Configure permissions.
-
For a vector index, the target field contains fixed-length Float32 vectors, and the field dimension matches the data.
Configure automatic storage optimization
Automatic storage optimization is configured in three steps. Step 1 is required for every Lance table that you want to optimize, because it sets the table-level maintenance configuration shared by both features. Step 2 and Step 3 are independent of each other: complete Step 2 to enable automatic data compaction, complete Step 3 to configure automatic index maintenance, or complete both.
Step 1: Configure table-level maintenance settings
-
Log on to the DLF console.
-
In the left-side navigation pane, choose Catalogs, open the target Lance Catalog, and then choose Namespace > Tables.
-
Switch the table to the latest version on the main branch, and then click the Storage Optimization tab.
-
In the Storage Optimization Management section, click Modify, configure the following parameters, and save the configuration.
| Parameter | Default | Description |
| Check Interval (Seconds) | 3600 | The maintenance check interval between two rounds of maintenance. Enter a positive integer and adjust the value based on the write frequency. |
| Maintenance Parallelism | 1 | The number of maintenance tasks that can run in parallel. Enter a positive integer. For large datasets, increase the value gradually based on resource usage. |
The check interval is the wait time after a round of maintenance completes successfully. It is not a schedule that runs at fixed clock times. If the previous round is still running, DLF does not start another round for the same table. When no data needs to be processed, reaching the check interval alone does not create a new version.
Step 2: Enable automatic data compaction
-
On the Storage Optimization tab, click the Compaction subtab.
-
In the Compaction Strategy section, click Modify, configure the following parameters, and save the configuration.
-
Turn on Automatic Compaction.
| Parameter | Default | Description |
target_rows_per_fragment |
1000000 | The target number of rows in each compacted data fragment. Adjust the value based on the size of each row. |
max_bytes_per_file |
90 G | A soft limit on the size of each output file. The console accepts MB, G, or a byte count, and uses base-1024 conversion, such as 256 MB or 1.5 G. The actual file size may exceed this value. |
materialize_deletions |
Enabled | Specifies whether to include fragments whose soft-deletion ratio exceeds the threshold in compaction. |
materialize_deletions_threshold |
0.1 | The soft-deletion ratio threshold. A value of 0.1 means 10%. A value of 0 includes a fragment whenever it has soft-deleted records. A value greater than or equal to 1 prevents the deletion ratio from triggering processing. |
If a row contains large text, vectors, or multi-modal content, set the target row count based on the row size. The target file size controls the output. It is not a task memory limit, and a large existing file alone does not automatically trigger compaction.
Automatic data compaction runs in the background at the check interval. To confirm that compaction has run, see View data compaction records.
Step 3: Configure automatic index maintenance
After you enable table-level automatic index building, DLF uses the set of index names in the maintenance configuration. Existing ordinary indexes that are not included in this configuration are removed during subsequent maintenance. Before you enable automatic index building, verify the name, field, and type of each existing index against the maintenance configuration — an index is not included solely because its field and type match.
-
Click the Index tab of the target table, and then click Create Index.
-
Select an index field and an index type. For supported types and build parameters, see Lance index types and build parameters.
-
For a vector index, the page reads and displays the vector dimension from the field. Select the distance type and configure Build Parameters (Optional) as needed.
-
Click Save Configuration. If table-level index building is not enabled, the page prompts Overall Index Building Must Be Enabled. After you confirm, all configured indexes are built automatically.
-
Verify that the table-level Automatic Index Building switch is turned on and that each index is listed on the Index tab with its coverage. Index building runs in the background and does not complete immediately when you save.
Automatic index building is controlled at two levels. The table-level Automatic Index Building switch on the Index tab applies to all indexes on the table, and the Overall Index Building Must Be Enabled prompt refers to this same switch. The index-level Automatic Building switch in each index row applies to that index only.
To confirm how much data each index covers, see View index coverage.
Configuration example
Consider a document retrieval table with the following fields:
| Field | Type | Purpose |
id |
Int64 | Document identifier |
category |
String | Document category |
text |
String | Document content |
embedding |
Fixed-length Float32 vector | Semantic vector of the document |
You can configure automatic storage optimization for this table as follows:
-
Data compaction — Start with the default target row count. If each row is large, reduce the row count and set an appropriate output file size.
-
Semantic search — Configure a vector index such as IVF_PQ for
embedding. For example, if the vector dimension is 768, set the number of PQ subvectors to 96 and configure the number of IVF partitions based on the data scale. -
Category filtering — Configure a BITMAP index for
categorywhen it has few distinct values. -
Keyword search — Configure a full-text index for
text, and issue full-text queries by using a client.
For small tables, reduce the number of partitions or wait until enough data has accumulated before you build the vector index.
View optimization results
View data compaction records
To view compaction commits, choose Storage Optimization > Compaction > Execution History. The execution history lists:
-
The Compact version ID and the version commit time.
-
The number of files involved in compaction and the number of files after compaction.
-
The total size of the participating files before and after compaction.
The page checks the most recent 100 versions and displays only the confirmed compaction commits among them. Jobs that have not committed or have failed are not listed.
File counts and sizes cover the data files and coverage files that this compaction actually replaced, and they exclude index files and deletion files. Some metrics may display No Data when metadata is missing.
View index coverage
On the Indexes tab, view the index type, field, total row count, covered row count, uncovered row count, and coverage percentage of each index.
| Status | Meaning |
| Covered | The index covers the data in the selected version. |
| Partially Covered | Some data has not been indexed, such as recently appended data. |
| Not Covered | The index does not yet cover the current data. |
| No Data | The table is empty, or metadata is insufficient to display coverage. |
During continuous writes, background index building lags behind newly added data. A temporary decrease in coverage is normal. Refresh the page after the next round of maintenance completes. To determine whether an index covers a historical version, check the results displayed for that version.
Modify or disable automatic storage optimization
-
Modify maintenance or compaction parameters — Edit and save the corresponding policy. The new configuration applies to subsequent maintenance jobs. Running jobs continue to use the original configuration.
-
Modify index field parameters — On the Indexes tab, click Configure in the corresponding index row. This option is available when automatic building is enabled for that index.
-
Modify the task-level index building policy — Choose Storage Optimization > Index > Edit.
-
Disable data compaction — Turn off Automatic Compaction to stop subsequent automatic compaction.
-
Disable all index maintenance — On the Indexes tab, turn off the table-level Automatic Index Building switch. Index definitions are retained, subsequent index building and cleanup stop, and committed indexes are retained.
-
Disable building for one index — Turn off the index-level Automatic Building switch in that index row. The index is removed from the maintenance configuration. If the table-level Automatic Index Building switch remains enabled, this index may be removed during subsequent maintenance. To pause building while retaining all existing indexes, turn off the table-level switch instead.
-
Resume maintenance — Turn the appropriate switches on again. DLF continues to check and process data based on the current configuration.
Turning off a switch does not immediately interrupt tasks that have already started. Running tasks may still commit results. Disabling a feature does not automatically remove its maintenance task registration.