DLF Lance overview and how it works
DLF Lance Catalog provides a fully managed catalog and storage for AI and multimodal applications. Applications can read and write data using the open source Lance SDK. The catalog is designed to handle hundreds of millions of tables. DLF Lance also provides managed compaction, index building, and high-performance caching.
This topic compares open source Lance with DLF Lance and explains how clients, compute engines, and DLF work together to access tables, refresh credentials, commit versions, optimize storage, and serve reads through the cache.
DLF Lance vs. open source Lance
DLF Lance supports continuous writes, vector search, and multimodal data processing. Applications can continue to use the native Lance SDK. Spark and Ray can process data and prepare training datasets. StarRocks can run hybrid searches and SQL analytics on the same Lance data for use cases such as knowledge retrieval for retrieval-augmented generation (RAG), multimodal search, and training data filtering.
Aspect | Open source Lance | DLF Lance |
Catalog and file layout | The Directory Catalog organizes tables and can discover them by scanning directories or consulting a directory inventory. | Organizes individual table paths into nested namespaces and stores namespace and table metadata separately. |
Table lookup and browsing at scale | The catalog implementation handles table lookup and listing. It can use a directory inventory and indexes to locate tables. | Resolves table identifiers to fixed paths and uses storage-layer pagination to browse tables by namespace. Accessing one table does not require loading the entire catalog. |
Read caching | The SDK provides in-process metadata and index caches that can be reused within a session. | Retains the SDK caches. When high-performance caching is enabled, the SDK uses the Object Storage Service (OSS) protocol to access the cache service. The service handles cache hits, reads from origin storage, and cache fills. |
Authentication and credential refresh | You can configure storage access credentials in the SDK. When connected to a REST catalog that issues credentials, the SDK can refresh them automatically based on their expiration. | The catalog handles authentication and issues temporary OSS credentials. The community SDK checks and refreshes the credentials automatically. |
Compaction | Provides a compaction API. Applications integrate the API and schedule compaction. | Automatically checks and compacts fragments based on table-level settings and makes the maintenance results available for inspection. |
Index building and maintenance | Provides APIs to create and optimize indexes. Applications schedule maintenance. | Automatically builds and maintains vector, scalar, and full-text indexes based on table-level settings. It indexes data that is not yet covered and consolidates index segments. |
Format and version semantics | Uses the open Lance format. The SDK reads snapshots, commits versions, and handles concurrent write conflicts. | Uses the community format and commit mechanism, with data stored in OSS. Both application writes and background maintenance publish their results as Lance versions. |
Management and monitoring | Provides APIs and diagnostics for inspecting versions, files, and indexes. Applications integrate them as needed. | The console shows table schemas, versions, data files, fragment distribution, index coverage, and maintenance history in one place. |
Multiple engines and AI applications | The Lance SDK and engine connectors can read, write, and search data. Applications coordinate data processing, search, and analytics. | The catalog provides a single place to discover tables and obtain access credentials. Spark and Ray process the data. StarRocks searches existing Lance vector indexes and uses the results in SQL joins and aggregations. |
Table access and maintenance workflow
DLF Lance Catalog locates a table by its identifier and returns access parameters. The open source Lance SDK reads and writes application data and commits versions. DLF runs automated maintenance in the background. Each table has its own directory in OSS, organized by namespace.
The following diagram shows how clients access tables, how DLF maintains them, and how OSS stores their files.
Blue indicates catalog access. Green indicates application I/O and automated maintenance. The directories in the diagram correspond to OSS object prefixes. For example, in the table directory team_a/train/images.lance/, the data/, _indices/, and _versions/ subdirectories store data files, index files, and version manifests, respectively.
Authentication and credential refresh
To access a Lance table, the SDK first gets the table location and storage access parameters from Lance Catalog. It then uses the temporary OSS credentials in the response to access the underlying data.
Resource | Authentication | Credential management |
Lance Catalog |
| Get |
OSS data | Temporary OSS credentials | The SDK obtains credentials through |
Access Lance Catalog
The Lance Catalog REST API uses bearer token authentication. The client obtains lance.bearer-token from the catalog configuration and sends it as Bearer <token> in the Authorization header. When the application opens a table with lance.dataset(namespace_client=..., table_id=...), the SDK calls DescribeTable to get the table location (location) and storage access parameters (storage_options).
Set the DLF_LANCE_ENDPOINT and DLF_LANCE_TOKEN environment variables to lance.endpoint and lance.bearer-token from the DLF Lance Catalog configuration, respectively. For instructions on obtaining these values, see Manage and use a Lance Catalog.
Set table_id to the identifier of an existing table in your catalog. This example uses the events table in the demo namespace.
import os
import lance
from lance.namespace import RestNamespace
catalog = RestNamespace(**{
"uri": os.environ["DLF_LANCE_ENDPOINT"],
"headers.Authorization": "Bearer " + os.environ["DLF_LANCE_TOKEN"],
})
table = lance.dataset(
namespace_client=catalog,
table_id=["demo", "events"],
)Access DLF-managed data
DLF manages Lance table data in OSS. The SDK caches the storage_options returned by DescribeTable and uses the temporary OSS credentials in those options to access the data. Before subsequent OSS requests, the SDK checks the following parameters to determine whether to refresh the credentials.
expires_at_millisindicates when the credentials expire.refresh_offset_millisspecifies how many milliseconds before expiration the refresh window opens. For example, with a value of"36000000", the window opens 10 hours before expiration.
The first OSS access after the refresh window opens triggers a refresh. The SDK calls DescribeTable again, updates the cached access parameters for the current dataset, and resumes the OSS request. Credential refresh is automatic, and the current dataset stays open.
The following sequence diagram shows how the SDK obtains, reuses, and refreshes temporary OSS credentials.
In this deployment example, the credentials obtained at 08:00 expire at 08:00 the next day. With a 10-hour refresh offset, the SDK reuses the cached credentials before 22:00 and refreshes them before its first OSS request at or after 22:00. The actual refresh time depends on the expiration returned by the server and when the client accesses OSS.
Version commits and concurrency control
Write files and publish versions
Lance uses a manifest to record the data files that each version references. The SDK writes the data files first, then uses a conditional create operation in OSS to publish the new version manifest atomically. Publishing the manifest commits the version and makes the new data readable in that version.
For example, even after the SDK uploads file C to OSS, v7 still references only A and B. Once the SDK publishes v8, reading v8 returns A, B, and C.
Concurrent appends and commit retries
When two datasets append data starting from the same version, read_version records the starting version for each dataset. The following diagram shows two compatible appends, with time moving from top to bottom. Both writers start from v7 and attempt to create v8. Writer 1 commits first. Writer 2's commit fails because v8 already exists. The SDK reads v8, checks that the two appends are compatible, merges the manifests, and retries the commit as v9.
Each version retains its own snapshot. The v7 snapshot contains A and B. The v8 snapshot adds C from writer 1, and the v9 snapshot also includes D from writer 2.
Automatic storage optimization
DLF compacts data and maintains indexes based on table-level settings to reduce fragmentation and improve index coverage. You can enable each feature separately. When both are enabled, DLF compacts the data and commits a version first. It then maintains indexes on the compacted data and commits another version.
Maintenance workflow and version publishing
The following diagram shows the workflow when both compaction and index maintenance are enabled.
Configure and enable maintenance in the table's Storage Optimization settings. Table-level settings include the check interval, maintenance parallelism, compaction policy, and the index fields and types to maintain.
Each maintenance cycle first checks the latest table state, including fragment distribution, deletions, and index coverage. It then runs the enabled maintenance tasks.
Compaction. DLF selects candidate fragments based on fragment size, the target row count, and the proportion of deleted rows. It rewrites valid data into larger fragments and handles soft-deleted data according to the compaction policy.
Index maintenance. DLF maintains vector, scalar, or full-text indexes on the configured fields. It adds index entries for uncovered data and consolidates index segments.
After maintenance completes, DLF waits for the configured check interval before starting the next check.
Maintenance settings and results
Setting | Default | Description |
Check interval | 3600 seconds | For example, if maintenance finishes at 10:05, the next check starts at 11:05. |
Maintenance parallelism | 1 | Controls the level of parallelism for maintenance tasks. |
Target rows per fragment | 1000000 | Compaction targets approximately 1 million rows per output fragment. |
Use the compaction history to identify the maintenance version and review statistics for that version. To assess compaction, compare file counts, file sizes, and fragment distribution before and after maintenance. To assess index maintenance, check index coverage.
High-performance caching
The high-performance cache provides an OSS-compatible interface. When caching is enabled, the Lance SDK calls DescribeTable to obtain the table location and storage access parameters, including the cache endpoint, credentials, and refresh information. The SDK uses the same OSS protocol to access the cache service, so you can keep using your existing SDK and application read and write code. On a cache hit, the service returns cached data directly. On a cache miss, the service reads the data from OSS, fills the cache, and returns the data to the client.
DLF organizes cache policies at the Catalog, Namespace, and Table levels. A lower level can inherit or override a policy from a higher level. The server manages cache policies, checks for cache hits, fetches data from OSS, and fills the cache.
Monitor the cache hit rate, read traffic from OSS, and read latency to evaluate cache performance.