Work with DLF Lance tables using Daft
This topic explains how to use Daft with a DLF Lance Catalog to create and write tables, analyze data, and write results to another table. DLF provides fully managed storage. Daft obtains table locations and access parameters from the Catalog.
Prerequisites
Create a Lance Catalog in the Running (RUNNING) state and obtain its service endpoint and authentication token. For instructions, see Manage and use a Lance Catalog.
Run the client in the VPC bound to the Lance Catalog. If the client runs in another VPC, ensure the two VPCs are connected.
Prepare a Python 3.11 environment.
Make sure you have access to the target Lance Catalog.
Install dependencies
The example was verified on Alibaba Cloud Linux 3 (x86_64) with Python 3.11.18 against a DLF Lance Catalog. The verification covered table creation, filtered reads, writing results, appends, and reading historical versions. Create an isolated virtual environment and install these versions:
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install "daft==0.7.26" "daft-lance==0.6.0" "pylance==13.0.0" "lance-namespace==0.11.1" "pyarrow==25.0.1"
This example uses the Namespace API provided by the daft-lance extension. For API details, see daft-lance. Keep the from daft_lance import read_lance, write_lance import shown below.
Configure the connection
Obtain the service endpoint and authentication token as described in Get connection information, and then provide them through environment variables:
export DLF_LANCE_ENDPOINT='<lance.endpoint>'
export DLF_LANCE_TOKEN='<lance.bearer-token>'
Parameter | Description |
| The full service endpoint returned by the Catalog. Do not append |
| The authentication token returned by the Catalog. This example uses Bearer authentication. Do not put the token in public code or logs. |
| Use |
| The service endpoint and authentication settings. |
| The full table identifier, such as |
When you supply Namespace parameters, do not also pass uri to read_lance or write_lance. For nested namespaces, include each level in the table_id array. For a table in the root namespace, use ["table_name"].
Procedure
Run the following Python snippets in order in the same script or interactive session.
This example creates two tables,
daft_documentsanddaft_guide_documents, in thedemonamespace. Make sure these table names are unused before the first run.The example uses a single-machine environment. For distributed execution, provide the connection settings securely to every execution process and keep dependency versions consistent.
Step 1: Connect to the Catalog and create a namespace
import os
import daft
import lance_namespace as ln
from daft_lance import read_lance, write_lance
properties = {
"uri": os.environ["DLF_LANCE_ENDPOINT"],
"header.Authorization": f"Bearer {os.environ['DLF_LANCE_TOKEN']}",
}
catalog = ln.connect("rest", properties)
catalog.create_namespace(
ln.CreateNamespaceRequest(id=["demo"], mode="exist_ok")
)
namespace_options = {
"namespace_impl": "rest",
"namespace_properties": properties,
}
If the namespace already exists, mode="exist_ok" reuses it.
Step 2: Create a Lance table and write data
Create three sample rows and write them to demo.daft_documents:
source_id = ["demo", "daft_documents"]
source = daft.from_pydict({
"id": [1, 2, 3],
"category": ["guide", "guide", "news"],
"text": [
"Lance data management",
"Vector search",
"Data lake update",
],
})
write_lance(
source,
table_id=source_id,
mode="create",
**namespace_options,
).collect()
The initial write uses mode="create". The client declares the table through the Catalog and writes its data. You do not need to specify a storage path or register the files separately.
After the write, you can find the table in the DLF console under Lance Catalog > demo > daft_documents.
Step 3: Read and analyze data
Select rows where category is guide, and keep the id and text columns:
df = read_lance(table_id=source_id, **namespace_options)
result = df.where(daft.col("category") == "guide").select("id", "text")
result.show()
The expected result contains rows with id values 1 and 2. You can use Daft column selection, filters, and transformations to extend this workflow.
Step 4: Write results to another table
result_id = ["demo", "daft_guide_documents"]
write_lance(
result,
table_id=result_id,
mode="create",
**namespace_options,
).collect()
read_lance(table_id=result_id, **namespace_options).show()
The expected daft_guide_documents table has two rows and only the id and text columns. It stores the result of this run and does not update automatically when the source table changes.
Step 5: Append data to the source table
more = daft.from_pydict({
"id": [4],
"category": ["guide"],
"text": ["Daft append example"],
})
write_lance(
more,
table_id=source_id,
mode="append",
**namespace_options,
).collect()
latest = read_lance(table_id=source_id, **namespace_options)
latest.select("id", "category", "text").show()
The expected daft_documents table has rows with id values 1, 2, 3, and 4. The appended data must match the original field names and types.
append does not deduplicate rows by id. Running this step again writes another copy. overwrite replaces the target table's data, so use it only when you intend to replace that data. If you repeat the initial table creation step, choose a new table name or use the existing table.
Step 6: Read a historical version
first_version = read_lance(
table_id=source_id,
version=1,
**namespace_options,
)
first_version.select("id", "text").show()
For a table created by these steps with its version history intact, version 1 contains the three rows from the initial write. Specify a version with version. To read the latest state, call read_lance again to create a new read plan.
Use automatic storage optimization
After frequent small writes, you can enable automatic data compaction to reduce fragmentation.
To maintain vector, scalar, or full-text indexes, you can configure automatic index building in the console.
You do not need to call an additional metadata synchronization notification API after writing. Automatic maintenance runs on its inspection schedule, so index coverage can briefly lag behind the latest write.
For configuration steps, see Automatic storage optimization for Lance tables.