Work with DLF Lance tables using Daft

Updated at:

This topic explains how to use Daft with a DLF Lance Catalog to create and write tables, analyze data, and write results to another table. DLF provides fully managed storage. Daft obtains table locations and access parameters from the Catalog.

Prerequisites

  • Create a Lance Catalog in the Running (RUNNING) state and obtain its service endpoint and authentication token. For instructions, see Manage and use a Lance Catalog.

  • Run the client in the VPC bound to the Lance Catalog. If the client runs in another VPC, ensure the two VPCs are connected.

  • Prepare a Python 3.11 environment.

  • Make sure you have access to the target Lance Catalog.

Install dependencies

The example was verified on Alibaba Cloud Linux 3 (x86_64) with Python 3.11.18 against a DLF Lance Catalog. The verification covered table creation, filtered reads, writing results, appends, and reading historical versions. Create an isolated virtual environment and install these versions:

python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install "daft==0.7.26" "daft-lance==0.6.0" "pylance==13.0.0" "lance-namespace==0.11.1" "pyarrow==25.0.1"

This example uses the Namespace API provided by the daft-lance extension. For API details, see daft-lance. Keep the from daft_lance import read_lance, write_lance import shown below.

Configure the connection

Obtain the service endpoint and authentication token as described in Get connection information, and then provide them through environment variables:

export DLF_LANCE_ENDPOINT='<lance.endpoint>'
export DLF_LANCE_TOKEN='<lance.bearer-token>'

Parameter

Description

DLF_LANCE_ENDPOINT

The full service endpoint returned by the Catalog. Do not append /v1.

DLF_LANCE_TOKEN

The authentication token returned by the Catalog. This example uses Bearer authentication. Do not put the token in public code or logs.

namespace_impl

Use rest to connect to the Lance Catalog.

namespace_properties

The service endpoint and authentication settings.

table_id

The full table identifier, such as ["demo", "daft_documents"]. The final element is the table name. Do not include the Catalog name or a .lance suffix.

When you supply Namespace parameters, do not also pass uri to read_lance or write_lance. For nested namespaces, include each level in the table_id array. For a table in the root namespace, use ["table_name"].

Procedure

Run the following Python snippets in order in the same script or interactive session.

Note
  • This example creates two tables, daft_documents and daft_guide_documents, in the demo namespace. Make sure these table names are unused before the first run.

  • The example uses a single-machine environment. For distributed execution, provide the connection settings securely to every execution process and keep dependency versions consistent.

Step 1: Connect to the Catalog and create a namespace

import os
import daft
import lance_namespace as ln
from daft_lance import read_lance, write_lance

properties = {
    "uri": os.environ["DLF_LANCE_ENDPOINT"],
    "header.Authorization": f"Bearer {os.environ['DLF_LANCE_TOKEN']}",
}
catalog = ln.connect("rest", properties)
catalog.create_namespace(
    ln.CreateNamespaceRequest(id=["demo"], mode="exist_ok")
)

namespace_options = {
    "namespace_impl": "rest",
    "namespace_properties": properties,
}

If the namespace already exists, mode="exist_ok" reuses it.

Step 2: Create a Lance table and write data

Create three sample rows and write them to demo.daft_documents:

source_id = ["demo", "daft_documents"]
source = daft.from_pydict({
    "id": [1, 2, 3],
    "category": ["guide", "guide", "news"],
    "text": [
        "Lance data management",
        "Vector search",
        "Data lake update",
    ],
})

write_lance(
    source,
    table_id=source_id,
    mode="create",
    **namespace_options,
).collect()

The initial write uses mode="create". The client declares the table through the Catalog and writes its data. You do not need to specify a storage path or register the files separately.

After the write, you can find the table in the DLF console under Lance Catalog > demo > daft_documents.

Step 3: Read and analyze data

Select rows where category is guide, and keep the id and text columns:

df = read_lance(table_id=source_id, **namespace_options)
result = df.where(daft.col("category") == "guide").select("id", "text")
result.show()

The expected result contains rows with id values 1 and 2. You can use Daft column selection, filters, and transformations to extend this workflow.

Step 4: Write results to another table

result_id = ["demo", "daft_guide_documents"]
write_lance(
    result,
    table_id=result_id,
    mode="create",
    **namespace_options,
).collect()

read_lance(table_id=result_id, **namespace_options).show()

The expected daft_guide_documents table has two rows and only the id and text columns. It stores the result of this run and does not update automatically when the source table changes.

Step 5: Append data to the source table

more = daft.from_pydict({
    "id": [4],
    "category": ["guide"],
    "text": ["Daft append example"],
})

write_lance(
    more,
    table_id=source_id,
    mode="append",
    **namespace_options,
).collect()

latest = read_lance(table_id=source_id, **namespace_options)
latest.select("id", "category", "text").show()

The expected daft_documents table has rows with id values 1, 2, 3, and 4. The appended data must match the original field names and types.

Note

append does not deduplicate rows by id. Running this step again writes another copy. overwrite replaces the target table's data, so use it only when you intend to replace that data. If you repeat the initial table creation step, choose a new table name or use the existing table.

Step 6: Read a historical version

first_version = read_lance(
    table_id=source_id,
    version=1,
    **namespace_options,
)
first_version.select("id", "text").show()

For a table created by these steps with its version history intact, version 1 contains the three rows from the initial write. Specify a version with version. To read the latest state, call read_lance again to create a new read plan.

Use automatic storage optimization

  • After frequent small writes, you can enable automatic data compaction to reduce fragmentation.

  • To maintain vector, scalar, or full-text indexes, you can configure automatic index building in the console.

  • You do not need to call an additional metadata synchronization notification API after writing. Automatic maintenance runs on its inspection schedule, so index coverage can briefly lag behind the latest write.

For configuration steps, see Automatic storage optimization for Lance tables.

Troubleshooting

The connection times out or the table cannot be read.

Check that the Catalog is running and that the client can reach the Catalog and data access endpoints.

The service returns 401 or 403.

Check that the token is complete and valid, the authentication value has the Bearer prefix, and the account has access.

Namespace parameters are not supported.

Check dependency versions and imports. Use the daft_lance API shown in this topic.

The table already exists during creation.

Use a new table name, or use append to continue writing to the existing table.

The write fails because the schema does not match.

Check field names and types, especially integer widths, floating-point types, and vector dimensions.

Old data appears after an append.

Call read_lance again to create a read plan. Do not reuse materialized data or a previously selected version.

Some distributed execution processes cannot read data.

Check that each process has the same dependencies and can reach the Catalog and data access endpoints.