Work with DLF Lance using Python

Updated at:

This topic shows how to connect to a DLF Lance Catalog from Python to create a table, read and write data, update and delete rows, search vectors, and read an earlier table version. DLF manages the storage and access configuration. The client uses the Catalog to locate tables.

Prerequisites

  • Create a Lance Catalog in the RUNNING state, and obtain its service endpoint and authentication token. For instructions, see Lance Catalog.

  • Run the client in the VPC bound to the Lance Catalog. If the client runs in another VPC, ensure the two VPCs are connected.

  • Prepare a Python 3.11 environment. The examples use a separate virtual environment.

  • Make sure you have access to the target Catalog. For instructions, see Configure permissions.

Connection parameters

To obtain the service endpoint and authentication token, see Get connection information.

Set the connection information as environment variables:

export DLF_LANCE_ENDPOINT='<lance.endpoint>'
export DLF_LANCE_TOKEN='<lance.bearer-token>'

Parameter

Description

DLF_LANCE_ENDPOINT

The complete service endpoint returned by the Catalog. Do not append /v1.

DLF_LANCE_TOKEN

The Catalog authentication token. The example sends it as Authorization: Bearer <token>.

namespace_id

Namespace identifier as a string array, such as ["demo"].

table_id

Full table identifier as a string array, such as ["demo", "documents"]. The last element is the table name. Do not include the Catalog name or append .lance.

The endpoint already identifies a Catalog. For a nested namespace, use ["first-level namespace", "second-level namespace", "table name"]. For a table in the root namespace, use ["table name"].

Procedure

Run the following Python examples in order in the same script or interactive session.

Note

Run the examples in a test namespace. Use a table name that does not already exist.

Step 1: Install dependencies

python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install "pylance==13.0.0" "lance-namespace==0.11.1" "pyarrow==25.0.1"

The R&D examples were verified on Alibaba Cloud Linux 3 (x86_64) with Python 3.11.18 and the listed client versions. The verification covered data reads and writes, updates and deletes, vector queries, and reading an earlier version. Before you upgrade a client, check read and write operations, version access, and index compatibility.

Step 2: Connect to the Catalog

import os
import lance
import lance_namespace as ln
import pyarrow as pa

endpoint = os.environ["DLF_LANCE_ENDPOINT"]
token = os.environ["DLF_LANCE_TOKEN"]

catalog = ln.connect(
    "rest",
    {
        "uri": endpoint,
        "header.Authorization": f"Bearer {token}",
    },
)

ln.connect("rest", ...) creates a REST Namespace client. Use it to manage namespaces and pass it to the Lance data read and write APIs.

Step 3: Create a namespace and write data

The following example creates the demo namespace and the documents table, then writes three rows. The embedding field contains an eight-dimensional Float32 vector for demonstration. Use vectors generated by your model for application data.

namespace_id = ["demo"]
table_id = ["demo", "documents"]

catalog.create_namespace(
    ln.CreateNamespaceRequest(id=namespace_id, mode="exist_ok")
)

schema = pa.schema([
    pa.field("id", pa.int64()),
    pa.field("category", pa.string()),
    pa.field("text", pa.string()),
    pa.field("embedding", pa.list_(pa.float32(), 8)),
])

data = pa.Table.from_pylist([
    {
        "id": 1, "category": "guide", "text": "Lance data management",
        "embedding": [1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
    },
    {
        "id": 2, "category": "guide", "text": "Vector search",
        "embedding": [0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
    },
    {
        "id": 3, "category": "news", "text": "Data lake update",
        "embedding": [0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0],
    },
], schema=schema)

dataset = lance.write_dataset(
    data,
    namespace_client=catalog,
    table_id=table_id,
    mode="create",
)
print("rows:", dataset.count_rows())

Expected output: rows: 3. You can view the table in the DLF console under Lance Catalog > demo > documents.

  • mode="exist_ok" uses the namespace if it already exists.

  • mode="create" creates a new table and returns an error if the table already exists.

  • The Lance client declares the table through the Catalog, obtains its storage location and access parameters, and writes the data. You do not need to register the data files again.

Step 4: Query data

Reopen the table and query rows whose category is guide:

dataset = lance.dataset(namespace_client=catalog, table_id=table_id)
result = dataset.to_table(
    columns=["id", "text"],
    filter="category = 'guide'",
)
print(result.to_pydict())

The expected result contains rows with IDs 1 and 2. Use columns to select fields and filter to set a filter condition.

Step 5: Append, update, and delete data

Append one row using the same field types as the existing table:

new_data = pa.Table.from_pylist([
    {
        "id": 4, "category": "guide", "text": "Append example",
        "embedding": [0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0],
    }
], schema=schema)

lance.write_dataset(
    new_data,
    namespace_client=catalog,
    table_id=table_id,
    mode="append",
)
dataset = lance.dataset(namespace_client=catalog, table_id=table_id)
print("rows:", dataset.count_rows())

Expected output: rows: 4.

Update the appended row and then delete it:

dataset.update({"text": "'Updated example'"}, where="id = 4")
dataset.delete("id = 4")
print("rows:", dataset.count_rows())

Expected output: rows: 3. Values passed to update are SQL expressions, so the string literal includes single quotation marks. delete takes a row filter condition.

Important: append writes another copy of a row each time you run it; it does not deduplicate rows by id. overwrite replaces the table data. Use it only when you intend to replace the data.

Step 6: Run a vector search

The following example uses L2 distance to find the two rows closest to the query vector:

dataset = lance.dataset(namespace_client=catalog, table_id=table_id)
result = dataset.to_table(
    columns=["id", "text", "_distance"],
    nearest={
        "column": "embedding",
        "q": [1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
        "k": 2,
        "metric": "L2",
    },
)
print(result.to_pydict())

_distance is the query distance. The query vector must have the same number of dimensions as the embedding field. When you query through an index, use the distance type configured for that index.

You can scan a small table directly. As the data grows, you can enable automatic index maintenance in the DLF console. The three sample rows demonstrate reads, writes, and queries; they are not suitable for training a quantized vector index.

Step 7: Read an earlier version

The following example reads the first version of a table created by this procedure, provided that its version history has not been cleaned up:

history = lance.dataset(
    namespace_client=catalog,
    table_id=table_id,
    version=1,
)
print(history.to_table(columns=["id", "text"]).to_pydict())

Use version to specify a version number. If you omit it, opening the dataset reads the latest version. A dataset object held for a long time does not automatically switch to a version committed by another client. Reopen the dataset to query the latest state.

Use with automatic storage optimization

  • After repeated small batch writes, you can enable automatic data compaction to reduce data fragmentation.

  • For vector search, conditional filtering, or full-text search, configure the corresponding indexes in the console. DLF automatically maintains them for new data.

  • The client does not need to call an additional notification API after a write to trigger metadata synchronization.

  • Automatic maintenance runs on a check interval, so there may be a delay between a write and index completion.

For configuration details, see Automatic storage optimization for Lance tables.

Troubleshooting

The Catalog connection times out.

Check that the Catalog is running, the endpoint is complete, and the client network can reach the target VPC.

The request returns 401 or 403.

Check that the token is complete and valid, the Authorization value starts with Bearer , and the current identity has access. Do not configure conflicting authentication headers.

Creating a table reports that it already exists.

Use an unused table name. If you intend to keep writing to the existing table, read it and use append.

A table exists in the Catalog but its fields or data cannot be read.

Check whether the table was only declared and the first data write has not completed successfully.

Appending data reports a field type mismatch.

Use the same Arrow Schema as the existing table, particularly for Float32 vectors and their fixed dimension.

A vector query reports a dimension or distance type error.

Check the query vector length, field dimension, and index distance type.

Query results do not change after a write.

Reopen the dataset instead of continuing to use an older dataset object.

Access credentials expire during a long-running process.

Keep using the Namespace connection so the client obtains access parameters through the protocol. Check Catalog authentication and network connectivity. Do not cache or manually reuse temporary credentials from DescribeTable for a long period.