Manage and use a Lance Catalog
DLF provides fully managed storage and metadata management for Lance tables. Use Lance Catalog to organize namespaces and tables, read and write data with Lance-compatible clients, and view table schemas, versions, indexes, and storage optimization details in the console. This topic explains how to create, connect to, and manage a Lance Catalog.
Prerequisites
-
Service activation: Activate DLF and complete the service authorization required for first-time use. For instructions, see Set up DLF.
-
Account permissions: Use an Alibaba Cloud account or a RAM user with the DLF administrator role and the required management API permissions. RAM users also need the relevant DLF API and data permissions. For instructions, see Configure permissions.
-
Network access: Prepare a virtual private cloud (VPC) and a vSwitch in the target region. The client environment must be able to reach the Lance Catalog endpoint in the selected network.
Limitations
-
Catalog structure: Lance Catalog manages its own namespaces and tables. You do not need to create a Paimon database or table first.
-
Sharing and permissions: Lance Catalog sharing and fine-grained permissions at the namespace or table level are not currently supported.
-
Data operations: Use a compatible client, such as Lance, to write, update, delete, and query data. The Catalog REST API is primarily for managing namespaces, table metadata, and access parameters.
-
Table registration: Table registration supports only the table paths specified by the Catalog. You cannot use the registration API to mount an arbitrary external data path.
-
Network settings: You cannot directly change network settings such as the VPC, vSwitch, or zone after creating a Catalog.
Concepts
|
Concept |
Description |
|
Catalog |
The top-level unit for managing Lance resources. It provides an endpoint and authentication information. |
|
Namespace |
A container for organizing Lance tables. Namespaces can have multiple levels. For example, |
|
Table |
A Lance dataset that stores structured fields, vectors, and other multimodal data. A client uses the table identifier to obtain its storage location and access parameters. |
|
Version |
A data state created when a write, update, delete, or maintenance operation is committed. You can use a version to read historical data. |
|
Tag |
A name assigned to a specific version so that you can refer to and read that version. |
Create a Lance Catalog
-
Log on to the DLF console Select the target region.
-
In the left-side navigation pane, choose Catalogs and click Create Catalog.
-
Set the Catalog type to Lance and configure the following settings.
Setting
Description
Catalog Name
Enter a name that is not in use in the current region. The name identifies the Catalog.
Description
Optional. Describe the intended use of the data or the associated business.
Storage Redundancy Type
Select local redundancy or zone-redundant storage based on your availability requirements. Available options depend on the region.
Zone
Select a zone that supports Lance Catalog.
VPC ID
Select a VPC that your clients can access.
vSwitch ID
Select a vSwitch in the selected VPC and zone.
-
Confirm the settings and submit the creation request.
-
Return to the Catalog list and wait until the Catalog status changes to Running (RUNNING).
Catalog creation is asynchronous. Connect to the Catalog and create tables after its status changes to Running.
Get connection information
-
In the Catalog list, click the name of the target Lance Catalog.
-
Open the Catalog Configuration tab and record the following connection settings.
You can also query the Catalog API and read the
optionsfield in the response.Setting
Purpose
lance.endpointThe Lance Catalog endpoint. Use the returned value without appending an API path.
lance.bearer-tokenThe authentication token for the Lance Catalog. Configure it as a Bearer Token in the client.
lance.versionThe current Catalog service version, which can help troubleshoot client compatibility issues.
Only users with the appropriate administrator identity can retrieve the full authentication token. If you see a masked value, use an identity with the required permissions. The token provides access to this Catalog. Pass it to your application through an environment variable or a secrets management service, and do not include it in public code or logs.
Clients obtain table storage locations and temporary access parameters from the Catalog. You do not need to prepare a storage bucket or construct data paths yourself.
Manage namespaces and tables
Create namespaces and tables
Use a Python or Daft client to create namespaces, declare tables, and complete the first write. For instructions, see Work with DLF Lance using Python or Work with DLF Lance tables using Daft. Refresh the console after the first write to view table schemas, versions, and other information.
Declaring a table allocates its storage location. A table that has been declared but has not received its first write does not have complete dataset metadata.
Follow these naming rules:
-
Each namespace level and table must have a nonempty name. The UTF-8 encoded length of each name cannot exceed 255 bytes.
-
Do not use
/,\,$, control characters, or.or..as a complete name. -
Namespace names cannot end in
.lance. For a multilevel namespace, pass each level as an element in a string array in the client.
Browse namespaces and tables
-
In the Catalog list, click the target Lance Catalog.
-
Starting from the root namespace, select each level until you reach the target namespace.
-
Click a table name to open its details page.
You can search for resources in the current namespace by name prefix. You can also enter a relative path such as demo/images: intermediate segments must match full names, while the final segment is matched by prefix. Begin the path with / to search from the Catalog root.
View table information
|
Main tab |
Description |
|
Table Details |
|
|
Files |
Browse files and subdirectories in the table storage directory, view file names, sizes, and update times, and download files. |
|
Snapshots |
|
|
Index |
View index fields, types, and data coverage for the selected version. Automatic index building: Configure automatic index building on the latest version of the main branch. When enabled, all configured indexes are built automatically. For instructions, see Step 3: Configure automatic index maintenance. |
|
Storage Optimization |
|
|
Data Files |
|
The Table Details, Snapshots, Index, and Data Files tabs support viewing by branch. After you select a historical version, the schema, index coverage, and data file analysis reflect that version. The Files tab shows files in the table storage directory, including files not referenced by the selected version. Storage Optimization applies to the main dataset. Switch back to the main branch before changing its settings.
Read a historical version
Specify a version number in a Lance client to read its data. You can also create a tag for a version you want to reference and read it by tag name. To read the latest data, reopen the dataset or refresh it to the latest version. For code examples, see Work with DLF Lance using Python or Work with DLF Lance tables using Daft.
Browsing versions, managing tags, and cleaning up versions are separate operations. Automatic data compaction does not replace a historical-version retention policy. Manage historical versions separately based on your business needs.
Delete tables, namespaces, and Catalogs
Before deleting resources, stop the applications that read or write them and confirm that the data is no longer needed.
|
Resource or operation |
Description |
|
Delete a table (DropTable) |
Deletes the table and its underlying data. First disable and remove its automatic maintenance tasks so that background tasks stop accessing it. |
|
Deregister a table (DeregisterTable) |
Removes the table registration from the Catalog while preserving the underlying data. Unlike deleting a table, this does not release the space occupied by the table data. |
|
Delete a namespace |
Only empty namespaces can be deleted. First remove their tables and child namespaces. |
|
Delete a Catalog |
First remove automatic maintenance tasks, tables, and namespace data, unbind associated cache resources, and then delete the Catalog in the console. |
Deletion may remove data. Do not confuse deregistering a table with deleting it, and do not assume that deleted data can be recovered.
Connect to the Catalog as shown in Work with DLF Lance using Python. In the following examples, catalog is the connected client and ln is the imported Lance Namespace module. Replace the namespace and table names with your own.
Delete the table and its underlying data:
catalog.drop_table(ln.DropTableRequest(id=["your_namespace", "your_table"]))To preserve the underlying data, deregister the table instead. For a given table, run either this operation or the preceding deletion operation:
catalog.deregister_table(ln.DeregisterTableRequest(id=["your_namespace", "your_table"]))After removing all tables and child namespaces, delete the empty namespace:
catalog.drop_namespace(ln.DropNamespaceRequest(id=["your_namespace"], behavior="restrict"))