Full-text search for DLF Paimon tables

Overview

Full-text search uses the Tantivy engine to provide inverted-index-based search on STRING columns in Paimon append-only tables. Common use cases include log search, document retrieval, and content filtering.

Limitations

Prerequisites

Verify that your environment meets the following requirements:

• PyPaimon: pypaimon-1.5.dev20260727 is not yet available on public PyPI. Download the package from this document. The runtime environment must be Linux x86_64 (glibc >= 2.28).

  1. Install pypaimon-1.5.dev20260704.tar.gz with full-text dependencies:

     pip install 'pypaimon-1.5.dev20260727.tar.gz[full-text]'

  2. (Optional) For DuckDB output, also install:

     pip install duckdb

• Spark SQL: Use Paimon Ali 1-ali-29.1. Prepare the following JARs and upload them to OSS:

  • paimon-ali-emr-spark-3.5-1-ali-29.1.jar

  • paimon-full-text-index-0.1.0.jar

  • paimon-full-text-1-ali-29.1.jar

  spark.emr.serverless.excludedModules              paimon
  spark.emr.serverless.user.defined.jars            oss://<your-bucket>/paimon-ali-emr-spark-3.5-1-ali-29.1.jar,oss://<your-bucket>/paimon-full-text-1-ali-29.1.jar,oss://<your-bucket>/paimon-full-text-index-0.1.0.jar

Create a full-text-indexed table

CREATE TABLE articles (
    id INT,
    title STRING,
    content STRING
) TBLPROPERTIES (
    'row-tracking.enabled' = 'true',
    'data-evolution.enabled' = 'true',
    'morax.full-text-index.enabled' = 'true',
    'global-index.full-text.index-column' = 'content'
);

Insert data and trigger index building

The following INSERT statement works with both Flink SQL and Spark SQL:

INSERT INTO articles VALUES
    (1, 'lake storage', 'apache paimon is a lake storage format for big data'),
    (2, 'stream engine', 'flink is a stream processing engine for real time analytics'),
    (3, 'data evolution', 'paimon supports data evolution and row tracking features'),
    (4, 'query engine', 'spark sql can query paimon tables directly with high performance'),
    (5, 'distributed', 'ray data provides distributed data processing capabilities');

After you insert data:

• DLF automatically schedules the full-text index build.

• Index building is asynchronous. Once complete, queries automatically use the index for faster searches.

• To check the index build progress, go to the target Catalog > target table > Files tab in the DLF console. After the build completes, you can see the full-text index file (for example, tantivy-global-index-<UUID>.index).

Set the full-text index check interval

morax.full-text-index.check-interval sets how often DLF checks whether a full-text index build task needs to be submitted. The default is 1h. To get newly written data into the full-text index sooner, shorten the interval. For example:

ALTER TABLE articles SET TBLPROPERTIES (
    'morax.full-text-index.check-interval' = '5min'
);

Run full-text searches

SELECT *
FROM full_text_search(
    'default.articles',
    'content',
    'paimon',
    10
);

from pypaimon import CatalogFactory

CATALOG_OPTIONS = {
    "metastore": "rest",
    "uri": "http://<DLF-ENDPOINT>",  # For VPC access, use http://<REGION>-vpc.dlf.aliyuncs.com
    "warehouse": "<YOUR-CATALOG>",
    "token.provider": "dlf",
    "dlf.region": "<REGION-ID>",
    "dlf.access-key-id": "<ACCESS-KEY-ID>",
    "dlf.access-key-secret": "<ACCESS-KEY-SECRET>",
    "dlf.oss-endpoint": "<OSS-ENDPOINT>",
}

catalog = CatalogFactory.create(CATALOG_OPTIONS)

table = catalog.get_table('default.articles')

builder = table.new_full_text_search_builder()
builder.with_query(
    'content',
    '{"match":{"query":"paimon"}}'
)
builder.with_limit(3)
result = builder.execute_local()

read_builder = table.new_read_builder()
read_builder = read_builder.with_projection(['id', 'title', 'content'])
scan = read_builder.new_scan().with_global_index_result(result)
splits = scan.plan().splits()
table_read = read_builder.new_read()
df = table_read.to_pandas(splits)
print(df)

# Continues from the builder and result variables above
score_fn = result.score_getter()
for row_id in result.results():
    print(f"row_id={row_id}, score={score_fn(row_id)}")

row_id=0, score=1.0508
row_id=2, score=1.1567
row_id=3, score=1.0508

builder = table.new_full_text_search_builder()
builder.with_query(
    'content',
    '{"match":{"query":"data processing","operator":"And"}}'
)
builder.with_limit(10)
result = builder.execute_local()

builder = table.new_full_text_search_builder()
builder.with_query(
    'content',
    '{"match":{"query":"spark flink","operator":"Or"}}'
)
builder.with_limit(10)
result = builder.execute_local()

# PyArrow Table
arrow_table = table_read.to_arrow(splits)
print(arrow_table)

# DuckDB
conn = table_read.to_duckdb(splits, 'articles')
print(conn.execute('SELECT * FROM articles').fetchdf())

-- Search for articles containing "paimon" and return the top 3 results
SELECT *
FROM full_text_search(
    'articles',
    'content',
    '{"match":{"query":"paimon"}}',
    3
);

SELECT id, title
FROM full_text_search(
    'default.articles',
    'content',
    '{"match":{"query":"data"}}',
    10
)
WHERE id > 1;