Submit a data exploration query with Python

Updated at:

This topic demonstrates how to submit data exploration SQL queries using the DLF Python SDK, poll task status, and read query results in Arrow format with PyArrow.

Note

The examples in this topic use alibabacloud-dlfnext20250310 3.8.0, the latest public version available on PyPI.

Prerequisites

  • Account permissions: The RAM user must have DLF API and data permissions. For more information, see Configure permissions.

  • Python environment: Python 3.8 or later is installed and pip is configured.

  • Access credentials: An AccessKey ID and AccessKey Secret for an Alibaba Cloud account or RAM user have been obtained.

  • Public endpoint (optional): If you use a public endpoint to access DLF, make sure the DLF Catalog has public access enabled. For more information, see Enable and configure public access.

Procedure

Step 1: Install dependencies

Run the following command in a terminal to install the DLF Python SDK and PyArrow:

pip install alibabacloud-dlfnext20250310==3.8.0 pyarrow

Step 2: Configure environment variables

Set the access credentials and region information required for the demo:

export ALIBABA_CLOUD_ACCESS_KEY_ID='<AccessKey ID>'
export ALIBABA_CLOUD_ACCESS_KEY_SECRET='<AccessKey Secret>'
export DLF_REGION_ID='cn-hangzhou'
export DLF_DEFAULT_CATALOG='dlf_samples'

# Optional: Public endpoint. The SDK resolves it automatically based on the region.
# export DLF_ENDPOINT='dlfnext.cn-hangzhou.aliyuncs.com'

Parameter

Description

ALIBABA_CLOUD_ACCESS_KEY_ID

AccessKey ID of the Alibaba Cloud account or RAM user

ALIBABA_CLOUD_ACCESS_KEY_SECRET

AccessKey Secret of the Alibaba Cloud account or RAM user

DLF_REGION_ID

Region where DLF is deployed, for example cn-hangzhou

DLF_DEFAULT_CATALOG

Default catalog name

DLF_ENDPOINT (optional)

Public endpoint. The SDK resolves it automatically based on the region.

Step 3: Run queries and read results

Save the following code as data_exploration_query_demo.py:

import json
import os
import time
from urllib.request import urlopen

import pyarrow as pa
from alibabacloud_dlfnext20250310.client import Client
from alibabacloud_dlfnext20250310.models import GetQueryRequest, SubmitQueryRequest
from alibabacloud_tea_openapi.models import Config

TERMINAL_STATUSES = {"COMPLETED", "FAILED", "TIMEOUT", "CANCELLED"}


def required_env(name: str) -> str:
    value = os.getenv(name)
    if not value:
        raise RuntimeError(f"Missing environment variable: {name}")
    return value


def create_client() -> Client:
    return Client(Config(
        access_key_id=required_env("ALIBABA_CLOUD_ACCESS_KEY_ID"),
        access_key_secret=required_env("ALIBABA_CLOUD_ACCESS_KEY_SECRET"),
        security_token=os.getenv("ALIBABA_CLOUD_SECURITY_TOKEN"),
        region_id=required_env("DLF_REGION_ID"),
        endpoint=os.getenv("DLF_ENDPOINT"),
    ))


def wait_for_result(client: Client, query_id: str, timeout_seconds: int = 300):
    deadline = time.monotonic() + timeout_seconds
    while time.monotonic() < deadline:
        body = client.get_query(query_id, GetQueryRequest()).body
        if body and body.status in TERMINAL_STATUSES:
            return body
        time.sleep(0.5)
    raise TimeoutError(f"Query {query_id} did not finish within {timeout_seconds}s")


def read_arrow_rows(download_url: str):
    with urlopen(download_url, timeout=30) as response:
        payload = response.read()
    table = pa.ipc.open_file(pa.BufferReader(payload)).read_all()
    return table.column_names, table.to_pylist()


def main():
    client = create_client()
    request = SubmitQueryRequest(
        default_catalog=required_env("DLF_DEFAULT_CATALOG"),
        sql="SELECT * FROM dlf_samples.search_samples.berkeley_deepdrive_100k_images LIMIT 10",
        tier="standard",
        limit=1000,
    )

    query_id = client.submit_query(request).body.query_id
    result = wait_for_result(client, query_id)

    if result.status != "COMPLETED":
        errors = [
            f"{item.error_code}: {item.error}"

            for item in (result.results or [])

            if item.error_code or item.error
        ]
        raise RuntimeError(
            f"Query {query_id} ended as {result.status}: {'; '.join(errors)}"
        )

    statement = result.results[0]
    print(f"query_id={query_id}, row_count={statement.row_count}")
    if not statement.download_url:
        print("Query completed with no result rows.")
        return

    columns, rows = read_arrow_rows(statement.download_url)
    print("columns:", columns)
    print(json.dumps(rows, ensure_ascii=False, default=str, indent=2))


if __name__ == "__main__":
    main()

Run the script in a terminal:

python data_exploration_query_demo.py

Notes

  • Call chain: submit_query → poll get_query → download from download_url → read results with PyArrow.

  • limit controls the maximum number of rows returned by the server. We recommend also adding LIMIT in the SQL statement.

  • The download_url is a temporary link. Read it immediately after the query completes. If it expires, call get_query again to obtain a new link.

  • The calling account must have query permissions on the target catalog, database, and table.