Submit a data exploration query with Python
This topic demonstrates how to submit data exploration SQL queries using the DLF Python SDK, poll task status, and read query results in Arrow format with PyArrow.
The examples in this topic use alibabacloud-dlfnext20250310 3.8.0, the latest public version available on PyPI.
Prerequisites
-
Account permissions: The RAM user must have DLF API and data permissions. For more information, see Configure permissions.
-
Python environment: Python 3.8 or later is installed and pip is configured.
-
Access credentials: An AccessKey ID and AccessKey Secret for an Alibaba Cloud account or RAM user have been obtained.
-
Public endpoint (optional): If you use a public endpoint to access DLF, make sure the DLF Catalog has public access enabled. For more information, see Enable and configure public access.
Procedure
Step 1: Install dependencies
Run the following command in a terminal to install the DLF Python SDK and PyArrow:
pip install alibabacloud-dlfnext20250310==3.8.0 pyarrow
Step 2: Configure environment variables
Set the access credentials and region information required for the demo:
export ALIBABA_CLOUD_ACCESS_KEY_ID='<AccessKey ID>'
export ALIBABA_CLOUD_ACCESS_KEY_SECRET='<AccessKey Secret>'
export DLF_REGION_ID='cn-hangzhou'
export DLF_DEFAULT_CATALOG='dlf_samples'
# Optional: Public endpoint. The SDK resolves it automatically based on the region.
# export DLF_ENDPOINT='dlfnext.cn-hangzhou.aliyuncs.com'
|
Parameter |
Description |
|
|
AccessKey ID of the Alibaba Cloud account or RAM user |
|
|
AccessKey Secret of the Alibaba Cloud account or RAM user |
|
|
Region where DLF is deployed, for example |
|
|
Default catalog name |
|
|
Public endpoint. The SDK resolves it automatically based on the region. |
Step 3: Run queries and read results
Save the following code as data_exploration_query_demo.py:
import json
import os
import time
from urllib.request import urlopen
import pyarrow as pa
from alibabacloud_dlfnext20250310.client import Client
from alibabacloud_dlfnext20250310.models import GetQueryRequest, SubmitQueryRequest
from alibabacloud_tea_openapi.models import Config
TERMINAL_STATUSES = {"COMPLETED", "FAILED", "TIMEOUT", "CANCELLED"}
def required_env(name: str) -> str:
value = os.getenv(name)
if not value:
raise RuntimeError(f"Missing environment variable: {name}")
return value
def create_client() -> Client:
return Client(Config(
access_key_id=required_env("ALIBABA_CLOUD_ACCESS_KEY_ID"),
access_key_secret=required_env("ALIBABA_CLOUD_ACCESS_KEY_SECRET"),
security_token=os.getenv("ALIBABA_CLOUD_SECURITY_TOKEN"),
region_id=required_env("DLF_REGION_ID"),
endpoint=os.getenv("DLF_ENDPOINT"),
))
def wait_for_result(client: Client, query_id: str, timeout_seconds: int = 300):
deadline = time.monotonic() + timeout_seconds
while time.monotonic() < deadline:
body = client.get_query(query_id, GetQueryRequest()).body
if body and body.status in TERMINAL_STATUSES:
return body
time.sleep(0.5)
raise TimeoutError(f"Query {query_id} did not finish within {timeout_seconds}s")
def read_arrow_rows(download_url: str):
with urlopen(download_url, timeout=30) as response:
payload = response.read()
table = pa.ipc.open_file(pa.BufferReader(payload)).read_all()
return table.column_names, table.to_pylist()
def main():
client = create_client()
request = SubmitQueryRequest(
default_catalog=required_env("DLF_DEFAULT_CATALOG"),
sql="SELECT * FROM dlf_samples.search_samples.berkeley_deepdrive_100k_images LIMIT 10",
tier="standard",
limit=1000,
)
query_id = client.submit_query(request).body.query_id
result = wait_for_result(client, query_id)
if result.status != "COMPLETED":
errors = [
f"{item.error_code}: {item.error}"
for item in (result.results or [])
if item.error_code or item.error
]
raise RuntimeError(
f"Query {query_id} ended as {result.status}: {'; '.join(errors)}"
)
statement = result.results[0]
print(f"query_id={query_id}, row_count={statement.row_count}")
if not statement.download_url:
print("Query completed with no result rows.")
return
columns, rows = read_arrow_rows(statement.download_url)
print("columns:", columns)
print(json.dumps(rows, ensure_ascii=False, default=str, indent=2))
if __name__ == "__main__":
main()
Run the script in a terminal:
python data_exploration_query_demo.py
Notes
-
Call chain:
submit_query→ pollget_query→ download fromdownload_url→ read results with PyArrow. -
limitcontrols the maximum number of rows returned by the server. We recommend also addingLIMITin the SQL statement. -
The
download_urlis a temporary link. Read it immediately after the query completes. If it expires, callget_queryagain to obtain a new link. -
The calling account must have query permissions on the target catalog, database, and table.