Holo Client Python
holo-client-py is the official Hologres Python client. Built on psycopg3, it provides a synchronous API (HoloClient) and an asynchronous API (AsyncHoloClient) for bulk data ingestion and high-QPS point queries. This topic explains when to choose holo-client-py and how to install it, write data (including Copy writes), and query data. It also compares the performance of each method and describes the HoloConfig parameters.
holo-client-py is the official Hologres Python client, built on psycopg3. It targets two workloads: writing large data volumes to Hologres, either in batches or in real time, and high-QPS point queries such as dimension table joins. holo-client-py provides a synchronous API (HoloClient) and an asynchronous API (AsyncHoloClient), and supports regular writes, primary key conflict handling, high-speed Copy writes, full primary key lookups, and Scan queries. It outperforms raw psycopg and JDBC for both bulk writes and point queries. For the source code, see holo-client-py.
Hologres integrates tightly with the Python ecosystem, which helps you build a real-time data warehouse directly from your Python applications. To connect to Hologres from Java instead, see Holo Client.
In the code examples in this topic, replace <HOST>, <PORT>, and <DATABASE> with the endpoint, port, and database name of your Hologres instance, and replace <USERNAME> and <PASSWORD> with the credentials that you use to log on to Hologres. If you log on as an Alibaba Cloud account or a RAM user, use its AccessKey ID and AccessKey secret. If you log on as a BASIC account, use the username and password of that account.
Choose a client
Both holo-client-py and Psycopg 3 let your Python applications access Hologres. They differ in the following ways:
-
Write performance:
holo-client-pyis optimized for ingestion with batchedPutrequests, the Fixed Copy streaming protocol, and Stage-based writes. A single process can sustain tens of thousands of TPS. With Psycopg 3, you batch writes and manage connection pools yourself, which makes performance tuning harder. -
Query model:
holo-client-pyfocuses on primary key lookups and range scans through a compact API. Psycopg 3 supports full SQL, which suits complex queries but requires you to write the SQL by hand. -
Error handling:
holo-client-pyretries automatically and aggregates errors across a batch. With Psycopg 3, you handle connection errors and transaction rollbacks yourself.
The following table recommends an access method for common scenarios, including when to use Hologres DataFrame.
|
Scenario |
Recommended method |
|
High-throughput batch writes, such as logs and monitoring data |
Holo Client Python ( |
|
High-QPS primary key lookups, such as real-time feature retrieval |
Holo Client Python ( |
|
Streaming writes, such as Kafka or Flink pipelines |
Holo Client Python (Fixed Copy protocol) |
|
Data analysis and ETL, such as aggregations and joins |
Hologres DataFrame |
|
Complex SQL queries, such as reporting and BI |
Psycopg 3 or Hologres DataFrame |
|
pandas ecosystem integration |
Hologres DataFrame |
For more information about Hologres DataFrame, see Hologres DataFrame. To connect to Hologres by using Psycopg 3, see Python.
Limits
-
In synchronous mode,
HoloClientwrites in parallel by using multiple processes and shared memory. In asynchronous mode,AsyncHoloClientwrites concurrently by using multipleasynciowriter tasks. -
Python 3.8 or later and
psycopg[binary]3.1 or later are required. -
Fixed Copy is supported in Hologres V1.3 and later, and applies to real-time write scenarios that involve no DELETE operations.
Install the connector
You can install holo-client-py directly from pip.
pip install hologres-client
To install from source, run the following commands.
cd holo-client-py
pip install -e .
The pip package is named hologres-client, while the source directory in the open source repository is named holo-client-py.
Connections
holo-client-py creates a number of database connections based on your configuration:
-
HoloClientcreates connections according towrite_parallelismandread_parallelism. -
A connection that stays idle for longer than
connection_max_idle_msis released. -
Before you write data, check how many connections are still available on your instance. For more information, see Manage connections.
-- View the maximum number of connections
select instance_max_connections();
-- View the number of connections in use
select count(*) from pg_stat_activity where backend_type='client backend';
Write data
Create a single HoloClient or AsyncHoloClient instance in your project, and control write concurrency with write_parallelism.
-
Synchronous client (
HoloClient): writes in parallel by using multiple processes and shared memory. -
Asynchronous client (
AsyncHoloClient): writes concurrently by using multipleasynciowriter tasks, which suitsasyncioapplications.
Write to a table without a primary key
Synchronous API
The following example uses HoloClient to write to a table without a primary key.
from hologres import HoloClient, HoloConfig, Put
config = HoloConfig(
host="<HOST>",
port=<PORT>,
database="<DATABASE>",
username="<USERNAME>",
password="<PASSWORD>",
write_parallelism=4, # In synchronous mode, start 4 worker processes to write in parallel
)
with HoloClient(config) as client:
schema = client.get_table_schema("t0")
put = Put(schema)
put.set_object("id", 1)
put.set_object("name", "name0")
put.set_object("address", "address0")
client.put(put)
# Force all pending put requests to commit. HoloClient also commits automatically based on write_batch_size, write_batch_byte_size, and write_max_interval_ms
# client.flush()
Asynchronous API
Submit put requests in bulk with asyncio.create_task so that the writer tasks can form full batches and reach maximum throughput. Do not call await client.put() for one request at a time, because each batch would then contain a single request.
import asyncio
from hologres import AsyncHoloClient, HoloConfig, Put
async def main():
config = HoloConfig(
host="<HOST>",
port=<PORT>,
database="<DATABASE>",
username="<USERNAME>",
password="<PASSWORD>",
write_parallelism=4, # In asynchronous mode, start 4 writer tasks to flush concurrently
)
async with AsyncHoloClient(config) as client:
schema = await client.get_table_schema("t0")
# Submit put requests in bulk instead of awaiting each one
tasks = []
for i in range(100):
put = Put(schema)
put.set_object("id", i)
put.set_object("name", f"name{i}")
put.set_object("address", f"address{i}")
tasks.append(asyncio.create_task(client.put(put)))
# Wait for all results together
await asyncio.gather(*tasks)
await client.flush()
asyncio.run(main())
Write to a table with a primary key
Synchronous API
When you write to a table with a primary key, use on_conflict_action to set the strategy for primary key conflicts, as shown in the following example.
from hologres import HoloClient, HoloConfig, Put, OnConflictAction
config = HoloConfig(
host="<HOST>",
port=<PORT>,
database="<DATABASE>",
username="<USERNAME>",
password="<PASSWORD>",
on_conflict_action=OnConflictAction.INSERT_OR_REPLACE, # Strategy for primary key conflicts
)
with HoloClient(config) as client:
# create table t0(id int not null, name0 text, address text, primary key(id))
schema = client.get_table_schema("t0")
put = Put(schema)
put.set_object("id", 1)
put.set_object("name0", "name0")
put.set_object("address", "address0")
client.put(put)
put = Put(schema)
put.set_object("id", 1)
put.set_object("name0", "newName")
put.set_object("address", "newAddress")
client.put(put)
# client.flush()
Asynchronous API
Submit put requests in bulk with asyncio.create_task so that the writer tasks can form full batches and reach maximum throughput.
import asyncio
from hologres import AsyncHoloClient, HoloConfig, Put, OnConflictAction
async def main():
config = HoloConfig(
host="<HOST>",
port=<PORT>,
database="<DATABASE>",
username="<USERNAME>",
password="<PASSWORD>",
on_conflict_action=OnConflictAction.INSERT_OR_REPLACE,
)
async with AsyncHoloClient(config) as client:
schema = await client.get_table_schema("t0")
tasks = []
for i in range(100):
put = Put(schema)
put.set_object("id", i)
put.set_object("name0", f"name{i}")
put.set_object("address", f"address{i}")
tasks.append(asyncio.create_task(client.put(put)))
await asyncio.gather(*tasks)
await client.flush()
asyncio.run(main())
Partially update a table with a primary key
Synchronous API
Set on_conflict_action to INSERT_OR_UPDATE for a partial update: on a primary key conflict, only the columns set in the Put are updated. The following example shows how.
from hologres import HoloClient, HoloConfig, Put, OnConflictAction
config = HoloConfig(
host="<HOST>",
port=<PORT>,
database="<DATABASE>",
username="<USERNAME>",
password="<PASSWORD>",
on_conflict_action=OnConflictAction.INSERT_OR_UPDATE, # Partial update
)
with HoloClient(config) as client:
# create table t0(id int not null, name0 text, address text, primary key(id))
schema = client.get_table_schema("t0")
# Put only the id and name0 columns. On a primary key conflict these columns are updated; otherwise the record is inserted
put = Put(schema)
put.set_object("id", 1)
put.set_object("name0", "name0")
client.put(put)
put = Put(schema)
put.set_object("id", 1)
put.set_object("name0", "newName")
client.put(put)
# client.flush()
Asynchronous API
import asyncio
from hologres import AsyncHoloClient, HoloConfig, Put, OnConflictAction
async def main():
config = HoloConfig(
host="<HOST>",
port=<PORT>,
database="<DATABASE>",
username="<USERNAME>",
password="<PASSWORD>",
on_conflict_action=OnConflictAction.INSERT_OR_UPDATE,
)
async with AsyncHoloClient(config) as client:
schema = await client.get_table_schema("t0")
tasks = []
for i in range(100):
put = Put(schema)
put.set_object("id", i)
put.set_object("name0", f"name{i}")
tasks.append(asyncio.create_task(client.put(put)))
await asyncio.gather(*tasks)
await client.flush()
asyncio.run(main())
Delete by primary key
A higher proportion of DELETE operations lowers the overall number of records written per second.
Synchronous API
Set mutation_type to MutationType.DELETE to delete data by full primary key, as shown in the following example.
from hologres import HoloClient, HoloConfig, Put, MutationType, OnConflictAction
config = HoloConfig(
host="<HOST>",
port=<PORT>,
database="<DATABASE>",
username="<USERNAME>",
password="<PASSWORD>",
on_conflict_action=OnConflictAction.INSERT_OR_REPLACE,
)
with HoloClient(config) as client:
# create table t0(id int not null, name0 text, address text, primary key(id))
schema = client.get_table_schema("t0")
put = Put(schema)
put.mutation_type = MutationType.DELETE
put.set_object("id", 1)
client.put(put)
# client.flush()
Asynchronous API
import asyncio
from hologres import AsyncHoloClient, HoloConfig, Put, MutationType, OnConflictAction
async def main():
config = HoloConfig(
host="<HOST>",
port=<PORT>,
database="<DATABASE>",
username="<USERNAME>",
password="<PASSWORD>",
on_conflict_action=OnConflictAction.INSERT_OR_REPLACE,
)
async with AsyncHoloClient(config) as client:
schema = await client.get_table_schema("t0")
tasks = []
for i in range(100):
put = Put(schema)
put.mutation_type = MutationType.DELETE
put.set_object("id", i)
tasks.append(asyncio.create_task(client.put(put)))
await asyncio.gather(*tasks)
await client.flush()
asyncio.run(main())
Copy writes
Fixed Copy is supported in Hologres V1.3 and later. Compared with HoloClient.put, Fixed Copy delivers higher throughput in stream mode, lower data latency, and lower client-side memory usage because it does not batch records. For real-time writes that involve no DELETE operations, use Fixed Copy.
Write to a table without a primary key by using Fixed Copy
Synchronous API
copy_writer uses the conflict strategy configured in HoloConfig, as shown in the following example.
from hologres import HoloClient, HoloConfig, Put, CopyFormat, CopyMode, OnConflictAction
config = HoloConfig(
host="<HOST>",
port=<PORT>,
database="<DATABASE>",
username="<USERNAME>",
password="<PASSWORD>",
on_conflict_action=OnConflictAction.INSERT_OR_UPDATE, # copy_writer uses the conflict strategy from HoloConfig
)
with HoloClient(config) as client:
# CREATE TABLE copy_demo (id INT NOT NULL, name TEXT NOT NULL, address TEXT, PRIMARY KEY(id));
columns = ["id", "name"]
with client.copy_writer(
"copy_demo",
columns=columns,
fmt=CopyFormat.BINARY,
mode=CopyMode.STREAM,
) as writer:
for i in range(10):
put = Put(writer.schema)
put.set_object("id", i)
put.set_object("name", "name0")
writer.write(put)
Asynchronous API
import asyncio
from hologres import AsyncHoloClient, HoloConfig, Put, CopyFormat, CopyMode, OnConflictAction
async def main():
config = HoloConfig(
host="<HOST>",
port=<PORT>,
database="<DATABASE>",
username="<USERNAME>",
password="<PASSWORD>",
on_conflict_action=OnConflictAction.INSERT_OR_UPDATE,
)
async with AsyncHoloClient(config) as client:
columns = ["id", "name"]
async with await client.copy_writer(
"copy_demo",
columns=columns,
fmt=CopyFormat.BINARY,
mode=CopyMode.STREAM,
) as writer:
for i in range(10):
put = Put(writer.schema)
put.set_object("id", i)
put.set_object("name", "name0")
await writer.write(put)
asyncio.run(main())
Write through a Stage
holo-client-py can write data to an internal Hologres Stage through CopyStageWriter and then load the data from the Stage into the target table with an INSERT statement. This approach is atomic and suits scenarios in which you stage data first and import it in bulk. For more information about Stage, see Near real-time import via Stage.
from hologres import HoloClient, HoloConfig, Put
config = HoloConfig(
host="<HOST>",
port=<PORT>,
database="<DATABASE>",
username="<USERNAME>",
password="<PASSWORD>",
)
with HoloClient(config) as client:
# CREATE TABLE copy_stage_demo (id INT NOT NULL, name TEXT NOT NULL, address TEXT, PRIMARY KEY(id));
columns = ["id", "name", "address"]
stage_name = "my_stage"
# Create an internal Stage with a TTL of 7200 seconds (2 hours). The Stage is cleaned up automatically after the TTL expires
client.create_stage(stage_name, ttl_seconds=7200)
try:
with client.copy_stage_writer(
"copy_stage_demo",
stage_name,
columns=columns,
) as writer:
for i in range(10):
put = Put(writer.schema)
put.set_object("id", i)
put.set_object("name", f"name{i}")
put.set_object("address", f"address{i}")
writer.write(put)
finally:
# Drop the temporary Stage (it is also cleaned up automatically based on its TTL)
client.drop_stage(stage_name)
Query data
holo-client-py provides two kinds of query interfaces: point queries by full primary key (Get and async_get) and range scans by primary key prefix (Scan). Both support synchronous and asynchronous calls.
Query by full primary key
Synchronous API
The following example uses Get to query data by full primary key.
from hologres import HoloClient, HoloConfig, Get
config = HoloConfig(
host="<HOST>",
port=<PORT>,
database="<DATABASE>",
username="<USERNAME>",
password="<PASSWORD>",
read_parallelism=4, # In synchronous mode, start 4 reader threads to query in parallel
)
with HoloClient(config) as client:
# create table t0(id int not null, name0 text, address text, primary key(id))
schema = client.get_table_schema("t0")
get = Get.builder(schema).set_primary_key("id", 1).build()
record = client.get(get)
if record:
print(record.get_object("name0"))
Non-blocking API (async_get)
async_get() returns a concurrent.futures.Future, which suits high-QPS point queries issued from a synchronous client. Submit multiple Get requests first and then wait for all results together, so that the reader threads can form full batches.
from hologres import HoloClient, HoloConfig, Get
config = HoloConfig(
host="<HOST>",
port=<PORT>,
database="<DATABASE>",
username="<USERNAME>",
password="<PASSWORD>",
read_parallelism=4,
)
with HoloClient(config) as client:
schema = client.get_table_schema("t0")
# Submit get requests in bulk without blocking on each result
futures = []
for i in range(100):
get = Get.builder(schema).set_primary_key("id", i).build()
futures.append(client.async_get(get))
# Wait for all results together
for future in futures:
record = future.result()
if record:
print(record.get_object("name0"))
Asynchronous API
Submit Get requests in bulk with asyncio.create_task so that the reader tasks can form full batches and reach maximum throughput. Do not call await client.get() for one request at a time, because each batch would then contain a single request.
import asyncio
from hologres import AsyncHoloClient, HoloConfig, Get
async def main():
config = HoloConfig(
host="<HOST>",
port=<PORT>,
database="<DATABASE>",
username="<USERNAME>",
password="<PASSWORD>",
read_parallelism=4, # In asynchronous mode, start 4 reader tasks to query concurrently
)
async with AsyncHoloClient(config) as client:
schema = await client.get_table_schema("t0")
tasks = []
for i in range(100):
get = Get.builder(schema).set_primary_key("id", i).build()
tasks.append(asyncio.create_task(client.get(get)))
records = await asyncio.gather(*tasks)
for record in records:
if record:
print(record.get_object("name0"))
asyncio.run(main())
Scan queries
Synchronous API
Scan scans data by primary key prefix and can return multiple rows, as shown in the following example.
from hologres import HoloClient, HoloConfig, Scan, SortKeys
config = HoloConfig(
host="<HOST>",
port=<PORT>,
database="<DATABASE>",
username="<USERNAME>",
password="<PASSWORD>",
)
with HoloClient(config) as client:
# create table t0(id int not null, name0 text, address text, primary key(id))
schema = client.get_table_schema("t0")
scan = (Scan.builder(schema)
.add_equal_filter("id", 102)
.add_range_filter("name0", start="3", end="4")
.with_selected_columns(["address"])
.build())
# Equivalent to select address from t0 where id=102 and name0>='3' and name0<'4' order by id
records = client.scan(scan)
for record in records:
print(record.get_object("address"))
# Without sorting
scan = (Scan.builder(schema)
.add_equal_filter("id", 102)
.add_range_filter("name0", start="3", end="4")
.with_selected_columns(["address"])
.set_sort_keys(SortKeys.NONE)
.build())
records = client.scan(scan)
Asynchronous API
import asyncio
from hologres import AsyncHoloClient, HoloConfig, Scan
async def main():
config = HoloConfig(
host="<HOST>",
port=<PORT>,
database="<DATABASE>",
username="<USERNAME>",
password="<PASSWORD>",
)
async with AsyncHoloClient(config) as client:
schema = await client.get_table_schema("t0")
scan = (Scan.builder(schema)
.add_equal_filter("id", 102)
.add_range_filter("name0", start="3", end="4")
.with_selected_columns(["address"])
.build())
records = await client.scan(scan)
for record in records:
print(record.get_object("address"))
asyncio.run(main())
Performance comparison
Write and query methods differ in throughput and latency. Choose the method that fits your workload.
Write performance
|
Write method |
Throughput |
Latency |
Use case |
|
Fixed Copy (synchronous) |
Highest |
Lowest |
Real-time writes with no DELETE operations, where maximum throughput matters most. |
|
|
High |
Low |
General writes, with support for DELETE operations and primary key conflict strategies. |
|
|
Medium |
Medium |
|
|
|
Low |
Low |
Low-throughput scenarios with minimal resource usage. |
Synchronous Put parallelizes writes through multiple processes and shared memory and is not limited by the GIL, so raising write_parallelism increases throughput linearly. For asynchronous Put, the producers and flush share one event loop, so a higher write_parallelism brings limited gains. Fixed Copy uses a streaming protocol that bypasses SQL construction and batching, and reaches maximum throughput over a single connection. A higher proportion of DELETE operations in your write requests lowers the overall number of records written per second.
Query performance
|
Query method |
Throughput |
Latency |
Use case |
|
|
Highest |
High |
High-QPS point queries from multi-threaded producers, where maximum throughput matters most. |
|
|
Highest |
Medium |
High-QPS point queries in |
|
|
Medium |
Low |
Regular point queries in synchronous applications, where latency matters most. |
|
|
Low |
Medium |
Prefix range scans that return multiple rows. |
async_get() returns a concurrent.futures.Future. Multiple threads can submit requests in bulk and then wait for all results together, which lets the reader threads form full batches and reach maximum throughput. AsyncHoloClient.get() combined with asyncio.create_task reaches the same throughput when requests are submitted in bulk; awaiting each call individually leaves one request per batch and degrades throughput. get() blocks until a single result returns, so the reader threads cannot form full batches and throughput is bounded by the round-trip latency of one request. Raising read_parallelism adds reader threads or tasks, which markedly improves throughput in bulk submission mode. Scan returns multiple rows in a single RPC call, so its throughput depends on the result set size and network latency.
Exception handling
holo-client-py raises two kinds of exceptions: HoloClientWithDetailsException, which carries the details of the records that failed to write, and HoloClientException, which is usually an unrecoverable fatal error. The following example shows how to handle both.
from hologres import HoloClient, HoloConfig, Put
from hologres.exceptions import HoloClientException, HoloClientWithDetailsException
config = HoloConfig(
host="<HOST>",
port=<PORT>,
database="<DATABASE>",
username="<USERNAME>",
password="<PASSWORD>",
)
with HoloClient(config) as client:
schema = client.get_table_schema("t0")
put = Put(schema)
put.set_object("id", 1)
put.set_object("name", "name0")
try:
client.put(put)
client.flush()
except HoloClientWithDetailsException as e:
# Exception that carries the details of the failed records
for record, cause in e.details:
print(f"Failed record: {record}, cause: {cause}")
except HoloClientException as e:
# Exceptions other than HoloClientWithDetailsException are usually fatal
raise
Custom operations
You can run custom SQL statements such as DDL through the sql method, as shown in the following example.
from hologres import HoloClient, HoloConfig
config = HoloConfig(
host="<HOST>",
port=<PORT>,
database="<DATABASE>",
username="<USERNAME>",
password="<PASSWORD>",
)
with HoloClient(config) as client:
client.sql("CREATE TABLE t0(id int)")
Appendix: HoloConfig parameters
The following sections describe all HoloConfig parameters.
Basic parameters
|
Parameter |
Default |
Description |
|
|
None |
Required. The endpoint of the Hologres instance. |
|
|
None |
Required. The port number. |
|
|
None |
Required. The database name. |
|
|
None |
Required. The username. |
|
|
None |
Required. The password. |
|
|
|
The application_name of the connection. |
Write parameters
|
Parameter |
Default |
Description |
|
|
|
The maximum concurrency for handling |
|
|
|
The strategy used on a primary key conflict when the INSERT target is a table with a primary key. Valid values:
|
|
|
|
The maximum batch size of each write thread. |
|
|
|
The maximum batch size in bytes of each write thread. |
|
|
|
The maximum batch size in bytes across all tables. |
|
|
|
Triggers a batch commit when this interval has elapsed since the last commit. Unit: milliseconds. |
|
|
|
Specifies whether to deduplicate batched records before writing. |
|
|
|
When this parameter is disabled, the written data does not generate a |
|
|
|
Specifies whether to strip |
Query parameters
|
Parameter |
Default |
Description |
|
|
|
The concurrency of point queries. In synchronous mode, this is the number of reader threads. In asynchronous mode, it is the number of reader tasks. |
|
|
|
The maximum number of |
|
|
|
The size of the request buffer queue in asynchronous mode. |
|
|
|
The timeout of a |
Connection parameters
|
Parameter |
Default |
Description |
|
|
|
The number of retries for writes and queries when a connection fails. |
|
|
|
The base retry wait time. Unit: milliseconds. |
|
|
|
The increment of the retry wait time. Unit: milliseconds. |
|
|
|
The maximum idle time of a database connection, after which the connection is released. Unit: milliseconds. |
|
|
|
The local cache time of |
|
|
|
When this parameter is enabled, |