Real-time data consumption

Features

High-speed data import API

gRPC is a high-performance, open-source RPC framework developed by Google. The high-speed data import API uses the gRPC protocol and segment-direct-write technology to transfer data efficiently, with flexible control over write and update operations. For more information, see High-speed data import API.

Real-time data source consumption

Real-time data source consumption allows an AnalyticDB for PostgreSQL database to consume data from Kafka topics in real time, combining stream processing and batch processing for analytics. For more information, see Consume data from Kafka in real time.

Scenarios

• Build real-time analytics reports and other real-time analytics applications.

• Build a unified real-time data warehouse for both stream and batch processing using incremental real-time materialized views.

• Use the high-speed data import API to achieve higher UPSERT throughput when SQL UPSERT or UPDATE statements cause poor write performance.

• Import data from Kafka topics into AnalyticDB for PostgreSQL without using other data integration tools.

Benefits

• Consuming Kafka data directly reduces dependencies on external real-time processing components.

• Writing data directly through compute nodes instead of client nodes significantly increases write throughput.

• Delivers higher performance than SQL-based UPSERT or UPDATE statements.

• Generates minimal load on the master node during writes.