MaxCompute acceleration

Background

MaxCompute is a fully managed, exabyte-scale data warehouse for batch storage and computation of structured data. For more information, see What is MaxCompute?
Hologres is a real-time data warehouse for writing, updating, processing, and analyzing large volumes of data. It queries MaxCompute data directly through external databases and foreign tables without data import, and connects to mainstream BI tools via PostgreSQL protocol. For more information, see Hologres + BI.
Starting from Hologres V3.1, the direct read capability is upgraded to version 2.0, rebuilt on the MaxCompute C++ Native SDK. Version 2.0 reads directly from the Apsara Distributed File System, eliminating MaxCompute-side computational costs and rate limiting, with over 2x the read performance of the Storage API method.

Note
To use the new routing path, add the following GUC at the session level. Place the SET hg_experimental_external_catalog_routing = 'odps:common_table'; command before your SQL query and execute them together.

Scenarios

Query MaxCompute data with external databases or foreign tables

External databases and foreign tables eliminate data movement, letting a single storage system support both offline and real-time computing workflows.

Execute the CREATE EXTERNAL DATABASE statement to create an external database that maps schemas and tables in a MaxCompute project. Execute the IMPORT FOREIGN SCHEMA statement to create multiple foreign tables at a time. To accelerate queries on specific MaxCompute tables or fields, execute the CREATE FOREIGN TABLE statement.

Note

Starting from Hologres V3.0, the 200 GB and 512-partition scan limits per foreign table query are removed. Large queries consume more resources. Isolate them with Getting started with virtual warehouse instances or Overview of Serverless Computing.

Import MaxCompute data to Hologres

Import data into Hologres to leverage indexing and storage optimization for real-time BI analysis, high-concurrency point queries, and multi-table joins. For more information, see Import data from MaxCompute using SQL.

Export data from Hologres to MaxCompute

Write real-time data and aggregated results back to MaxCompute for historical archiving, hot-cold tiered storage, or offline warehouse replenishment using INSERT INTO or INSERT OVERWRITE. For more information, see Export data to MaxCompute.

Features

MaxCompute acceleration capabilities by Hologres version:

Hologres version	Supported capability	Description
V3.1	Direct read capability upgraded to version 2.0, rebuilt on the MaxCompute C++ Native SDK for improved performance and user experience.	Version 2.0 reads directly from the Apsara Distributed File System, eliminating MaxCompute-side computational costs and rate limiting. Read performance exceeds 2x that of the Storage API method.
	Direct reading of MaxCompute Delta table data. Only full access to the latest snapshot is supported.	Delta tables are an incremental data format in MaxCompute that supports minute-level ingestion, UPSERT semantics, and CDC-format incremental reads and writes using AliORC. For more information, see Access MaxCompute Delta tables.
	Direct reading of MaxCompute data with dynamic data masking enabled.	Dynamic data masking hides or replaces sensitive data in real time during access. For more information, see Dynamic data masking.
	Direct reading of MaxCompute Schema Evolution tables.	Supported operations: adding columns, dropping columns, modifying column types, and reordering columns.
	Direct reading of MaxCompute clustered tables, including range-clustered tables with DATETIME, TIMESTAMP, or DECIMAL cluster columns.	None.
V3.0.7	CREATE EXTERNAL DATABASE statement supported for mapping schemas and tables in a MaxCompute project.	Create an external database to load external data source metadata into Hologres for centralized metadata management with the lakehouse architecture. For more information, see CREATE EXTERNAL DATABASE.
V3.0.7	Scan limits of 200 GB and 512 partitions per foreign table query removed.	Use virtual warehouse instances or Serverless Computing to isolate large queries from other workloads.
V2.2.1	The Auto Load feature supports MaxCompute projects that use the three-layer model.	MaxCompute schemas classify tables, resources, and functions in a project. For more information about schemas, see Schema. For more information about Auto Load, see Load foreign tables automatically.
	The Auto Load feature supports automatic updates of metadata.	None.
	The Auto Load feature supports schema evolution on MaxCompute foreign tables.	Schema evolution includes deleting columns, reordering columns, and changing column data types. For more information, see Load foreign tables automatically.
	Service-linked role of Hologres supported for accessing MaxCompute.	A service-linked role is a RAM role that allows authorized cross-service access. For more information, see Hologres service-linked role.
V2.1	Write data to tables in a specified schema in a MaxCompute three-layer model project.	MaxCompute schemas classify tables, resources, and functions in a project. For more information, see Schema.
V2.1	Access tables in a specified MaxCompute three-layer model schema from the HoloWeb console.	For more information, see Create MaxCompute foreign tables in HoloWeb.
V2.0	Query engine upgraded to Hologres Query Engine (HQE) for improved compatibility and stability.	None.
V1.3	Foreign table queries on MaxCompute transactional tables.	None.
	Read data from MaxCompute tables in the schema evolution state.	A table is in the schema evolution state when columns are deleted, reordered, or have their data types changed.
	Dual-signature authentication between Hologres and MaxCompute.	For more information, see Hologres external tables.
	Access MaxCompute tables encrypted with Bring Your Own Key (BYOK).	For more information, see Query encrypted MaxCompute data.
V1.1.43	Auto Load feature supported. Foreign tables are automatically created for queried MaxCompute tables or all tables as needed.	For more information, see Load foreign tables automatically.

Limits

Limits for using external databases or foreign tables to accelerate MaxCompute queries:

Your Hologres instance must be in the same region as the MaxCompute project.
Only internal MaxCompute tables are supported. External tables and views are not supported.
MaxCompute partition fields are mapped to regular Hologres fields and can be used as filter conditions.
MAP and STRUCT data types are not supported.
Range-clustered tables with DATETIME, TIMESTAMP, or DECIMAL clustering key columns are not supported.
Only standard storage is supported. Infrequent access (IA) and long-term storage are not supported.