Release notes archive

Core feature enhancements

Description

References

Computing engine enhancements and performance optimization

• Upgraded core execution engine (QE v2)

  • Breakthrough in concurrent processing: When worker resources are sufficient but shard distribution is sparse, QE v2 increases parallelism for CPU-intensive operators (JOIN, AGG) without shuffle overhead, fully utilizing resources and reducing query latency.

  • Enhanced information in adaptive execution: The execution plan now displays more detailed operator states, including filter conditions, join conditions, redistribution columns, and filter columns.

• Intelligent query acceleration (History-Based Optimization & Rewrite)

  • Automatic query rewrite for Dynamic Tables: The optimizer automatically redirects single-table queries on base tables to pre-computed Dynamic Tables, achieving millisecond-level responses without SQL changes.

  • Expanded History-Based Optimization (HBO) rules: Introduced new rules such as Adaptive Join Order, Adaptive Segment Agg, and Adaptive Runtime Filter.

  • Forced optimization for large queries: For queries that run longer than 20 seconds and can be optimized, the engine forces subsequent executions to use the HBO plan. The execution plan clearly indicates when the HBO plan is used.

• Advanced operator and real-time analytics optimizations

  • Enhanced high-speed UV calculation: Supports User Id Encoding, which maps high-cardinality strings to integer IDs. Combined with Dynamic Table incremental computation, this enables low-cost, high-speed large-scale real-time UV calculation.

  • Optimized stream-based execution for Window TopN: In Hash Partition scenarios, Window TopN now uses stream-based mini-batch sorting per partition instead of full sorting, reducing peak memory pressure.

  • Enhanced scan performance: The TopN Filter in the Scan process can now use the result cache, effectively reducing underlying scan overhead.

  • Optimized Hybrid DML for row-oriented tables: A new sampling mechanism for INSERT on row-oriented tables dynamically determines the join strategy, improving performance for large data imports into small tables.

• Dynamic Table query rewrite

• Supported incremental refresh functions

• Hologres Dynamic Table solution for long-period UV calculation

• Optimize query performance

Deep lakehouse integration

• EXTERNAL_FILES analytics function without external tables

  • Simplified ad hoc query workflow: The new EXTERNAL_FILES function supports direct analysis of Parquet or ORC files on OSS by using standard SQL, without the need to create an external table.

  • Data export capability: This function exports data from internal tables directly to OSS, optimizing workflows for cold data archiving and cross-platform data exchange.

• Enhanced direct-read capabilities for MaxCompute

  • Fine-grained pruning support: When directly reading from partitioned tables, the engine now supports partition pruning and clustering key pruning, which significantly reduces I/O scans.

  • Native type compatibility: Added support for directly reading the MaxCompute JSON data type. This improves the processing efficiency of semi-structured data.

• Near real-time ingestion support

  • Near real-time ingestion support: Supports near real-time ingestion through a temporary storage stage, balancing write performance, resource cost, and data visibility better than existing real-time and offline batch write paths.

• File reading function (EXTERNAL_FILES)

• Access MaxCompute by using a common table link

Multi-modal search and analysis

• Enhanced full-text search

  • Enhanced tokenizer capabilities: Supports IK, N-gram, and Pinyin tokenizers for Chinese text search, log fuzzy search, and Pinyin-based name search.

• Enhanced vector search

  • Optimized vector index resources: The HGraph in-memory index now supports compression. This can save 50% of memory with only a 5% performance trade-off.

  • Optimized vector index performance: HGraph now attaches column data to index files, enabling direct column retrieval during search without querying the target table.

• Full-text inverted index

• HGraph index usage guide (Recommended)

Enterprise-grade O&M and system stability

• Architectural workload isolation

  • Isolated query thread pool: Query execution is now isolated in a dedicated thread pool, preventing interference among query workloads, real-time writes, and storage engine control paths in high-concurrency scenarios.

• Enhanced Serverless capabilities

  • Enhanced adaptive Serverless Computing: Supports automatic workload isolation, routing traffic to a Serverless Computing resource pool under high load to improve throughput and stability.

• Enhanced resource management and stability

  • Dynamically switch warehouses with SQL: Switch the connected warehouse using standard SQL for enhanced multi-tenant isolation and flexible O&M.

  • Enhanced table schema change (Rebuild) capability: Optimized metadata and data migration reduces the write-unavailability window during schema changes to under 10 seconds. Tables remain queryable.

  • Lock wait timeout control: The hg_experimental_lock_wait_timeout_ms parameter is introduced, allowing users to customize the lock-wait timeout for non-FixedQE write paths.

  • Fine-grained memory management:

    • Full-text indexes are now loaded through the block cache instead of being resident in memory, which effectively frees up memory space.

    • The mini-batch algorithm is optimized, and stream-based ConcatRecordBatch is now supported. This partially mitigates errors caused by Arrow structures exceeding 2 GB and reduces peak memory pressure.

• Adaptive Serverless Computing

• Use SQL to switch the current warehouse

• REBUILD

Feature

Description

References

Enhanced AI and retrieval capabilities

• (Beta) The LLM-powered AI Function lets you analyze and retrieve unstructured data such as text and images. All models are fully hosted in Hologres AI nodes to ensure performance, scalability, governance, data security, and compliance. The AI Function supports the following use cases:

  • Use embedding models with an Object Table to perform vector search and full-text search on unstructured data such as text and images.

  • Summarize key insights from text and images.

  • Filter and classify text using natural language, and translate and localize multilingual content.

  • Perform sentiment analysis and dimension-based analysis to optimize services.

  • Parse documents to support data analysis and RAG workflows.

• (Beta) HGraph vector search improves performance by 10x+ with hybrid scalar-vector search for image/video retrieval, recommendations, and security and fraud detection. Hybrid in-memory and on-disk indexes reduce memory by 80% with only 5% QPS drop (per VectorDB Benchmark), enabling cost-effective vector search on large datasets such as autonomous driving.

• (Beta) This version adds support for a full-text inverted index and built-in tokenizers to enable full-text search. Supported scenarios include:

  • Searching by keyword, phrase, or natural language.

  • Calculating BM25 scores for text similarity retrieval.

  • Combining with vector search to support hybrid full-text and vector search.

  • Supporting hybrid full-text and scalar search.

• (Beta) Global secondary index enables efficient key-value point queries on non-primary key columns, ideal for feature stores and e-commerce platforms.

• AI Function

• Full-text inverted index

• Global secondary index

Engine enhancements

• This version supports the TopN Runtime Filter to accelerate data queries in TopN scenarios.

• (Beta) Hologres now supports Time Travel for internal tables, letting you query historical data at any point within a defined time period.

• (Beta) This version supports History-Based Optimization (HBO). The system collects execution details of slow queries, automatically analyzes query plans for optimization opportunities, and intelligently adjusts the plans based on this history.

/

Dynamic Table

(Beta) External Dynamic Tables write processed data back to Paimon in full or incremental mode, supporting near real-time warehouse-to-warehouse, lake-to-warehouse, warehouse-to-lake, and lake-to-lake processing. Combined with serverless instances, this enables ultra-low-cost data lake processing.

• Dynamic Table

• Serverless instances

Syntax enhancements

This version supports the QUALIFY clause, letting you filter the results of window functions.

QUALIFY (Beta)

Function and ecosystem extensions

Enhanced compatibility with the ClickHouse ecosystem. This version adds support for multiple time truncation functions, including toDayOfMonth, toDayOfYear, and toHour. These functions can improve performance by up to 50% compared with the original extract(field from timestamp) function.

Date and time functions

Serverless and elasticity

• (Beta) Virtual warehouse instances now support multi-cluster elastic scaling. A virtual warehouse consists of one or more clusters and automatically scales out based on load for resource isolation and high concurrency.

• (Beta) Virtual warehouse instances provide strong write isolation. Any virtual warehouse can perform batch writes without relying on the leader warehouse or loading a table group.

• (Beta) The hot-upgrade capability for virtual warehouse instances is enhanced. During an upgrade, SQL execution is lossless and connections are seamlessly re-established after a brief interruption.

• You can now switch the virtual warehouse for the current connection using SQL.

• Serverless Computing now supports reading from and writing to DLF foreign tables.

• Multi-cluster and auto scaling (Beta)

• Virtual warehouse management

• Virtual warehouse instance architecture

• Manage user permissions for virtual warehouses

• Switch the virtual warehouse for the current connection using SQL

• Accelerate access to data lakes in OSS by using DLF

• Use Serverless Computing to run read and write tasks

Data lake analytics

(Beta) MaxCompute data mirroring pre-warms data into mirrored internal tables with zero ETL, matching Hologres internal table query performance without dedicated synchronization jobs.

/

Core features

Description

Related documents

Engine enhancements

• Supports adaptive reuse or inlining of CTE expressions.

• Optimizes BETWEEN expression calculations.

CTE reuse strategy optimization

Dynamic Table

• The incremental refresh mode now supports functions like ARRAY_AGG and STRING_AGG.

• DataWorks Data Map now supports lineage analysis of Dynamic Tables.

• ARRAY_AGG

• STRING_AGG

Enhanced serving capabilities

Fixed Plan now supports simple expressions, enabling high-QPS key-value point queries and writes across more scenarios.

Accelerate SQL execution using Fixed Plan

Function and ecosystem extensions

Adds support for lambda expressions and higher-order array functions.

Lambda expressions and related functions

Serverless capabilities

• Serverless Computing now supports reading from and writing to encrypted tables, including internal tables and MaxCompute external tables.

• Query Queue is enhanced to classify SQL statements by table and automatically route requests for specific tables to Serverless Computing.

• Serverless Computing user guide

• Query Queue

Data lake analytics capabilities

• Supports data lake table mirroring for Paimon to accelerate data lake queries.

• Integrates with the DLF 2.5 metadata management system, supporting Paimon catalogs on the data lake by using the DLF REST API.

• This feature uses zero ETL to pre-warm data from the data lake into mirrored internal tables, significantly improving query efficiency.

• The Time Travel feature for Paimon lets you read historical data by specifying a timestamp or tag.

• Supports reading branch data from Paimon tables, including reading from a specified branch and reading with fallback behavior.

• You can now disable full table scans on partitioned tables to prevent excessive resource consumption.

• Improves performance by 2x for queries on Paimon tables in the TPC-H 1 TB benchmark.

• Data lake table mirroring

• Access Paimon data based on DLF catalog

Ecosystem enhancements

Adds support for trimming and compressing binary logs to reduce I/O consumption.

/

Feature

Description

Documentation

The Hologres serverless instance (Beta) is available for a free, invitation-only preview.

A new serverless instance type built on cloud-native architecture provides flexible, scalable Hologres computing and storage without pre-purchasing dedicated resources or incurring idle costs.

To apply for a trial, fill out the form using your primary Alibaba Cloud account.

Serverless instance

Core feature enhancements

Description

Related documentation

Dynamic table

• Supports dynamic partitioning based on logical partitioned tables to significantly simplify their use.

• Adds an Auto Refresh mode. You can specify the desired data freshness, and the engine automatically optimizes the refresh strategy to provide greater flexibility.

• The incremental refresh now supports dual-stream JOIN scenarios, enhancing flexibility in real-time data processing.

• The incremental refresh now supports RoaringBitmap functions, enabling incremental calculations for complex scenarios like unique visitor (UV) and page view (PV) analysis.

• Full refresh mode now supports Adaptive Execution (Beta), improving stability by reducing OOM errors through dynamic resource derivation and plan adjustments while maintaining low latency.

• CREATE DYNAMIC TABLE

• Incremental refresh

Serverless capabilities

• Serverless Computing now supports more complex DML scenarios, including INSERT OVERWRITE, RESHARDING, and CTAS, as well as stored procedures, REBUILD operations, and encrypted tables.

• Introduces lossless virtual warehouse scaling (Beta) to ensure business continuity during scaling operations.

• Supports automatic throttling (Beta). This feature dynamically limits query queue concurrency based on the workload, significantly improving cluster stability.

• Adds adaptive routing for Serverless Computing, which automatically routes large queries to serverless resources for execution.

• Supports setting a daily usage limit for Serverless Computing.

• Optimizes cache reuse for Serverless Computing in high-concurrency scenarios to improve query performance.

• Supports using a RAM role to scale virtual warehouses.

• Serverless Computing User Guide

• Serverless Computing Release Notes

• Manage virtual warehouses

• Scheduled scaling (Beta)

• Automatic throttling (Beta)

• Daily usage limit

Performance optimization and query enhancements

• The refactored query engine (QEv2) supports computation on lightweight encoding, delivering a 33% performance improvement on the TPC-H 1 TB benchmark test.

• The engine's adaptive optimization automatically pushes down aggregation plans based on a cost model. This reduces the data volume in JOIN operations, significantly lowering both latency and CPU overhead.

• The engine now automatically infers the NOT NULL attribute for JOIN keys and pushes down the NOT NULL condition to filter out NULL values earlier. It also automatically eliminates constant fields from the GROUP BY clause in aggregations.

• Adds a Query Cache feature that accelerates queries by caching their results.

• Enhances the diagnostic capabilities of the hg_stats_missing view by adding new fields, including autovacuum_enabled (indicates whether AUTO ANALYZE is enabled) and reason (explains why statistics are missing). This simplifies diagnosing and completing missing statistics.

• The AUTO ANALYZE system is optimized. Enhanced statistics collection now automatically retrieves row counts for tables without statistics, improving query plan quality. Additionally, statistics are more persistent because they are no longer cleared by schema changes that do not affect statistical validity, such as RENAME operations or moving data to cold storage. This reduces system load and improves the quality of execution plans.

• Benchmark Test Results

• Query Cache

• HG_STATS_MISSING View

Data management and write optimization

• Storage and index optimization

  • Introduces new logical partitioned tables (Beta). This feature offers greater flexibility with lightweight metadata management and simplified data administration.

  • Adds support for stored generated columns (Beta), which simplify data processing and accelerate queries through pre-computation.

  • Introduces the REBUILD tool (Beta). It supports modifying lightweight indexes (such as distribution key, clustering key, and segment key) and making other table structure changes.

• Write capability enhancements

  • The COPY command for primary key tables now supports partial column updates, reducing the need for FIXED COPY scenarios. When a fixed frontend (FE) is used, its connections do not count toward the original FE's connection limit.

  • Adds native support for the INSERT OVERWRITE syntax, allowing for more flexible overwrite operations on both standard tables and logical partitioned tables.

• Logical partitioned table (Beta)

• Generated columns

• REBUILD

• COPY

• Best Practices for Batch Data Writes in Hologres

• INSERT OVERWRITE

Function and ecosystem extensions

• New built-in functions

  • Adds funnel functions for property association and dimension grouping.

  • Expands the set of Spark- and Presto-compatible functions to improve development efficiency across engines.

  • Some Roaring Bitmap functions now support 64-bit integers, expanding their use cases in user profile analysis.

• Remote function support

  • Supports invoking a remote UDF through Function Compute, providing a flexible way to extend ETL capabilities.

• Property association funnel function (finder_funnel)

• Dimension grouping funnel function (finder_group_funnel)

• Spark-compatible functions

• Presto-compatible functions

• Data types summary

• RoaringBitmap functions

• User-defined functions (Remote UDF)

Enterprise-grade feature upgrades

• Enhances enterprise-grade permission management. You can now specify a security token in the PostgreSQL protocol connection options to log in with a RAM role by using JDBC or PSQL.

• Introduces the table recycle bin, which allows you to recover tables and their data after accidental deletion.

• Enhances data masking to support masking for view results and non-TEXT data types. This significantly strengthens the protection of sensitive data and helps prevent brute-force attacks.

• Table recycle bin

• Data masking

Data lake analytics capabilities

• External data source support

  • Enhances integration with external data sources, allowing seamless access from mainstream BI tools like Quick BI, Tableau, and Superset.

  • You can now specify a metadata refresh interval for external data sources.

  • ANALYZE and AUTO ANALYZE operations are now supported for external data sources.

  • Supports writing data to Paimon primary key tables by using INSERT INTO, which facilitates data flow between the data lake and the warehouse.

  • Supports writing data to Iceberg tables by using INSERT INTO, expanding compatibility with open data lake formats.

• Transparent acceleration for MaxCompute

  • Direct access to MaxCompute data is now V2.0 (Beta). The direct-read mechanism has been re-architected based on the MaxCompute C++ Native SDK, further improving the performance and user experience of accessing MaxCompute data sources.

  • Supports directly reading MaxCompute Delta Tables (Beta).

  • Supports directly reading dynamically masked data from MaxCompute, providing an integrated data masking experience (Beta).

  • Supports directly reading MaxCompute tables with evolved schemas. Supported operations include adding, deleting, and reordering columns, and modifying column types.

  • You can now use the new Data Development feature in DataWorks to map MaxCompute projects, schemas, and tables to Hologres with a single click.

  • You can now use the new Data Development feature in DataWorks to import data from MaxCompute tables into Hologres with a single click.

• ANALYZE and AUTO ANALYZE

• Accelerate access to MaxCompute

• Access MaxCompute Delta Tables from Hologres

• Map MaxCompute metadata to Hologres external tables

• Synchronize MaxCompute data to Hologres internal tables

Feature enhancements

Description

Related documentation

Engine capability enhancements

• Introduces Dynamic Table with full and incremental refresh modes. Data flows and refreshes automatically, enabling real-time data layering, unified stream-batch processing, and flexible data freshness for analytics.

• Upgrades Serverless Computing to support SELECT and COPY operations. This provides a cloud-native resource solution for temporary, large-scale queries.

• Virtual Warehouses now support scheduled auto scaling (Beta), allowing you to scale computing resources based on a schedule. This helps you meet fluctuating demands, prevent workload interference, and maximize resource utilization.

• Adds support for query queues:

  • You can create query queues based on business requirements and configure their concurrency and length to improve instance stability.

  • Enhances large query governance. You can now set a timeout for large queries in a queue to reduce their impact on the instance. Timed-out queries can then be rerun using Serverless Computing resources.

  • You can use Serverless Computing to execute all queries in a query queue.

• Improves the performance of writes, updates, and point queries using Fixed Plan by approximately 10% compared to V2.2.

• Supports executing INSERT OVERWRITE on parent partitioned tables.

• Adds support for stored procedures (Beta), which allow you to define commonly used SQL to simplify application logic.

• Enhances schema evolution to support modifying column data types.

• Enhances the COPY command. When you import data into a table with a primary key, you can configure a full-row update policy to avoid primary key conflict errors.

• Adds cross join support for better non-equality predicate performance, and partial aggregation to limit memory usage and reduce OOM risks from queries with many GROUP BY keys.

• Enhances the storage engine, improving the update performance of column-oriented tables when a segment key is set to a non-sequentially written column.

• Enhances function capabilities:

  • Enhances the TRY_CAST function to support DATE, TIMESTAMP, and TIMESTAMPTZ as target data types.

  • Supports running ARRAY_AGG and STRING_AGG functions with DISTINCT and ORDER BY clauses on the Hologres Query Engine (HQE) to improve query performance.

• Serverless Computing usage guide

• Scheduled auto scaling (Beta)

• Use Serverless Computing resources to execute queries in a query queue

• Best practices for stress testing data writes, updates, and point lookups

• INSERT OVERWRITE

• Stored procedures

• ALTER TABLE

• COPY

• EXPLAIN and EXPLAIN ANALYZE

• Troubleshoot common OOM issues

• Type conversion functions

• Function release history

O&M and stability improvements

• Introduces SQL audit in collaboration with SLS to monitor, record, and analyze database activities to ensure data security and policy compliance.

• Enhances the scale-out capabilities of Virtual Warehouses, so that read and write operations can continue without interruption during the scaling process.

• Supports logging DML operations and queries that take less than 100 ms to the query log system table in an aggregated format, which improves SQL observability and analysis.

• CloudLens for Hologres

• Manage Virtual Warehouses

• Scheduled auto scaling (Beta)

• View and analyze slow query logs

Lakehouse integration

• The new External Database feature supports catalog-level metadata mapping for data sources such as Data Lake Formation (DLF) and MaxCompute, enhancing the metadata and data management capabilities of data lakes.

• Supports connecting to a Hive metastore to map metadata and transparently accelerate EMR clusters.

• Supports writing data to Paimon append tables using the INSERT INTO statement.

• Supports reading the Iceberg table format, which further expands the data lake ecosystem.

• Enhances security capabilities. By default, service-linked roles are used for identity pass-through when accessing DLF 2.0. You can also access DLF 2.0 using a RAM role.

• Enhances table capabilities.

  • The reader for the Delta Lake format is re-architected, significantly improving read performance.

  • Supports Paimon deletion vector optimization, improving query performance in scenarios where a large amount of data has been deleted but not yet compacted.

• Starting from V3.0.22, Hologres supports accessing MaxCompute Delta Tables.

• Access data in an OSS data lake by using a Hive metastore (Beta)

• Access MaxCompute Delta Tables from Hologres

Feature enhancements

Description

Related documentation

Hologres Serverless Computing completes its Beta phase

Hologres Serverless Computing has completed its Beta phase, is now production-ready, and is covered by an SLA. It becomes generally available on July 1, 2024 (UTC+8).

• Usage and billing

• Serverless Computing

• Hologres Real-time Data Warehouse Service Level Agreement (SLA)

Feature enhancements

Description

Related documentation

Engine capability enhancements

• Overall engine performance improves ~15% over previous versions, focusing on HQE and Query Optimizer (QO) evolution:

  • Optimizes the Hologres Query Engine (HQE), improving performance in the following scenarios:

    • Enhances the Runtime Filter capability to support shuffle join scenarios, improving query efficiency in these scenarios by approximately 30%.

    • Optimizes the RPC connection mechanism of HQE. Data within each worker is merged before being distributed to other workers. This significantly reduces network overhead and improves query performance by 8% in scenarios involving shuffles.

  • Optimizes the Query Optimizer (QO), increasing the processing speed of SQL statements in the planning phase by 40%:

    • Optimizes the memory allocation mechanism and join algorithm to improve query performance in multi-join scenarios.

    • Optimizes the behavior of the DATE_PART function to improve query efficiency on time-related fields, such as year.

    • Optimizes the comparison behavior for DATE and TIMESTAMP type fields to improve query efficiency on time-based columns.

    • Optimizes operations in complex functions with filters. This optimization reorders multiple filters to reduce the amount of computed data, improving query efficiency.

• Introduces Serverless Computing to run long-running jobs (data import, ETL) in a shared resource pool, reducing task contention and improving instance stability. Available in select regions only.

• Instances with Virtual Warehouses now support automatic routing for write operations. The system automatically routes these operations to the appropriate Virtual Warehouse. This eliminates the need for manual specification and simplifies usage.

• Auto Partition for dynamic partitions now supports customizing partition creation and deletion times, as well as the migration time for cold storage tables, which improves the feature's usability.

• Adds support for SQL hint syntax. You can use hints to change a SQL statement's execution plan, which allows for fine-grained performance optimization.

• Optimizes the hg_stat_activity function to provide more accurate CPU and memory metrics. It also supports viewing the progress of data imports from MaxCompute to Hologres, which significantly improves the observability of active queries.

• Adds new path analysis functions. You can analyze traffic and time spent at each node in a path to inform product operation strategies and design optimizations.

• Enhances function capabilities:

  • Supports the try_cast type conversion function, which automatically converts invalid data to NULL instead of raising an error, which simplifies the handling of exceptional data.

  • Adds new date and time functions: dateadd, datediff, and last_day.

  • Supports running more general-purpose aggregate functions on the Hologres Query Engine (HQE) to improve query performance.

• Serverless Computing usage guide

• Authorize data access for Virtual Warehouses

• Manage dynamic partitions

• HINT

• Manage queries

• Path analysis functions

• Type conversion functions

• Function release history

O&M and stability improvements

• Adds support for SQL fingerprints and reports them in slow query logs. Clustering and analyzing SQL fingerprints improves issue diagnosis and anomaly monitoring capabilities.

• Exposes monitoring metrics for QE, FixedQE, and binlogs to improve observability and maintainability.

• HoloWeb now supports Query Insight, which allows you to retrieve query execution information, table metadata, and lock troubleshooting details with a single click to further improve troubleshooting efficiency.

• Adds support for cross-AZ disaster recovery. (This feature is available only in select regions.)

• When a Virtual Warehouse is rebalanced after scaling out, the system automatically retries SELECT queries. This reduces business impact and improves the stability of online services.

• Optimizes engine error codes and error messages to improve the efficiency of slow query log analysis.

  • Optimizes the DDL duration calculation logic to improve the accuracy of collected execution times for DDL operations.

  • Supports recording EXPLAIN ANALYZE results in slow query logs, providing detailed runtime data for each operator.

• Optimizes the underlying technology for version upgrades. This optimization uses the new Physical Restore technology to significantly reduce upgrade time for instances with large amounts of metadata, which minimizes the business impact.

• Upgrades FE node table locks to short-term locks, resolving stalled DDL operations and metadata inconsistency errors for improved FE stability.

• Upgrades OpenAPI capabilities with new APIs for Virtual Warehouses, data lake acceleration, and resource groups to enhance instance operations and management.

• View and analyze slow query logs

• Query Insight

• Manage Virtual Warehouses

• Common issues with Hologres SQL statements

• API overview

Ecosystem extensions

• The Auto Load feature for external tables now supports the three-layer model of MaxCompute. You can use the hg_experimental_auto_load_foreign_schema_mapping parameter to specify schema mappings. This feature also supports schema evolution for MaxCompute external tables, including adding or deleting columns, and modifying column names or order.

• The Auto Load feature for external tables now supports automatic loading via DLF metadata to accelerate queries on tables stored in OSS.

• Upgrades the data lake architecture. External tables in ORC and Parquet formats now support multi-level caching using built-in high-speed disks and memory, and also support predicate pushdown. This significantly improves read performance.

• Access MaxCompute external tables through a service-linked role, simplifying permission configuration. Authorize with a single click in the Hologres console.

• HoloWeb now allows you to visually browse data in OSS and tables within schemas specified by the MaxCompute three-layer model.

• Auto Load for external tables

• Hologres service-linked role

• Create an OSS external table

• Create a MaxCompute external table

Feature

Description

Related documentation

Engine enhancements

• Automatically optimizes runtime performance for single and multiple COUNT DISTINCT expressions, significantly improving query efficiency in COUNT DISTINCT scenarios.

• Row Group Filter for column-oriented tables records min/max values per row group, enabling the optimizer to prune entire groups without reading data, reducing I/O and improving query performance.

• Optimizes the Runtime Filter capability to support multi-column join scenarios, significantly improving join efficiency.

• You can now manually trigger a full compaction to merge small files and improve query efficiency.

• Adds range-based funnel analysis functions to analyze and compare user activity conversion.

• Adds the Bit-Sliced Index (BSI) extension to improve performance and usability for high-cardinality tag scenarios and for joint queries on user profile tags and behavior tags.

• The clustering key now supports descending order. You can use the clustering key in sorting scenarios to improve query performance.

• Optimizes the caching mechanism for infrequent access storage to improve its query performance.

• Adds CREATE TABLE WITH and ALTER TABLE SET syntax to replace the original set_table_property syntax, simplifying the process of setting table properties.

• Optimizes writes to tables without a primary key. Batch writes to such tables now use row-level locks instead of table-level locks, allowing them to run concurrently with Fixed Plan.

• Proxima vector computing now supports creating a table and importing vector data before creating a vector index. This shortens index creation time and simplifies using vector computing.

• Function enhancements:

  • Several array functions can now run on HQE to improve function performance.

  • Adds the KeyValue function to split strings.

  • Adds the IF function to simplify type checking scenarios and reduce MySQL migration costs.

• Count distinct optimization

• Runtime Filter

• Compaction (Beta)

• Funnel functions

• BSI functions

• Profile analysis - BSI optimization solution

• Clustering key

• Data tiered storage

• CREATE TABLE

• Locks and lock troubleshooting

• Proxima vector computing

• Function release notes

• KeyValue function

• IF

O&M and stability improvements

• Enhances slow query diagnostics for more efficient analysis.

  • You can now record EXPLAIN ANALYZE results in slow query logs, providing detailed runtime data for each operator.

  • Enhances Fixed Plan diagnostics. In write scenarios, affected_rows data is reported to the metadata warehouse. In query scenarios, result_rows and result_bytes data are reported to the metadata warehouse.

• Adds the hg_relation_size function to query the detailed storage usage of a table.

• Adds compatibility with native PostgreSQL behavior and support for load balancing. This enables load balancing and automatic failover in a primary/secondary architecture, improving service availability.

• Enhances the OpenAPI with new APIs for creating, renewing, scaling, and releasing instances, improving instance O&M and management.

• View and analyze slow query logs

• Accelerate SQL execution with fixed plans

• Table storage functions

• JDBC-based load balancing

• API overview

Ecosystem expansion

Data lake acceleration now supports the Paimon storage format.

Accelerate access to data in OSS data lakes by using DLF

Feature

Description

Related documentation

Virtual Warehouse instances

These instances support multiple types of workload isolation within a single instance, including read/write, write/write, and read/read isolation. They also support online elastic scaling and flexible traffic switching. This architecture enables flexible data sharing and resource isolation across teams, satisfying the requirements for enterprise-level critical services.

• Virtual Warehouse instance architecture

Feature

Description

Related documentation

Engine enhancements

• Introduces the Runtime Filter to optimize filtering during join operations. This feature reduces the amount of data scanned and lowers I/O overhead, improving performance by over 20% in typical multi-table join scenarios.

• The query engine adds a Lazy Create Fragment Instance mechanism. In preview scenarios that use a LIMIT N clause on large tables, this mechanism reduces overhead and significantly improves query performance.

• Fully optimizes the display format for execution plans (EXPLAIN and EXPLAIN ANALYZE) for better readability, simplifying SQL performance tuning.

• Enhances distributed transaction support, now including transactions with multiple DML statements.

• Supports DROP COLUMN.

• Supports CREATE TABLE AS syntax to simplify schema iteration and optimization scenarios.

• Supports streaming COPY, which improves write throughput and eliminates the need for batching.

• Supports setting a bitmap index in JSONB columnar storage to accelerate queries with equality filters.

• Supports setting the DATE type as a primary key and as a partition key for a partitioned table. Partition pruning is optimized and is supported even when the length of the IN array for the partition field exceeds the threshold (default: 100).

• More internal engine optimizations:

  • The storage engine optimizes the Tablet Lazy Open mechanism, which is supported for both primary and secondary instances. Tables that have not been accessed for more than 24 hours are automatically closed to free up memory. When the amount of data in open tables exceeds a threshold, a least recently used (LRU) policy is used to dynamically select a corresponding tablet to close. This reduces the resident memory overhead in scenarios with many tables.

  • The storage engine optimizes the schema storage management mechanism. A Meta Tablet now manages all schemas, which reduces resident memory overhead in environments with many tables and shards.

  • The storage engine optimizes its rapid recovery capability. It can now start in repair mode for a quick recovery if some tables have issues with conventional recovery. Metadata management now supports logical recovery by default, which effectively shortens recovery time in scenarios with many partitions. For scenarios with tens of thousands of partitions, recovery is more than five times faster.

• Function enhancements:

  • More functions can now run on HQE to improve function performance.

    • The Table Function support framework is re-architected to support running generate_series (INT, BIGINT, and NUMERIC) on HQE.

    • The PQE function support framework is re-architected to support running left, right, text::timestamp, and timestamp::text on HQE.

  • Adds array functions, including array_max, array_min, array_contains, array_except, array_distinct, and array_union.

  • Adds aggregate functions max_by and min_by to simplify ordered operations in window scenarios.

• Runtime Filter

• EXPLAIN and EXPLAIN ANALYZE

• SQL transaction capabilities

• Delete column (Beta)

• CREATE TABLE AS

• COPY

• Columnar JSONB

• CREATE PARTITION TABLE

• Function release notes

• ARRAY functions

• MAX_BY

• MIN_BY

O&M and stability improvements

• Introduces hg_stat_activity on top of pg_stat_activity. It is compatible with the original view but provides much richer runtime diagnostic information, such as execution stage, execution engine type, resource usage, and runtime lock information.

• Implements shard-level replication. This supports high availability and load balancing for throughput scaling within a single instance. It can tolerate partial machine failures and uneven hotspots.

• Auto Analyze is redesigned with a distributed architecture, now supporting external tables, lakehouse acceleration clusters, and incremental analysis for partitioned tables. Resolves analysis failures on very large or wide tables, leading to more stable execution plans and lower overhead.

• Optimizes storage encryption configuration to support flexible, per-table encryption settings.

• Improves the data lineage mechanism. It now supports cross-engine lineage analysis between MaxCompute and Hologres in DataWorks and can parse the lineage of expressions such as CTEs.

• Adds management OpenAPI operations for flexible control over the instance lifecycle.

• Query the hg_stat_activity view

• Single-instance shard-level multi-replica

• Table-level storage encryption settings

• Data lineage

• API overview

Ecosystem expansion

• Upgrades the query acceleration engine for MaxCompute foreign tables for improved compatibility and stability.

• In lakehouse acceleration scenarios with integrated DLF metadata management, you can now use DLF data catalogs (Multi-Catalog) for metadata isolation. This simplifies metadata isolation among testing, development, and cross-departmental clusters.

• In lakehouse acceleration scenarios, this release adds support for accelerating data stored on OSS-HDFS (also known as JindoFS). This better serves data lake computing scenarios in the Hadoop big data ecosystem and AI fields.

• Adds ClickHouse-compatible functions to simplify data and job migration.

• Accelerate access to data in OSS data lakes by using DLF

• Accelerate access to data in OSS-HDFS data lakes by using DLF

• Import data from ClickHouse

Feature

Description

References

Engine enhancements

• Adds support for real-time materialized views to improve query performance in real-time aggregation scenarios (Beta).

• Adds columnar storage for JSONB, which significantly improves statistical query efficiency and increases the data compression ratio.

• Adds dynamic partition management for partitioned tables, including automatic creation and deletion of partition sub-tables.

• Adds the UNIQ precise deduplication function, which significantly improves deduplication efficiency, optimizes multi-COUNT DISTINCT scenarios, and reduces memory usage.

• Engine optimizations:

  • Adds support for writing directly to the parent table of a partitioned table by using INSERT statements that conform to Fixed Plan.

  • Adds support for filtering in aggregation expressions, including string_agg() and array_agg().

  • Adds support for the RowType data type and related functions such as row() and row_to_json().

  • Adds support for modifying a table's schema.

  • Introduces the CTE Reuse operator to improve the performance of with expressions.

• Adds support for reading the MaxCompute three-layer model (project.schema.table).

• Improves MaxCompute integration: you can now read from and write to transactional tables, read tables after schema evolution (column deletion, reordering, or type changes), and write back ARRAY and DATE data.

• Manage materialized views with SQL

• JSON and JSONB types

• CREATE PARTITION TABLE

• Optimize query performance

• INSERT

• Accelerate SQL execution with fixed plans

• String functions

• JSON functions

• ALTER TABLE

• CREATE FOREIGN TABLE

• IMPORT FOREIGN SCHEMA

• Accelerate queries on MaxCompute data using a foreign table

• Export to MaxCompute

O&M and stability improvements

• Introduces self-service configuration of a shared storage secondary instance to enhance elasticity and high availability.

• Added the table_info metadata warehouse table to enhance data governance capabilities.

• Reduces the memory footprint of metadata by optimizing memory usage.

• Adds support for automatic periodic backup and manual backup, enabling data recovery from accidental operations.

• Multi-instance high-availability deployment with read/write splitting (shared storage)

• Viewing and analysis of table statistics information

• Local backup and recovery

• Recover

Ecosystem expansion

• Adds production-grade support for the PostGIS extension.

• Adds the Oracle extension package, which provides many Oracle-compatible functions.

• Using DLF, you can now read Hudi and Delta format foreign tables, and write data in CSV, Parquet, SequenceFile, and ORC formats to OSS foreign tables.

• Improves BI compatibility, achieving a pass rate of 99%+ in the Tableau compatibility test (TDVT).

• PostGIS spatial functions

• Oracle-compatible functions

• Based on DLF to accelerate access to OSS data lakes

Feature

Description

Related documentation

Operations and maintenance improvements

• Adds resource group isolation (Beta). You can create resource groups to isolate workloads at the thread level for different users within an instance. This feature better supports multi-user and multi-scenario use cases.

• Supports online hot upgrades for Hologres instances. Read (query) operations are unaffected during an upgrade. To use this feature, apply for access by joining the Hologres community group.

• Isolate computing resources in a single instance (Beta)

• For information about how to join the group, see Obtain online support.

Engine enhancements

• Supports row-column hybrid storage, which allows a single data copy to serve both point query and OLAP workloads.

• Supports real-time consumption of Hologres binary logging via JDBC (Beta).

• Supports on-demand enabling and dynamic configuration of binary logging.

• Supports renaming columns.

• Adds JSONB indexes (Beta) to accelerate queries on JSON data.

• Optimizes in-memory metadata management by adding caching and compression for more efficient memory usage.

• CREATE TABLE

• Consume Hologres binary logs by using JDBC

• Subscribe to Hologres binary logs

• ALTER TABLE

• JSON and JSONB data types

Foreign table enhancements

• Supports reading data in CSV, Parquet, SequenceFile, and ORC formats from Object Storage Service (OSS) by using Data Lake Formation (DLF).

• Supports cross-database queries and federated queries across multiple Hologres instances.

• Accelerate access to data in OSS data lakes based on DLF

• Cross-database queries (Beta)

Security enhancements

• Supports data storage encryption for internal Hologres tables (Beta) to enhance data access security.

• Supports reading encrypted data from MaxCompute (Beta) to improve compatibility with the MaxCompute ecosystem.

• Data storage encryption

• Query encrypted data in MaxCompute

Feature

Description

Related documentation

Engine enhancements

• Supports automatic collection of table statistics. The system automatically samples table statistics during data writes and updates to generate better query plans, which eliminates the need to manually execute ANALYZE TABLE.

• Millisecond-level high reliability for point queries (Beta) with shard-level multi-replica configurations, millisecond failover, and query retries for serving scenarios.

• Adds a RoaringBitmap extension, which provides native support for the Bitmap data type and related functions.

• Adds the bit_construct and bit_match functions. These functions are optimized for scenarios like user segmentation and attribution analysis, providing more efficient filtering for aggregate conditions based on user IDs.

• Adds the range_retention_count and range_retention_sum functions to optimize multi-day range queries for retention analysis.

• Introduces a resharding tool with a built-in function that allows you to change the shard count without recreating tables, simplifying the tuning process.

• Optimizes the default compression format for column store to AliORC, improving the storage compression ratio by 30% to 50%.

• ANALYZE and AUTO ANALYZE

• High-throughput reads from multiple replicas of a shard in a single instance (Beta)

• RoaringBitmap functions

• Functions for user segmentation

• Funnel functions

• Manage Table Groups and shard counts

• Change the data storage format of a column-oriented table

Foreign table query enhancements

• Improves query performance for MaxCompute foreign tables (Beta) by 30% to 100% compared to the previous version, thanks to a new acceleration engine.

• Adds integration with Data Lake Formation (DLF) (Beta) to read data from Object Storage Service (OSS).

• Optimize query performance for MaxCompute foreign tables

• Accelerate access to data in OSS data lakes based on DLF

Performance optimizations

• Improves point query performance, increasing total throughput by 100% for row store and 30% for column store.

• Optimizes update operations, improving the performance of UPDATE and DELETE operations by 30%.

• Optimizes the query plan cache to reduce optimizer overhead.

/

Enterprise-grade O&M and security optimizations

• Introduces slow query identification. The built-in query status history allows you to view all queries from the last month to quickly identify slow or failed ones.

View and analyze slow query logs

Feature

Description

Related documentation

Engine enhancements

• Adds support for a wider range of data types.

  • JSON and JSONB types.

  • Time types: interval, timetz, time.

  • Network type: inet.

  • Monetary type: money.

  • PostgreSQL system types: name, uuid, oid.

  • Other types: bytea, bit, varbit.

• Adds support for more function types, including PostgreSQL-compatible functions and Hologres extension functions.

  • Array functions: Adds array_length and array_positions.

  • Functions for viewing table and database storage size: pg_relation_size and pg_database_size.

• Supports exporting Hologres data to MaxCompute for archiving by using Hologres SQL commands.

• Supports subscribing to Hologres binary logging (Beta).

• Supports dynamic modification of table bitmap indexes and dictionary encoding. Supports automatic creation of dictionary encoding based on data characteristics.

• Releases the Holo Client library, designed for large-scale data synchronization and high-QPS point query scenarios. It uses automatic batching to improve throughput.

• Optimizes the JDBC write pipeline and query optimizer to significantly improve the engine's write efficiency.

• Improves connectivity with the BI ecosystem by adding support for more BI tools, such as Tableau Server and Superset, to meet various business analysis needs.

• JSON and JSONB data types

• Data types

• ARRAY functions

• Table storage functions

• Export data to MaxCompute

• Subscribe to Hologres binary logs

• ALTER TABLE

• Holo Client

Security enhancements

• Supports logging in to Hologres with an STS account by assuming a role, providing a more secure and flexible login option than using Alibaba Cloud accounts.

RAM role-based authorization

Core feature enhancements

Description

Related documentation

Engine enhancements

• Use the CREATE VIEW statement to create a view. A view can be based on a single table, multiple tables (including internal and foreign tables), or other views.

• Hologres now supports the following data types: SERIAL, DATE, TIMESTAMP, VARCHAR(n), and CHAR(n). In addition, MaxCompute foreign tables now support Array data type mapping.

• Hologres now supports INSERT ON CONFLICT. This feature allows you to update or skip duplicate rows based on the primary key when you insert data.

• Hologres now supports the TRUNCATE statement.

• Hologres now includes the built-in Proxima engine to support vector search on massive datasets. This feature is currently in Beta.

• VIEW

• Data types

• INSERT ON CONFLICT (UPSERT)

• Proxima vector search

Security enhancements

• Hologres now supports data masking. You can configure various masking policies to mask sensitive information, such as phone numbers, addresses, and ID card numbers.

• Hologres now integrates with CloudMonitor to support custom metric monitoring and one-click alerts.

• Data masking

• CloudMonitor

MaxCompute foreign table query constraints and limitations

• When you query MaxCompute partitioned tables, you can scan up to 512 partitions, an increase from 50 in earlier versions.

• You can now scan up to 200 GB of underlying data per query, regardless of the number of foreign tables and columns (previously 100 GB).

Constraints and limitations