Kernel release notes

Category

Description

Engine

Fixes

• Fixed a BE crash in BinaryColumnBase::append when _bytes is empty.

• Fixed a bad_weak_ptr crash in the SinkBuffer brpc closure callback (QueryContext premature destruction scenario).

• Fixed an issue where ALTER TABLE SET storage_volume did not write EditLog, causing Follower FE out of sync as well binding loss after Leader restart.

• Fixed an SLF4J LinkageError during JNI Connector loading in ChildFirstClassLoader.

• Fixed a deadlock caused by bulk publish with a partition version gap.

• Fixed an issue where PipelineTimerTask got stuck in waitUtilFinished.

• Fixed a logic error in EventScheduler's check_short_circuit.

Storage-compute separation

Fixes

• Fixed uneven sub-bucket routing in Composite SV when newly allocated partition IDs have the same lower-bit parity.

• Fixed an issue where schema-change/rollup shadow tablets in Composite SV were rerouted to wrong partitions after shard path changes.

• Fixed silent fallback override when FileStore lookup by ID fails in Composite SV.

Lakehouse

Enhancements

• Added support for MaxCompute three-level namespaces (new catalog property odps.enable.namespace.schema; default: false).

• Upgraded Iceberg to version 1.10.0.

• Upgraded the DLF iceberg plugin.

Fixes

• Fixed an issue where user_agent_extended was not propagated from FE to BE during Paimon queries.

• Fixed a resource leak caused by FlussSplitScanner not closing FlussConnection.

Category

Description

Engine

New features

• [Beta] Added native SQL AI built-in function support.

• [Beta] Added support for submitting Spark Load jobs to EMR Serverless Spark.

Enhancements

• Added a timeout mechanism for the HdfsFsManager file system lock to prevent cascading deadlocks.

• Exposed monitoring metrics for the Scan and Connector Scan thread pools.

• Added Slow Lock histogram monitoring metrics for critical locks.

• Added FE transaction latency monitoring metrics.

• Added a dedicated thread pool for cloud-native Tablet Metadata fetching to reduce contention on the main thread pool.

Fixes

• Fixed multiple BE crash issues, including BinaryColumnBase append exceptions and SLF4J Linkage Errors during JNI Connector loading.

• Fixed a self-deadlock in RoutineLoadJob during the transaction publish phase caused by an incorrect lock acquisition order, which could result in import state loss after an FE restart.

• Fixed an issue where transaction dependencies were lost when deleting a node from the transaction dependency graph.

• Fixed an issue where the default value of partition_refresh_number did not take effect.

• Fixed a duplicate definition for Thrift Field Ordinal 117.

• Fixed an issue where the ErrorURL was missing in the "too many filtered rows" import failure scenario.

Storage-compute separation

New features

• [Beta] Lake Optimizer: an automated background maintenance and optimization engine for storage-compute separated tables. Disabled by default. To enable it, set enable_lake_optimizer to true.

• [Beta] Composite Storage Volume (multi-bucket per table): allows a single table's Storage Volume to span multiple OSS buckets, intended for scenarios where a single table's data volume exceeds the performance limits of a single bucket.

Enhancements

• The Compaction Service now supports table allowlists for fine-grained control over background compaction scope.

• Introduced CacheOptions to support fine-grained configuration of Tablet Metadata caching.

• Lake Optimizer:

  • Shortened the Daemon check interval to improve optimization task responsiveness.

  • Time Travel queries on Lake Optimizer tables now throw an UnsupportedException.

Important changes

Lakehouse

New features

• [Beta] Paimon Native Writer type expansion: added support for DATE, VARBINARY, and TIMESTAMP_WITH_LOCAL_TIME_ZONE types.

Enhancements

• Paimon Writer now supports Bucket Partition Shuffle to improve bucketed write performance.

• Paimon Writer now supports writing NULL partitions.

• HMS external tables now support partition pruning with IN predicates.

• JniScanner error messages now include Paimon troubleshooting hints.

• Improved Fluss table query capabilities: added support for Fluss table partition pruning, Native Reader for Fluss Lake Splits, and access to Fluss from ECS environments.

Fixes

• Fixed an isolation issue when multiple users shared a Paimon catalog.

• Fixed incorrect Paimon Count optimization results.

• Fixed an issue where Paimon data was written to the wrong partition.

• Fixed an issue where FormatDataSplit was not supported when paimon_force_native_reader was enabled.

• Fixed an issue where CREATE PAIMON DATABASE IF NOT EXISTS failed.

• Fixed various NPE and Plan exceptions when querying Paimon tables.

• Fixed a ClassCastException in FlussPredicateConverter during Timestamp predicate pushdown.

• Fixed Fluss Metadata cache key collisions and NPE issues.

• Fixed a missing OSS Token in the Iceberg Table Sink.

• Fixed a ClassNotFoundException when reading an Iceberg Metadata Table.

Behavior changes

• Upgraded paimon-cpp to 0.10 and Paimon Java SDK to 1-ali-25.3. Downstream systems using Paimon Native Reader/Writer should verify compatibility after this upgrade.

• Composite Storage Volume: new cross-Bucket Storage Volume configuration semantics have been added. Review StarRocks CREATE STORAGE VOLUME to confirm compatibility with existing Storage Volume configurations before use.

• Tablet Metadata caching policy update: the introduction of CacheOptions changes the cache interface semantics. If you have custom cache configurations, update them to use the new interface.

Category

Description

Engine

Enhancements

• Optimized partition range mapping performance by pre-calculating lowerPoints/upperPoints outside the loop. This reduces the time complexity of partition intersection calculation and many-to-many detection from O(M*N) to O(M*logN+N).

Fixes

• Fixed a Null Pointer Exception (NPE) that occurred when using Broker Load to import data in scenarios involving a UDF.

• Fixed an issue where backslash escape sequences were handled incorrectly in LIKE pattern matching.

• Fixed a crash that occurred when the array_map function processed an array of null literals.

• Fixed an issue where the active transaction ID of the rollup handler was not included in the computeMinActiveTxnId calculation.

• Fixed an issue where transaction dependencies were lost when deleting a node from the transaction dependency graph.

• Fixed a signature conflict in UpdateTabletSchemaTask when multiple alter jobs were executed concurrently.

• Fixed an issue where data was lost from the hash table during the set_finishing stage of an aggregation due to an overflow.

• Fixed an issue where BaseScalarOperatorShuttle was missing the visitDictionaryGetOperator implementation.

• Fixed a race condition in finishTransactionBatch caused by the edit log write occurring after afterVisible, which could lead to premature cleanup of transaction logs.

• Fixed a self-deadlock in RoutineLoadJob.toThrift() during the transaction publish phase, which was caused by an incorrect lock acquisition order. This issue could prevent the VISIBLE state from being persisted, leading to NoSuchKey errors after an FE restart.

Storage-compute separation

Enhancements

• The Compaction Service now supports table allowlists. Table names are pre-resolved to table IDs during configuration parsing.

• Cross-cluster replication for storage-compute separation now supports Fast Cancel for file copy operations, reducing the waiting time for canceled tasks.

• Cross-cluster replication for storage-compute separation now supports parallel tablet file copying to improve replication throughput.

Fixes

• Fixed an issue where transient object storage exceptions could cause silent file truncation. In cross-cluster replication for storage-compute separation, copy operations for non-segment files (such as .sst, .delvec, .del, and .cols) now include file size integrity checks and an automatic retry mechanism.

Lakehouse

Fixes

• Fixed cache key collisions and Null Pointer Exceptions (NPEs) in Fluss metadata queries.

• Fixed a ClassNotFoundException when reading an Iceberg metadata table.

Category

Description

Engine

Enhancements

Routine load jobs now output logs upon completion, providing more comprehensive diagnostic information.

Fixes

• Fixed an issue where index properties were lost during materialized view creation.

• Fixed an issue where the auto_refresh_partitions_limit parameter did not take effect when refreshing a materialized view with hourly partitions of the DATETIME type.

• Fixed an issue where error messages were empty during materialized view refresh.

• Fixed an issue where Stream Load used an expired timestamp when importing data into a DEFAULT CURRENT_TIMESTAMP column.

• Fixed a BE crash in clusters not using storage-compute separation, which was caused by OlapMetaReader not setting the count flag.

Storage-compute separation

Enhancements

• Reduced the overhead of collecting tablet statistics for Lake Primary Key (PK) tables in storage-compute separation mode. This also fixes an issue where show tablet/partitions returned 0 for data size and row count.

• Added an API to create physical partitions for random distribution tables. The migration tool now supports migrating random bucket tables that contain multiple physical partitions to the current StarRocks cluster version.

• In storage-compute separation mode, Cluster Linking now supports data migration from versions earlier than 3.2.4.

Fixes

• Fixed an issue where the lake compaction transaction map was not cleared when a transaction finished. This resolves a bug where compaction scheduling did not work for some tables after an FE restart.

• Fixed an issue where deleting a storage-compute separation table did not clear shard group metadata in StarManager, which could cause the FE to run out of memory (OOM).

Lakehouse

Important change

To ensure compatibility with Paimon Native Writer for writes of the TIMESTAMP_WITH_LOCAL_TIME_ZONE type, the current version extends the Arrow type conversion logic to map this type from StarRocks to the Arrow TIMESTAMP type.

This change may cause exceptions when starrocks-spark-connector and starrocks-flink-connector read DATETIME type fields from StarRocks. If you encounter related errors after the upgrade, add the enable_native_arrow_new_type dynamic configuration parameter to the compute group and set its value to false to restore compatibility.

Enhancements

• Upgraded paimon-cpp to version 0.9.

• The Paimon native writer now supports writing DATE values.

• The Paimon native writer now supports writing VARBINARY values.

• The Paimon native writer now supports writing TIMESTAMP_WITH_LOCAL_TIME_ZONE values.

• Upgraded Fluss to version 0.9-ali-0.1.

• Added support for querying Paimon Lance tables.

• Added support for querying Paimon views.

• Improved the observability of the Paimon cache by adding cache metrics.

Fixes

• Fixed an error that occurred when writing the TINYINT type using the Paimon JNI writer.

• Fixed an issue where the token did not have sufficient permissions when writing to Paimon tables in DLF 2.5.

• Fixed incorrect partition values when querying Paimon tables.

• Fixed a Null Pointer Exception (NPE) when querying Paimon tables, caused by a missing mergedRowCountAvailable check.

• MaxCompute Catalog now supports creating sessions asynchronously and splitting data dynamically.

Category

Description

Storage-compute separation

New features

• Introduced the Compaction Service feature. When scheduling compaction in a storage-compute separation architecture, you can now use a dedicated compute group to handle compaction tasks. This isolates the compaction environment while ensuring a high cache hit rate for business queries through Cache Peering.

• Introduced the server-side Merge Commit feature for high-concurrency Stream Load, significantly increasing real-time data ingestion throughput and reducing compaction resource consumption.

• In high-concurrency, small-batch ingestion scenarios, the Merge Commit feature combines multiple concurrent Stream Load requests into a single transaction. This batch submission boosts real-time ingestion throughput, reduces data versions from frequent commits, and minimizes the resource and I/O overhead from compaction and small files.

• For large-scale data import scenarios, the new Bulk Load Optimization feature merges data blocks during the ingestion phase. This reduces the creation of small files, improves post-ingestion query performance, and reduces subsequent compaction resource usage.

• Introduced the Cluster Snapshot feature for storage-compute separation, which supports unified backup of cluster-level metadata to OSS, enabling enterprise-level disaster recovery.

Enhancements

• Enhanced Cluster Linking to support online migration from self-hosted storage-compute separation clusters to Serverless StarRocks storage-compute separation clusters.

• Enhanced the show proc '/transactions' interface to display records for multiple transaction types simultaneously.

• You can now perform a manual partition refresh for a materialized view on the warehouse associated with the current session.

Fixes

• Fixed an exception caused by unclean metadata when migrating a Primary Key (PK) table with an index in a storage-compute separation cluster.

• Fixed an issue where tablet distribution could not be viewed when using a non-default warehouse.

Lakehouse

Enhancements

Improved query performance for tables in Paimon format.

Fixes

• Fixed incorrect results returned when querying Paimon tables that use a Timestamp type as a filter condition or partition field.

• Fixed an error that occurred when querying the Paimon binlog and auditlog system tables.

• Fixed an issue where queries on Fluss tables returned empty results in specific scenarios.

Category

Description

Engine

New features

• Unified Catalog now supports integration with DLF Catalog.

Fixes

• Fixed incorrect predicate rewriting when referencing columns on the constant side of outer joins.

Storage-compute separation

Fixes

• Fixed an issue where partition filter conditions were lost for dynamic bucket tables in storage-compute separation mode.

Lakehouse

New features

• MaxCompute Catalog now supports MaxCompute three-level namespaces.

Enhancements

• Upgraded the DLF Iceberg Plugin.

• Upgraded Iceberg to version 1.10.0.

Fixes

• Fixed a query error caused by dependency conflicts between Hive and Paimon.

• Fixed an issue where some parameters were not passed correctly when querying Paimon data, causing query failures.

• Fixed a resource leak caused by an unclosed FlussConnection in FlussSplitScanner.

Category

Description

Engine

New features

• [Beta] Added native SQL AI built-in function support (adapted for the 3.3 releases).

• [Beta] Added support for submitting Spark Load jobs to EMR Serverless Spark.

• Added the max_pt function.

Enhancements

• Added support for Filter pushdown on CTE Consumers.

• Added support for downloading UDF jars from a FileSystem.

• CREATE TEXT FILE TABLE now supports specifying a delimiter.

• Added a timeout mechanism for the HdfsFsManager file system lock to prevent cascading deadlocks.

• Exposed monitoring metrics for the Scan and Connector Scan thread pools.

Fixes

• Fixed a BE crash caused by memtable finalize failure.

• Fixed Slow DB Lock caused by redundant O(N) calculations in the MV partition intersection check.

• Fixed data loss from the hash table during the set_finishing stage of an aggregation overflow.

• Fixed rssid rowid truncation in inline PK index compaction.

• Fixed an issue where transaction dependencies were lost when deleting a node from the transaction dependency graph.

• Fixed an issue where the default value of partition_refresh_number did not take effect.

• Fixed an issue where the ErrorURL was missing in the "too many filtered rows" import failure scenario.

• Fixed a crash when the array_map function processed an array of null literals.

• Fixed incorrect handling of backslash escape sequences in LIKE pattern matching.

• Fixed an NPE when creating a UDF with CreateFunctionStmt.

• Fixed an NPE when using Broker Load with a UDF.

Storage-compute separation

Enhancements

• Lake Optimizer improvements (introduced in 2.0.0, further refined in this release): shortened the Daemon check interval to improve optimization task responsiveness. Time Travel queries on Lake Optimizer tables now throw an UnsupportedException.

• The Compaction Service now supports table allowlists for fine-grained control over background compaction scope.

Lakehouse

New features

• [Beta] DLF now supports row-level filtering (Row Filter).

Enhancements

• Paimon Writer now supports Bucket Partition Shuffle to improve bucketed write performance.

• Paimon Writer now supports writing NULL partitions.

• HMS external tables now support partition pruning with IN predicates.

• JniScanner error messages now include Paimon troubleshooting hints.

• Improved Fluss table query capabilities: added support for Fluss table partition pruning, Native Reader for Fluss Lake Splits, and access to Fluss from ECS environments.

Fixes

• Fixed an isolation issue when multiple users shared a Paimon catalog.

• Fixed incorrect Paimon Count optimization results.

• Fixed an issue where Paimon data was written to the wrong partition or with incorrect partition value ordering.

• Fixed an issue where CREATE PAIMON DATABASE IF NOT EXISTS failed.

• Fixed an issue where FormatDataSplit was not supported when paimon_force_native_reader was enabled.

• Fixed a ClassCastException in FlussPredicateConverter during Timestamp predicate pushdown.

• Fixed Fluss Metadata cache key collisions and NPE issues.

• Fixed a missing OSS Token in the Iceberg Table Sink.

Behavior changes

• Added the enable_native_arrow_new_type session variable: when set to true (default), DATE maps to Arrow Date32Type and DATETIME maps to Arrow TimestampType; when set to false, both DATE and DATETIME map to Arrow utf8() (backward-compatible). If you encounter type errors when reading DATETIME fields with starrocks-spark-connector or starrocks-flink-connector, set this variable to false as a workaround.

• Upgraded paimon-cpp to 0.10 and Paimon Java SDK to 1-ali-25.3. Downstream systems using Paimon Native Reader/Writer should verify compatibility after this upgrade.

Category

Description

Base engine

Enhancements

• Supports the SSL protocol.

• Supports transaction log output for Routine Load to provide more comprehensive diagnostic information.

Fixes

• Fixed an issue where profiles could not be logged after the materialized view (MV) profile feature was enabled.

• Fixed a query error in NEST CTE Reuse scenarios.

• Fixed a null pointer exception (NPE) that occurred when using Broker Load to import data with UDFs.

Storage-compute separation

New features

Introduces Compaction Service for the storage-compute separation architecture. This service uses a dedicated warehouse for compaction scheduling, which isolates the compaction execution environment while ensuring a high cache hit rate for business queries via Cache Peering.

Enhancements

• Performs a preliminary major compaction during the publish phase for primary key tables in storage-compute separation mode.

• Supports manual refresh of materialized view partitions on the warehouse associated with the current session.

Fixes

• Fixed a metadata cleanup issue that occurred when migrating a primary key table with an index in a storage-compute separation cluster.

• Fixed an out-of-memory (OOM) issue caused by partial updates in column mode during large-scale data imports.

• Fixed an FE OOM issue caused by frequently dropping tables.

• Fixed an issue where tablet distribution could not be viewed when using a non-default warehouse.

Lakehouse

Important changes

To ensure compatibility with Paimon Native Writer for the TIMESTAMP_WITH_LOCAL_TIME_ZONE type, this version extends the Arrow type conversion logic, mapping this StarRocks type to Arrow's TIMESTAMP type.

This change may cause starrocks-spark-connector and starrocks-flink-connector to fail when reading StarRocks DATETIME fields. If you encounter related errors after upgrading, restore the previous behavior by adding the dynamic parameter enable_native_arrow_new_type to your warehouse and setting it to false.

New features

• Introduces Lake Optimizer to significantly improve point query QPS on Paimon tables and enhance system stability for large-scale queries.

• Supports querying data from Paimon Lance tables.

• Supports querying Paimon views.

Enhancements

• The Paimon native writer now supports writing date and datetime data types.

• Improved the observability of Paimon queries.

Fixes

• Fixed an issue where queries on Fluss tables returned empty results.

• Fixed an issue where queries on Fluss tables failed due to expired snapshots.

• Fixed an issue where equality queries on Paimon timestamp fields returned incorrect results in some scenarios.

• Fixed an error that occurred when querying the Paimon binlog and audit_log system tables.

• Fixed an issue with insufficient token permissions when writing to Paimon tables in DLF 2.5.

• Fixed a character escaping issue for timestamp partition fields when querying Paimon tables.

• Fixed an issue where queries on Paimon tables with a LIMIT clause returned empty results in specific scenarios.

• Fixed a null pointer exception that occurred when querying older versions of Paimon tables.

Category

Description

Base engine

Enhancements

• Enhanced the fault tolerance of Routine Load. Routine Load can now automatically skip malformed records when importing JSON data and log the corresponding errors.

• Improved error messages for external table query failures, making troubleshooting more intuitive and efficient.

Fixes

Fixed a stability issue in conditional update scenarios, which could cause a BE node to crash during an update operation that contained only DELETE records.

Storage-compute separation

New features

Supports using Apache Ranger to control permissions for warehouse-level resources in storage-compute separation instances to meet enterprise-level security and compliance requirements.

Enhancements

Added a retry mechanism for storage read and write operations in storage-compute separation instances to enhance fault tolerance.

Fixes

Fixed an issue where too many bRPC connections were created in storage-compute separation instances under high concurrency.

Lakehouse

New features

• Supports Paimon Deletion Vector v2.

• Supports reading from and writing to Paimon Format Tables using Native Reader (Beta).

• Supports filtering virtual partitions in the Hive Metastore that are prefixed with hive_var to avoid metadata interference.

Category

Description

Storage-compute separation

Enhancements

Supports warehouse-level query latency metrics.

Lakehouse

Fixes

Fixed a query planning failure on a Paimon external table that has an empty string as a partition key.

Category

Description

Base engine

Enhancements

• Improved the fast-fail detection for long-running queries.

• Supports partial updates on Clucene inverted indexes.

Fixes

• Fixed an issue where queries did not terminate immediately after a BE crash caused by an abnormal exit of the CoordinatorMonitor thread.

• Fixed a TXN_NOT_EXISTS error caused by transaction loss due to redirection and retry mechanisms.

• Fixed a deadlock caused by MultiJoinBinder during the FE planning phase.

• Fixed an empty segment file marking issue caused by schema changes in partial update mode.

Storage-compute separation

Fixes

Fixed a 'shard not exist' error that occurred during long-running INSERT OVERWRITE operations.

Lakehouse

Fixes

• Fixed an error retrieving RAM user information when using a DLF Catalog.

• Fixed an issue where partition metadata became inconsistent between the leader and follower FEs after a materialized view partition was deleted.

Category

Description

Base engine

Enhancements

Improved the observability of long-running tasks with enhanced monitoring and diagnostic capabilities for real-time status tracking.

Fixes

Fixed an issue where SHOW BACKUP returned an incorrect error message.

Storage-compute separation

Fixes

Eliminated invalid requests sent to OSS when deleting the local cache, reducing unnecessary network overhead.

Lakehouse

Fixes

Fixed a query failure on Paimon tables partitioned by the Date type.

Category

Description

Base engine

New features

• Supports ACID capabilities (Beta).

• MaxCompute Catalog now supports predicate pushdown.

• Supports the first_day and last_day functions.

• Supports the bitmap_hash64 function.

Enhancements

• Optimized partition cleanup to resolve high FE memory usage from delayed recycling.

• Reduced FE planning time.

Fixes

• Fixed a bRPC failure in elastic scaling scenarios.

• Fixed a BE crash caused by an uninitialized delta writer load channel.

• Fixed a Flink task commit failure caused by backend stream loss.

• Fixed an incorrect isconst check in the array_contains function.

• Fixed an issue where json_extract was not supported during Trino migration.

• Fixed a CTE reuse issue.

• Fixed a refresh failure on materialized views created on external tables.

• Fixed an issue where the number of threads increased after JIT was enabled.

• Fixed a concurrency issue in the tablet manager.

• Fixed an issue where FE scheduling was stuck due to a profile collection failure.

• Fixed an issue where low-cardinality optimization errors were not thrown correctly.

• Fixed a BE crash during Parquet writing caused by the split function incorrectly handling null values.

• Fixed a materialized view refresh failure when a CASE WHEN expression contained a VARCHAR type.

• Fixed a concurrency lock issue caused by compaction in shared-nothing instances.

• Fixed an issue where the CBO could not correctly prune CTE-related expressions.

• Fixed issues related to the HTTP handler.

• Fixed a BE crash caused by a bRPC concurrency issue that affected the load channel and tablet sink.

Storage-compute separation

New features

• INSERT and UPDATE operations now support partial column updates in column mode.

• The SHOW ALTER TABLE command now supports viewing metadata modification tasks.

• Supports scheduling of compaction tasks based on the max compaction score.

Fixes

• Fixed an issue where transactions became stuck when compaction was enabled during migration.

• Fixed an issue where metadata was not cleared from the metadata cache and added support for dynamically updating the cache capacity.

• Fixed a BE crash caused by a lake persistent index initialization failure.

Lakehouse

New features

• Supports Paimon Native Writer (Beta), which significantly improves performance for data processing in lakehouse scenarios.

• Paimon queries now support complex types.

• Supports DLF Iceberg Catalog (Beta).

• Supports reading Fluss tables (Beta).

• Supports the Paimon TimeTravel feature.

• INSERT OVERWRITE now supports non-partitioned tables.

Enhancements

• Improved Paimon column statistics.

• Improved COUNT performance for Iceberg and Paimon tables.

Fixes

• Fixed a failure of the SHOW GRANT command for a user.

Full-text search

Fixes

• Fixed a null value issue with the match function.

Category

Description

Base engine

Enhancements

• Improved the performance of collecting partition statistics.

• Supports dynamic adjustment of the thread pool size for the FE query fragments deployer.

• Added configurable FE checkpoint parameters and logging for long-running checkpoint tasks.

Fixes

• Fixed an issue where the number of JDBC connections could exceed the jdbc_connection_pool_size limit.

• Fixed a partition creation failure caused by partition name conflicts during dynamic partition creation.

• Fixed an issue where scan data information was missing from import task statistics.

• Fixed an issue where inverted index queries incorrectly filtered null fields.

• Fixed an issue in full-text search where MATCH_REGEXP returned incorrect results when the pattern was an empty string.

• Fixed an issue where INSERT OVERWRITE operations failed on manually created partitions.

• Fixed an issue where an Alter Job was canceled due to a conflict with partition creation.

• Fixed an issue where the status of an ANALYZE job for statistics collection was abnormal.

• Fixed an issue where import transaction logs on the FE were incomplete.

Storage-compute separation

Fixes

• [Important] Fixed an issue introduced in community version 3.3.13 where a graceful exit of a BE or CN node made tablet metadata files unreadable.

• Fixed an issue where the max compaction score metric on the leader FE was not updated.

• Fixed an issue where transaction logs were lost during concurrent Batch Publish and schema change operations.

Lakehouse

Fixes

• Fixed an issue where the UUID format in Paimon tables was processed incorrectly.

• Fixed a user authentication issue for Paimon materialized views in DLF 2.5.

Category

Description

Base engine

Enhancements

• Added fe.load logs to improve data loading observability.

• Added the information_schema.loads field to the sink_bytes table.

• Improved Routine Load error logs by adding key information such as offsets to facilitate troubleshooting.

Fixes

• Fixed an issue where global_state_manager in ConnectContext could be null, which prevented query failures, import failures, or missing Profiles.

• Addressed missing column statistics to ensure their accuracy.

• Fixed the "Query version not found" error.

• Added the slow_lock_print_stack parameter to prevent process stuttering caused by thread stack printing in large-scale clusters.

• Fixed an FE out-of-memory (OOM) issue caused by an incorrect execution plan for a Union operator.

• Fixed a query failure that occurred after a column was renamed.

• Fixed an issue where the primary key index was lost due to a snapshot loading failure.

• Fixed a refresh issue caused by an abnormal change in materialized view properties.

• Added the FE parameter task_runs_max_history_number to control the number of historical TaskRuns saved in the information_schema.task_runs view, reducing memory overhead.

• Fixed an issue where data versions were lost during import.

• Fixed an issue where Task Run information was abnormal on follower FE nodes.

• Fixed an issue where Broker Load could not be resumed after an FE failover.

• Fixed an issue where a query returned null after a column was modified.

• Fixed a partition creation failure that occurred when writing to multiple tables in a single transaction.

• Fixed a query error caused by a CASE WHEN statement that returned an incorrect data type.

• Fixed a cache failure caused by duplicate cache keys in concurrent scenarios.

• Fixed a partial update failure on primary key tables.

• Fixed an issue with the lock mechanism for submitting tasks.

• Fixed a global dictionary issue for the Union operator in low-cardinality scenarios.

• Fixed an issue where empty partitions were incorrectly pruned.

• Fixed an error caused by the loss of null partitions during materialized view rewriting.

• Fixed an issue where the splitter could not prune scan columns.

• Fixed issues related to Left Anti Join.

Storage-compute separation

New features

• Full-text search is now supported in storage-compute separation instances.

• Added the distributed compaction feature to alleviate single-node compaction performance bottlenecks.

• Supports adaptive I/O mode for the cache to prevent performance degradation caused by cache disk I/O bottlenecks.

• Added warehouse-level query metrics for better query performance monitoring.

Enhancements

• Enabled Index Cache and Metadata Cache by default to further improve query performance.

• Optimized an issue where minimum version information was missing for storage-compute separation tables.

Fixes

• Fixed an issue where data was written incorrectly to the destination warehouse during an INSERT INTO EXTERNAL TABLE operation in storage-compute separation mode.

• Fixed a concurrent transaction failure that occurred after Batch Publish was enabled.

• Fixed an issue where the maximum number of compaction tasks could not be limited after a warehouse was specified for compaction.

• Fixed a preheating error in Cache Select after a schema change.

• Fixed a failure to modify the data distribution using an ALTER TABLE statement.

Lakehouse

New features
Paimon tables now support column-level data lineage analysis and feature improved statistics.

Enhancements
Improved Paimon query performance by optimizing metadata retrieval from DLF 2.5.

Fixes

• Fixed a memory leak issue with Iceberg equality delete files.

• Fixed a DLF 2.5 token expiration issue.

• Fixed an issue where materialized views could not be refreshed after the underlying external table was rebuilt.

• Fixed a refresh failure on materialized views of external tables.

• Fixed a failure to write data to Hive tables using INSERT OVERWRITE.

• Fixed a materialized view refresh failure caused by using the str2date function on external tables.

Full-text search

New features

• Full-text search is now supported in storage-compute separation instances.

• Full-text search is now supported on primary key tables.

• Added multiple Match functions to enrich full-text search capabilities.

Enhancements
Optimized the number of index files to improve access efficiency in storage-compute separation mode.

Category

Description

Base engine

New features

• Supports SSL encryption.

• Statistics collection tasks now support a synchronous mode option.

• Spark Load now supports dynamic configuration updates.

Enhancements

• Optimized the reload performance of materialized views (MVs), significantly improving FE startup efficiency.

• Optimized the coordination between aggregate operation pushdown and Broadcast Join.

• Fixed an issue where the row limit was not effective in Broadcast operations.

• Improved the accuracy of execution plans generated by GROUP_CONCAT.

• Optimized the lock granularity for materialized view refreshes.

• Enhanced the Metrics API to provide more monitoring metrics.

• Optimized the timeout handling of the Get Shard Info RPC in high-concurrency scenarios.

Fixes

• Fixed a CN/BE crash caused by a schema change when upgrading from v3.2 to v3.3.

• Fixed a data loss issue during insertion caused by Report Exec State.

• Fixed a CN crash that occurred after Fast Schema Evolution was enabled.

• Fixed an issue where an insert task did not terminate properly after a table was dropped.

• Fixed an issue where a version could not be found during a query.

• Fixed an issue where a materialized view refresh task got stuck after the task was killed.

• Fixed an issue where incorrect execution plans were generated during multi-stage materialized view rewriting.

• Fixed an issue where tablets could not be fully loaded.

• Fixed an issue where materialized view relationships were not correctly built after an FE image was loaded.

• Fixed a query failure (Query Terminated exception) caused by the DESC Table Cache.

• Fixed an issue where transaction results were abnormal.

• Fixed an issue where incorrect results were returned for Join and Union queries.

• Fixed an issue where the n_gram function returned incorrect query results.

• Fixed an issue that prevented data lineage from functioning correctly in DataWorks.

• Fixed an issue where DESC TABLE returned incorrect information for Paimon tables.

• Fixed an issue where Paimon materialized views were always fully refreshed.

Storage-compute separation

New features

• Supports configurable Compaction Punish time.

• Supports Tablet Meta Cache and Index Cache to significantly improve query performance.

• The _query_plan interface now supports specifying a warehouse.

• Supports compaction scheduling across multiple warehouses.

Enhancements

• Improved compaction observability.

• Switched to the Poco HTTP Client to optimize OSS access efficiency.

• Supports clearing the local cache after a DROP PARTITION operation to avoid occupying disk space with invalid data.

• Improved the performance of retrieving shard information.

• Improved the direct read performance of COUNT operations.

• Enhanced the observability of load tasks in a multi-warehouse environment.

• Supports adaptive file formats to reduce IOPS and improve query performance.

Fixes

• Fully enabled the cloud-native primary key (PK) index and fixed several related metadata issues.

• Fixed an issue where compaction could not be scheduled properly.

Lakehouse

New features

• Supports Alibaba Cloud Data Lake Formation (DLF) 2.5.

• Paimon external tables now support time-type fields.

• Supports querying properties of Paimon external tables using SHOW CREATE TABLE.

Enhancements

• Reduced redundant I/O when materialized views retrieve metadata frequently or on a schedule, improving refresh efficiency.

• Optimized Paimon metadata cache configurations.

• Fixed an issue where filtering on the FLOAT type was not effective in Paimon queries.

• Improved the performance of reading partition information from DLF.

• Supports metadata refresh for MaxCompute external tables.

• Uses UUIDs instead of creation times to optimize Paimon query performance.

• Improved the query performance of Paimon Deletion Vector tables.

• Optimized the scanning efficiency of Parquet files.

Fixes

• Fixed an issue where incorrect time zones were applied to the DATETIME type in Paimon queries.

• Fixed a failure to read the VARCHAR type from Hudi data sources.

• Fixed an issue where Paimon commit information was written incorrectly.

Full-text search

New features

• Released the full-text search feature in beta. This version resolves multiple CLucene performance issues, improving text search performance by more than 100%.

• Supports Tokenize-related functions.

Enhancements

• Optimized the file structure of inverted indexes to improve query efficiency.

• Enhanced the Index Tool to facilitate faster problem identification and resolution.