Append Delta Table - Hash Cluster(邀测)

更新时间:
复制 MD 格式

MaxComputeAppend Delta Table表格式中进一步支持Hash Cluster,在支持增量数据处理的同时提升查询性能。本文介绍其与其他表类型的差异、建表语法、SQL及数据通道 SDK 使用示例。

适用场景

推荐在以下场景使用 Hash Cluster:

  • 等值过滤查询:按固定列做点查或等值过滤,减少扫描量。

  • 等值JOIN / GROUP BY:多表按相同KEY关联或聚合,减少 Shuffle。

与相似表类型对比

表类型

聚簇方式

增量写入(ACID)

Hash Clustered Table

Hash

不支持

有 Hash Clustering(Shuffle + Sort)优化,但不支持 ACID。

PK Delta Table

Hash

支持

有 ACID 能力与 Hash Clustering 优化,适用于有主键数据。读写性能弱于无主键的表。

Append Delta Table - Range Cluster

Range

支持

有 ACID 能力与重聚簇支持,但聚簇方式为 Range,写入性能弱于 Hash。

Append Delta Table - Hash Cluster

Hash

支持

集 Hash Clustering 优化(Shuffle + Sort)与完整 ACID 能力于一体,同时支持后台增量与全量重聚簇。是上述三类表中综合能力最完整的类型。

前提条件

开启以下 Session 参数后再建表:

SET odps.table.append2.enable=true;
SET odps.table.hash.delta.enable=true; -- hash delta 建表试用开关

建表语法

CREATE TABLE [IF NOT EXISTS] <table_name>
             [(<col_name> <data_type> [comment <col_comment>], ...)]
             [PARTITIONED BY (<col_name> <data_type> [comment <col_comment>], ...)]
             CLUSTERED BY (<col_name> [, <col_name>, ...])
             [SORTED BY (<col_name> [, <col_name>, ...])] -- 仅支持升序
             INTO <number_of_buckets> BUCKETS
             TBLPROPERTIES ('table.format.version' = '2'); 

参数说明

参数

说明

CLUSTERED BY

指定 Hash 分桶列。建议选择在等值过滤、JOIN、GROUP BY 或 WINDOW PARTITION BY 中频繁使用的列,且该列的取值种类尽量多,以充分发挥分桶效果。

SORTED BY

可选。指定桶内排序列。当前仅支持升序排序。建议选择范围/等值过滤列、窗口计算或版本时间列。

INTO ... BUCKETS

指定逻辑 Bucket 数。建议结合数据量、查询并发和分桶列基数设置。 Bucket 数量和写表时并发度,读表 Shuffle 优化时并发度相关。

table.format.version

设置为 2,即表示创建适用于Hash Cluster的表数据格式。

SQL 使用示例

本示例以商品状态和价格版本表为例。业务上通常会按 item_id 点查或关联商品,因此将 item_id 作为 Hash 分桶列;版本生效时间 event_time 用于查看历史变化,因此将 event_time 作为桶内排序列。该设计适用于缓慢变化维表、商品价格版本表、状态变更明细表等场景。

准备工作

SET odps.sql.type.system.odps2=true;
SET odps.table.append2.enable=true;
SET odps.table.hash.delta.enable=true;

建表

非分区表

CREATE TABLE hash_delta_sales_demo (
  item_id BIGINT,
  event_time TIMESTAMP,
  price DOUBLE,
  status STRING
)
CLUSTERED BY (item_id)
SORTED BY (event_time)
INTO 256 BUCKETS 
TBLPROPERTIES ('table.format.version' = '2');

分区表

如果数据需要按日期管理,也可以创建分区表:

CREATE TABLE hash_delta_sales_demo_pt (
  item_id BIGINT,
  event_time TIMESTAMP,
  price DOUBLE,
  status STRING
)
PARTITIONED BY (ds STRING)
CLUSTERED BY (item_id)
SORTED BY (event_time)
INTO 256 BUCKETS
TBLPROPERTIES ('table.format.version' = '2');

通过DESC EXTENDED hash_delta_sales_demo;查看表信息。表的分桶和排序定义如下:

ClusterType:              hash
BucketNum:                256
ClusterColumns:           [item_id]
SortColumns:              [event_time ASC]

增量写入

以下以非分区表为例,演示增量写入和重聚簇过程。

  • 初始写入数据

    INSERT INTO TABLE hash_delta_sales_demo VALUES
      (1001, TIMESTAMP '2026-05-01 10:00:00', 10.00, 'active'),
      (1001, TIMESTAMP '2026-05-03 10:00:00', 13.00, 'active'),
      (1002, TIMESTAMP '2026-05-01 11:00:00', 20.00, 'active');
    
    DESC EXTENDED hash_delta_sales_demo;

    单击查看执行结果示例

    关键字段:

    • DataPhysicalClustered: true -- 数据已按 item_id 物理分桶

    • DataFullySorted: true -- 桶内数据已按 event_time 全量排序

    +------------------------------------------------------------------------------------+
    | Owner:                    ALIYUN$***_com                                           |
    | Project:                  test                                                     |
    | TableComment:                                                                      |
    +------------------------------------------------------------------------------------+
    | CreateTime:               2026-07-08 16:38:34                                      |
    | LastDDLTime:              2026-07-08 16:38:34                                      |
    | LastModifiedTime:         2026-07-08 16:39:38                                      |
    +------------------------------------------------------------------------------------+
    | InternalTable: YES      | Size: 4823                                               |
    +------------------------------------------------------------------------------------+
    | Native Columns:                                                                    |
    +------------------------------------------------------------------------------------+
    | Field    | Type   | Label | ExtendedLabel | Nullable | DefaultValue | Comment      |
    +------------------------------------------------------------------------------------+
    | item_id  | bigint |       |               | true     | NULL         |              |
    | event_time | timestamp |  |               | true     | NULL         |              |
    | price    | double |       |               | true     | NULL         |              |
    | status   | string |       |               | true     | NULL         |              |
    +------------------------------------------------------------------------------------+
    | Extended Info:                                                                     |
    +------------------------------------------------------------------------------------+
    | TableID:                  65**8e                                                   |
    | IsArchived:               false                                                    |
    | PhysicalSize:             14469                                                    |
    | FileNum:                  5                                                        |
    | ColdStorageStatus:        N/A                                                      |
    | CompressionStrategy:      normal                                                   |
    | DataFullySorted:          true                                                     |
    | DataPhysicalClustered:    true                                                     |
    | IsolationMin:             NONSTRICT_SNAPSHOT_ISOLATION                             |
    | OverlapDepth:             2                                                        |
    | OverlapRatio:             1.000000                                                 |
    | StoredAs:                 AliOrc                                                   |
    | Transactional:            true                                                     |
    | encryption_enable:        false                                                    |
    | odps.timemachine.retention.days: 1                                                        |
    | ClusterType:              hash                                                     |
    | BucketNum:                256                                                      |
    | ClusterColumns:           [item_id]                                                |
    | SortColumns:              [event_time ASC]                                         |
    | StorageTier:              Standard                                                 |
    | StorageTierLastModifiedTime:  2026-07-08 16:39:38                                  |
    +------------------------------------------------------------------------------------+
  • 删除操作

    删除一部分数据后查看表状态:

    DELETE FROM hash_delta_sales_demo WHERE item_id = 1002;
    
    DESC EXTENDED hash_delta_sales_demo;

    单击查看执行结果示例

    关键字段:

    • DataPhysicalClustered: true -- 数据已按 item_id 物理分桶

    • DataFullySorted: true -- 删除操作不破坏已有数据的排序状态

    +------------------------------------------------------------------------------------+
    | Owner:                    ALIYUN$***_com                                           |
    | Project:                  test                                                     |
    | TableComment:                                                                      |
    +------------------------------------------------------------------------------------+
    | CreateTime:               2026-07-08 16:38:34                                      |
    | LastDDLTime:              2026-07-08 16:38:34                                      |
    | LastModifiedTime:         2026-07-08 16:41:02                                      |
    | LastAccessTime:           2026-07-08 16:40:57                                      |
    +------------------------------------------------------------------------------------+
    | InternalTable: YES      | Size: 7082                                               |
    +------------------------------------------------------------------------------------+
    | Native Columns:                                                                    |
    +------------------------------------------------------------------------------------+
    | Field    | Type   | Label | ExtendedLabel | Nullable | DefaultValue | Comment      |
    +------------------------------------------------------------------------------------+
    | item_id  | bigint |       |               | true     | NULL         |              |
    | event_time | timestamp |  |               | true     | NULL         |              |
    | price    | double |       |               | true     | NULL         |              |
    | status   | string |       |               | true     | NULL         |              |
    +------------------------------------------------------------------------------------+
    | Extended Info:                                                                     |
    +------------------------------------------------------------------------------------+
    | TableID:                  65**8e                                                   |
    | IsArchived:               false                                                    |
    | PhysicalSize:             21246                                                    |
    | FileNum:                  10                                                       |
    | ColdStorageStatus:        N/A                                                      |
    | CompressionStrategy:      normal                                                   |
    | DataFullySorted:          true                                                     |
    | DataPhysicalClustered:    true                                                     |
    | IsolationMin:             NONSTRICT_SNAPSHOT_ISOLATION                             |
    | OverlapDepth:             2                                                        |
    | OverlapRatio:             1.000000                                                 |
    | StoredAs:                 AliOrc                                                   |
    | Transactional:            true                                                     |
    | encryption_enable:        false                                                    |
    | odps.timemachine.retention.days: 1                                                        |
    | ClusterType:              hash                                                     |
    | BucketNum:                256                                                      |
    | ClusterColumns:           [item_id]                                                |
    | SortColumns:              [event_time ASC]                                         |
    | StorageTier:              Standard                                                 |
    | StorageTierLastModifiedTime:  2026-07-08 16:41:02                                  |
    +------------------------------------------------------------------------------------+
    
  • 补写历史数据

    补写一条历史版本。该版本的 event_time 位于 item_id=1001 已有时间范围中间:

    INSERT INTO TABLE hash_delta_sales_demo VALUES
      (1001, TIMESTAMP '2026-05-02 09:00:00', 12.00, 'active');
    
    DESC EXTENDED hash_delta_sales_demo;

    单击查看执行结果示例

    关键字段:

    • DataPhysicalClustered: true -- 数据已按 item_id 物理分桶

    • DataFullySorted: false -- 新增文件使桶内数据不再全量有序

    +------------------------------------------------------------------------------------+
    | Owner:                    ALIYUN$***_com                                           |
    | Project:                  test                                                     |
    | TableComment:                                                                      |
    +------------------------------------------------------------------------------------+
    | CreateTime:               2026-07-08 16:38:34                                      |
    | LastDDLTime:              2026-07-08 16:38:34                                      |
    | LastModifiedTime:         2026-07-08 16:42:54                                      |
    | LastAccessTime:           2026-07-08 16:40:57                                      |
    +------------------------------------------------------------------------------------+
    | InternalTable: YES      | Size: 10705                                              |
    +------------------------------------------------------------------------------------+
    | Native Columns:                                                                    |
    +------------------------------------------------------------------------------------+
    | Field    | Type   | Label | ExtendedLabel | Nullable | DefaultValue | Comment      |
    +------------------------------------------------------------------------------------+
    | item_id  | bigint |       |               | true     | NULL         |              |
    | event_time | timestamp |       |               | true     | NULL         |              |
    | price    | double |       |               | true     | NULL         |              |
    | status   | string |       |               | true     | NULL         |              |
    +------------------------------------------------------------------------------------+
    | Extended Info:                                                                     |
    +------------------------------------------------------------------------------------+
    | TableID:                  65**8e                                                   |
    | IsArchived:               false                                                    |
    | PhysicalSize:             32115                                                    |
    | FileNum:                  13                                                       |
    | ColdStorageStatus:        N/A                                                      |
    | CompressionStrategy:      normal                                                   |
    | DataFullySorted:          false                                                    |
    | DataPhysicalClustered:    true                                                     |
    | IsolationMin:             NONSTRICT_SNAPSHOT_ISOLATION                             |
    | OverlapDepth:             2                                                        |
    | OverlapRatio:             1.000000                                                 |
    | StoredAs:                 AliOrc                                                   |
    | Transactional:            true                                                     |
    | encryption_enable:        false                                                    |
    | odps.timemachine.retention.days: 1                                                        |
    | ClusterType:              hash                                                     |
    | BucketNum:                256                                                      |
    | ClusterColumns:           [item_id]                                                |
    | SortColumns:              [event_time ASC]                                         |
    | StorageTier:              Standard                                                 |
    | StorageTierLastModifiedTime:  2026-07-08 16:42:54                                  |
    +------------------------------------------------------------------------------------+
    
    

Bucket Pruning

对分桶列做等值查询时,MaxCompute 会根据 Hash 分布直接定位目标 Bucket,跳过其余 Bucket 的扫描。以下查询按 item_id = 1001 过滤,实际只读取该值所在的 1 个逻辑Bucket,而非全表扫描:

SELECT * FROM hash_delta_sales_demo
  WHERE item_id = 1001
  ORDER BY event_time
  LIMIT 10;

-- 返回结果:
+------------+---------------------+------------+--------+
| item_id    | event_time          | price      | status |
+------------+---------------------+------------+--------+
| 1001       | 2026-05-01 10:00:00 | 10.0       | active |
| 1001       | 2026-05-02 09:00:00 | 12.0       | active |
| 1001       | 2026-05-03 10:00:00 | 13.0       | active |
+------------+---------------------+------------+--------+

全量重聚簇

如果需要重新整理存量数据,执行 RECLUSTER FULL。该操作会保留表中的历史数据语义,并按照当前表定义重新组织存储数据。

ALTER TABLE hash_delta_sales_demo RECLUSTER FULL;

DESC EXTENDED hash_delta_sales_demo;

单击查看执行结果示例

关键字段:

  • DataPhysicalClustered: true -- 数据已按 item_id 物理分桶

  • DataFullySorted: true -- 桶内数据已全量排序

+------------------------------------------------------------------------------------+
| Owner:                    ALIYUN$***_com                                           |
| Project:                  test                                                     |
| TableComment:                                                                      |
+------------------------------------------------------------------------------------+
| CreateTime:               2026-07-08 16:38:34                                      |
| LastDDLTime:              2026-07-08 16:38:34                                      |
| LastModifiedTime:         2026-07-08 16:42:54                                      |
| LastAccessTime:           2026-07-08 16:40:57                                      |
+------------------------------------------------------------------------------------+
| InternalTable: YES      | Size: 20218                                              |
+------------------------------------------------------------------------------------+
| Native Columns:                                                                    |
+------------------------------------------------------------------------------------+
| Field    | Type   | Label | ExtendedLabel | Nullable | DefaultValue | Comment      |
+------------------------------------------------------------------------------------+
| item_id  | bigint |       |               | true     | NULL         |              |
| event_time | timestamp |       |               | true     | NULL         |              |
| price    | double |       |               | true     | NULL         |              |
| status   | string |       |               | true     | NULL         |              |
+------------------------------------------------------------------------------------+
| Extended Info:                                                                     |
+------------------------------------------------------------------------------------+
| TableID:                  65**8e                                                   |
| IsArchived:               false                                                    |
| PhysicalSize:             60654                                                    |
| FileNum:                  20                                                       |
| ColdStorageStatus:        N/A                                                      |
| CompressionStrategy:      normal                                                   |
| DataFullySorted:          true                                                     |
| DataPhysicalClustered:    true                                                     |
| IsolationMin:             NONSTRICT_SNAPSHOT_ISOLATION                             |
| OverlapDepth:             2                                                        |
| OverlapRatio:             1.000000                                                 |
| StoredAs:                 AliOrc                                                   |
| Transactional:            true                                                     |
| encryption_enable:        false                                                    |
| odps.timemachine.retention.days: 1                                                        |
| ClusterType:              hash                                                     |
| BucketNum:                256                                                      |
| ClusterColumns:           [item_id]                                                |
| SortColumns:              [event_time ASC]                                         |
| StorageTier:              Standard                                                 |
| StorageTierLastModifiedTime:  2026-07-08 16:42:54                                  |
+------------------------------------------------------------------------------------+

Hash Delta 表支持 INSERT、UPDATE、DELETE、MERGE INTO 等增量写入,同时保留 Hash 分布信息。优化器基于当前数据状态选择执行计划:数据有序时利用有序存储,数据变为非全量有序后仍可利用 Hash 分桶,必要时通过RECLUSTER FULL 恢复全量排序。

数据通道使用示例

以上文中 hash_delta_sales_demo 表为例,介绍通过数据传输服务 SDK 进行 Hash Clustered Delta Table 的数据上传/下载。

  1. 导入SDK依赖包

    建议使用最新版本,至少需要升级到0.59版本及以上。详情参见版本更新记录

  2. 示例代码

    单击查看示例代码

    /**
     * 使用 MaxStorageClient 对 hash_delta_sales_demo 表进行上传和下载的示例代码。
     *
     * 表结构:
     * CREATE TABLE hash_delta_sales_demo (
     *   item_id BIGINT,
     *   event_time TIMESTAMP,
     *   price DOUBLE,
     *   status STRING
     * );
     */
    public class MaxStorageClientExample {
    
        private static final String ENDPOINT = "<your-endpoint>";
        private static final String TUNNEL_ENDPOINT = "<your-tunnel-endpoint>";
        private static final String PROJECT = "<your-project>";
        private static final String ACCESS_ID = "<your-access-id>";
        private static final String ACCESS_KEY = "<your-access-key>";
        private static final String TABLE_NAME = "hash_delta_sales_demo";
    
    
        public static void main(String[] args) throws Exception {
            RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
    
            // 1. 构建 MaxStorageClient
            MaxStorageClient client = MaxStorageClient.builder()
                    .endpoint(ENDPOINT)
                    .tunnelEndpoint(TUNNEL_ENDPOINT)
                    .credentialsProvider(
                            new StaticCredentialProvider(new AliyunAccount(ACCESS_ID, ACCESS_KEY).getCredentials()))
                    .project(PROJECT)
                    .bufferAllocator(allocator)
                    .build();
    
            try {
                // 2. 上传数据
                uploadData(client);
                Thread.sleep(5000);
    
                // 3. 下载数据
                downloadData(client);
            } finally {
                client.close();
                allocator.close();
            }
        }
    
        /**
         * 上传数据到 hash_delta_sales_demo 表
         */
        private static void uploadData(MaxStorageClient client) throws Exception {
            TableIdentifier tableId = TableIdentifier.of(PROJECT, TABLE_NAME);
    
            // 创建写入会话,withOverwrite(true) 表示覆盖写入
            TableWriteSession writeSession = client
                    .createTableWriteSessionBuilder(tableId)
                    .withOverwrite(true)
                    .build();
    
            System.out.println("Write session created: " + writeSession.getId());
    
            // 使用 RecordWriter 写入数据(高级 API,按行写入)
            try (RecordWriter writer = writeSession.createWriterBuilder("stream-1", 1)
                    .build()
                    .getAsRecordWriter(1024)) {
    
                for (int i = 0; i < 1000; i++) {
                    Record record = writer.newRecord(false);
                    record.set(0, (long) i);                                    // item_id: BIGINT
                    record.set(1, LocalDateTime.of(2025, 5, 18, 10, 30, i % 60).atZone(ZoneId.systemDefault())
                            .toInstant()); // event_time: TIMESTAMP
                    record.set(2, 99.9 + i * 0.1);                             // price: DOUBLE
                    record.set(3, i % 2 == 0 ? "paid" : "pending");            // status: STRING
                    writer.write(record);
                }
            }
    
            // 提交会话,数据对外可见
            writeSession.commit();
            System.out.println("Successfully uploaded 1000 records to " + TABLE_NAME);
        }
    
        /**
         * 从 hash_delta_sales_demo 表下载数据
         */
        private static void downloadData(MaxStorageClient client) throws Exception {
            TableIdentifier tableId = TableIdentifier.of(PROJECT, TABLE_NAME);
    
            // 创建读取会话,可以选择列和过滤条件
            TableReadSession readSession = client.createTableReadSessionBuilder(tableId)
                    .withColumns(Arrays.asList("item_id", "event_time", "price", "status"))
                    .withSplitOptions(SplitOptions.newBuilder()
                            .withSplitMode(SplitMode.ROW_OFFSET)
                            .build())
                    .build();
    
            System.out.println("Read session created: " + readSession.getId());
    
            // 获取数据分片
            List<InputSplit> splits = readSession.getSplits();
            System.out.println("Total splits: " + splits.size());
    
            int totalRecords = 0;
    
            // 遍历每个分片读取数据
            for (InputSplit split : splits) {
                try (ArrowReader reader = readSession.createReaderBuilder(split).build()) {
                    Schema schema = reader.getSchema();
                    System.out.println("Schema: " + schema);
    
                    while (reader.nextBatch()) {
                        VectorSchemaRoot root = reader.getCurrentValue();
                        int rowCount = root.getRowCount();
                        totalRecords += rowCount;
    
                        // 打印前 5 行作为示例
                        int printCount = Math.min(rowCount, 5);
                        for (int i = 0; i < printCount; i++) {
                            System.out.printf("  item_id=%s, event_time=%s, price=%s, status=%s%n",
                                    root.getVector("item_id").getObject(i),
                                    root.getVector("event_time").getObject(i),
                                    root.getVector("price").getObject(i),
                                    root.getVector("status").getObject(i));
                        }
                        if (rowCount > 5) {
                            System.out.println("  ... (" + (rowCount - 5) + " more rows in this batch)");
                        }
                    }
                }
            }
    
            System.out.println("Total records downloaded: " + totalRecords);
        }
    }

常见问题

如何选择合适的单 Bucket 存储量?

建议将单 Bucket 存储量控制在数百 MB 到数十 GB 之间。

  • Bucket 过小:存储开销上升,Shuffle 代价增大。

  • Bucket 过大:写表时间较长,查询时 Bucket Pruning 的过滤效果下降, Shuffle 优化效果下降。

建议根据预期数据增长量(而非当前量)来设置 Bucket 数,避免频繁修改表结构。

如果业务数据量确实偏大,单 Bucket 也可支持更大存储,也可以将 Bucket Num 设置的更大,但需结合实际写入与查询性能综合评估。