测试结果参考

更新时间:
复制 MD 格式

本文根据测试方案介绍中的场景,提供1 TB TPC-DS数据集下Hologres内表与Iceberg外表的查询性能测试结果,供您评估Hologres在复杂分析场景下的性能表现时参考。

本文在Hologres实例上,对比两种存储形态在1 TB TPC-DS数据集上的查询性能:

  • 内表(Internal Table):数据以Hologres原生列存格式存储在实例本地。

  • 数据湖外表(Iceberg External Table):数据以Apache Iceberg表格式存储在对象存储上,通过DLF REST Catalog挂载,由Hologres计算引擎直接查询。

测试准备

Hologres配置

参数

说明

软件版本

Hologres V4.2.12

计算资源

64 CU

内表与外表测试运行在完全相同的64 CU实例上,客户端、网络与并发配置一致,唯一变量是被查询数据的存储形态。

存储形态

内表

Iceberg外表

存储格式

Hologres原生列存

Apache Iceberg(Parquet数据文件)

元数据或Catalog

Hologres内置

DLF REST Catalog(dlf-rest

存储位置

实例本地存储

对象存储(数据湖)

查询引擎

Hologres

Hologres(同一引擎直接查询外表)

数据集

TPC-DSscale factor1000,即约1 TB原始数据,共24张表、99条查询。内表与外表加载的是同一份逻辑数据。

测试方法

两组测试均采用hot run方法,与业界基准一致:

  1. 每次运行前清理实例数据缓存,保证每条查询的warmup run从冷启动开始。

  2. 采用长连接(persistent connection)逐条执行查询。

  3. 每条查询预热1次(不计时),再正式执行3次取平均值。

  4. 外表侧在查询前对全部24张表执行ANALYZE收集统计信息(外表不支持VACUUM);内表侧执行VACUUMANALYZE

内表测试流程请参见测试方案介绍

外表测试步骤:当DLF数据湖上的Iceberg数据已按附录:Iceberg TPC-DS表结构DDL准备就绪后,使用同一测试工具包,在配置Hologres连接信息时将db填写为外部数据库名,然后按仅执行查询测试的方式运行即可。

测试结果

下表为TPC-DS 99条查询各自的平均响应时间,单位为毫秒(ms)。

Query

V4.2.12内表查询耗时(ms)

V4.2.12 Iceberg外表查询耗时(ms)

Total

263,047

340,286

q1

627

781

q2

1,732

2,035

q3

318

711

q4

14,008

16,364

q5

569

1,109

q6

373

609

q7

2,674

2,372

q8

285

588

q9

7,889

7,518

q10

573

897

q11

10,480

10,849

q12

118

198

q13

2,911

3,852

q14

10,010

15,457

q15

671

1,098

q16

326

653

q17

922

1,295

q18

1,150

2,573

q19

318

697

q20

154

347

q21

118

209

q22

257

369

q23

37,056

50,806

q24

9,605

12,665

q25

748

1,146

q26

333

832

q27

956

1,699

q28

9,537

10,041

q29

1,430

1,329

q30

424

609

q31

2,372

3,011

q32

67

289

q33

533

835

q34

999

2,409

q35

1,464

1,800

q36

435

1,057

q37

100

209

q38

4,458

5,029

q39

530

859

q40

194

496

q41

56

92

q42

111

282

q43

715

1,075

q44

151

286

q45

798

879

q46

4,828

5,466

q47

3,786

4,692

q48

1,969

2,203

q49

409

998

q50

1,598

1,800

q51

2,305

3,044

q52

114

365

q53

304

885

q54

347

842

q55

111

362

q56

384

718

q57

1,800

2,203

q58

278

667

q59

3,281

5,096

q60

559

999

q61

573

919

q62

697

933

q63

282

671

q64

3,752

5,534

q65

2,642

3,751

q66

1,067

1,598

q67

26,594

31,735

q68

1,901

5,904

q69

337

537

q70

908

1,564

q71

1,415

1,817

q72

4,727

4,860

q73

526

1,565

q74

8,123

8,056

q75

8,931

8,595

q76

4,862

5,769

q77

533

810

q78

10,916

9,704

q79

4,525

6,845

q80

1,194

1,665

q81

824

1,006

q82

176

402

q83

275

464

q84

234

464

q85

889

1,430

q86

183

340

q87

4,794

5,803

q88

3,987

4,458

q89

577

1,228

q90

497

1,329

q91

216

555

q92

67

314

q93

2,641

4,356

q94

424

992

q95

867

2,102

q96

3,045

4,054

q97

5,500

7,283

q98

527

654

q99

1,191

1,564

结论综述

在同一64 CU Hologres实例上,内表凭借原生列存、完善统计信息与本地缓存,在1 TB TPC-DS数据集上整体查询耗时低于Iceberg外表,外表的总耗时约为内表的1.29倍。而Hologres直接查询Iceberg外表在无需任何数据导入的前提下,即可交付与内表可比的分析性能。

上述结果验证了Hologres一套引擎、内表与湖表统一查询的湖仓一体能力:对追求性能的核心负载可落地为内表;对以数据湖为主存储、强调开放格式与即席直接查询的场景,则可直接查询外表。二者性能梯度平滑,可按需选择。

附录:Iceberg TPC-DS表结构DDL

以下为本次测试使用的24IcebergDDL。只需数据湖上的数据符合此schema即可执行查询测试,数据写入方式不限。

维度表DDL

CREATE TABLE IF NOT EXISTS customer_address (
    ca_address_sk             INT        NOT NULL,
    ca_address_id             STRING     NOT NULL,
    ca_street_number          STRING,
    ca_street_name            STRING,
    ca_street_type            STRING,
    ca_suite_number           STRING,
    ca_city                   STRING,
    ca_county                 STRING,
    ca_state                  STRING,
    ca_zip                    STRING,
    ca_country                STRING,
    ca_gmt_offset             DECIMAL(5,2),
    ca_location_type          STRING
) USING iceberg
PARTITIONED BY (bucket(64, ca_address_sk))
TBLPROPERTIES (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy'
);

CREATE TABLE IF NOT EXISTS customer_demographics (
    cd_demo_sk                INT        NOT NULL,
    cd_gender                 STRING,
    cd_marital_status         STRING,
    cd_education_status       STRING,
    cd_purchase_estimate      INT,
    cd_credit_rating          STRING,
    cd_dep_count              INT,
    cd_dep_employed_count     INT,
    cd_dep_college_count      INT
) USING iceberg
PARTITIONED BY (bucket(64, cd_demo_sk))
TBLPROPERTIES (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy'
);

CREATE TABLE IF NOT EXISTS date_dim (
    d_date_sk                 INT        NOT NULL,
    d_date_id                 STRING     NOT NULL,
    d_date                    DATE       NOT NULL,
    d_month_seq               INT,
    d_week_seq                INT,
    d_quarter_seq             INT,
    d_year                    INT,
    d_dow                     INT,
    d_moy                     INT,
    d_dom                     INT,
    d_qoy                     INT,
    d_fy_year                 INT,
    d_fy_quarter_seq          INT,
    d_fy_week_seq             INT,
    d_day_name                STRING,
    d_quarter_name            STRING,
    d_holiday                 STRING,
    d_weekend                 STRING,
    d_following_holiday       STRING,
    d_first_dom               INT,
    d_last_dom                INT,
    d_same_day_ly             INT,
    d_same_day_lq             INT,
    d_current_day             STRING,
    d_current_week            STRING,
    d_current_month           STRING,
    d_current_quarter         STRING,
    d_current_year            STRING
) USING iceberg
PARTITIONED BY (bucket(64, d_date_sk))
TBLPROPERTIES (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy'
);

CREATE TABLE IF NOT EXISTS warehouse (
    w_warehouse_sk            INT        NOT NULL,
    w_warehouse_id            STRING     NOT NULL,
    w_warehouse_name          STRING,
    w_warehouse_sq_ft         INT,
    w_street_number           STRING,
    w_street_name             STRING,
    w_street_type             STRING,
    w_suite_number            STRING,
    w_city                    STRING,
    w_county                  STRING,
    w_state                   STRING,
    w_zip                     STRING,
    w_country                 STRING,
    w_gmt_offset              DECIMAL(5,2)
) USING iceberg
PARTITIONED BY (bucket(64, w_warehouse_sk))
TBLPROPERTIES (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy'
);

CREATE TABLE IF NOT EXISTS ship_mode (
    sm_ship_mode_sk           INT        NOT NULL,
    sm_ship_mode_id           STRING     NOT NULL,
    sm_type                   STRING,
    sm_code                   STRING,
    sm_carrier                STRING,
    sm_contract               STRING
) USING iceberg
PARTITIONED BY (bucket(64, sm_ship_mode_sk))
TBLPROPERTIES (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy'
);

CREATE TABLE IF NOT EXISTS time_dim (
    t_time_sk                 INT        NOT NULL,
    t_time_id                 STRING     NOT NULL,
    t_time                    INT        NOT NULL,
    t_hour                    INT,
    t_minute                  INT,
    t_second                  INT,
    t_am_pm                   STRING,
    t_shift                   STRING,
    t_sub_shift               STRING,
    t_meal_time               STRING
) USING iceberg
PARTITIONED BY (bucket(64, t_time_sk))
TBLPROPERTIES (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy'
);

CREATE TABLE IF NOT EXISTS reason (
    r_reason_sk               INT        NOT NULL,
    r_reason_id               STRING     NOT NULL,
    r_reason_desc             STRING
) USING iceberg
PARTITIONED BY (bucket(64, r_reason_sk))
TBLPROPERTIES (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy'
);

CREATE TABLE IF NOT EXISTS income_band (
    ib_income_band_sk         INT        NOT NULL,
    ib_lower_bound            INT,
    ib_upper_bound            INT
) USING iceberg
PARTITIONED BY (bucket(64, ib_income_band_sk))
TBLPROPERTIES (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy'
);

CREATE TABLE IF NOT EXISTS item (
    i_item_sk                 INT        NOT NULL,
    i_item_id                 STRING     NOT NULL,
    i_rec_start_date          DATE,
    i_rec_end_date            DATE,
    i_item_desc               STRING,
    i_current_price           DECIMAL(7,2),
    i_wholesale_cost          DECIMAL(7,2),
    i_brand_id                INT,
    i_brand                   STRING,
    i_class_id                INT,
    i_class                   STRING,
    i_category_id             INT,
    i_category                STRING,
    i_manufact_id             INT,
    i_manufact                STRING,
    i_size                    STRING,
    i_formulation             STRING,
    i_color                   STRING,
    i_units                   STRING,
    i_container               STRING,
    i_manager_id              INT,
    i_product_name            STRING
) USING iceberg
PARTITIONED BY (bucket(64, i_item_sk))
TBLPROPERTIES (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy'
);

CREATE TABLE IF NOT EXISTS store (
    s_store_sk                INT        NOT NULL,
    s_store_id                STRING     NOT NULL,
    s_rec_start_date          DATE,
    s_rec_end_date            DATE,
    s_closed_date_sk          INT,
    s_store_name              STRING,
    s_number_employees        INT,
    s_floor_space             INT,
    s_hours                   STRING,
    s_manager                 STRING,
    s_market_id               INT,
    s_geography_class         STRING,
    s_market_desc             STRING,
    s_market_manager          STRING,
    s_division_id             INT,
    s_division_name           STRING,
    s_company_id              INT,
    s_company_name            STRING,
    s_street_number           STRING,
    s_street_name             STRING,
    s_street_type             STRING,
    s_suite_number            STRING,
    s_city                    STRING,
    s_county                  STRING,
    s_state                   STRING,
    s_zip                     STRING,
    s_country                 STRING,
    s_gmt_offset              DECIMAL(5,2),
    s_tax_percentage          DECIMAL(5,2)
) USING iceberg
PARTITIONED BY (bucket(64, s_store_sk))
TBLPROPERTIES (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy'
);

CREATE TABLE IF NOT EXISTS call_center (
    cc_call_center_sk         INT        NOT NULL,
    cc_call_center_id         STRING     NOT NULL,
    cc_rec_start_date         DATE,
    cc_rec_end_date           DATE,
    cc_closed_date_sk         INT,
    cc_open_date_sk           INT,
    cc_name                   STRING,
    cc_class                  STRING,
    cc_employees              INT,
    cc_sq_ft                  INT,
    cc_hours                  STRING,
    cc_manager                STRING,
    cc_mkt_id                 INT,
    cc_mkt_class              STRING,
    cc_mkt_desc               STRING,
    cc_market_manager         STRING,
    cc_division               INT,
    cc_division_name          STRING,
    cc_company                INT,
    cc_company_name           STRING,
    cc_street_number          STRING,
    cc_street_name            STRING,
    cc_street_type            STRING,
    cc_suite_number           STRING,
    cc_city                   STRING,
    cc_county                 STRING,
    cc_state                  STRING,
    cc_zip                    STRING,
    cc_country                STRING,
    cc_gmt_offset             DECIMAL(5,2),
    cc_tax_percentage         DECIMAL(5,2)
) USING iceberg
PARTITIONED BY (bucket(64, cc_call_center_sk))
TBLPROPERTIES (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy'
);

CREATE TABLE IF NOT EXISTS customer (
    c_customer_sk             INT        NOT NULL,
    c_customer_id             STRING     NOT NULL,
    c_current_cdemo_sk        INT,
    c_current_hdemo_sk        INT,
    c_current_addr_sk         INT,
    c_first_shipto_date_sk    INT,
    c_first_sales_date_sk     INT,
    c_salutation              STRING,
    c_first_name              STRING,
    c_last_name               STRING,
    c_preferred_cust_flag     STRING,
    c_birth_day               INT,
    c_birth_month             INT,
    c_birth_year              INT,
    c_birth_country           STRING,
    c_login                   STRING,
    c_email_address           STRING,
    c_last_review_date_sk     INT
) USING iceberg
PARTITIONED BY (bucket(64, c_customer_sk))
TBLPROPERTIES (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy'
);

CREATE TABLE IF NOT EXISTS web_site (
    web_site_sk               INT        NOT NULL,
    web_site_id               STRING     NOT NULL,
    web_rec_start_date        DATE,
    web_rec_end_date          DATE,
    web_name                  STRING,
    web_open_date_sk          INT,
    web_close_date_sk         INT,
    web_class                 STRING,
    web_manager               STRING,
    web_mkt_id                INT,
    web_mkt_class             STRING,
    web_mkt_desc              STRING,
    web_market_manager        STRING,
    web_company_id            INT,
    web_company_name          STRING,
    web_street_number         STRING,
    web_street_name           STRING,
    web_street_type           STRING,
    web_suite_number          STRING,
    web_city                  STRING,
    web_county                STRING,
    web_state                 STRING,
    web_zip                   STRING,
    web_country               STRING,
    web_gmt_offset            DECIMAL(5,2),
    web_tax_percentage        DECIMAL(5,2)
) USING iceberg
PARTITIONED BY (bucket(64, web_site_sk))
TBLPROPERTIES (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy'
);

CREATE TABLE IF NOT EXISTS household_demographics (
    hd_demo_sk                INT        NOT NULL,
    hd_income_band_sk         INT,
    hd_buy_potential          STRING,
    hd_dep_count              INT,
    hd_vehicle_count          INT
) USING iceberg
PARTITIONED BY (bucket(64, hd_demo_sk))
TBLPROPERTIES (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy'
);

CREATE TABLE IF NOT EXISTS web_page (
    wp_web_page_sk            INT        NOT NULL,
    wp_web_page_id            STRING     NOT NULL,
    wp_rec_start_date         DATE,
    wp_rec_end_date           DATE,
    wp_creation_date_sk       INT,
    wp_access_date_sk         INT,
    wp_autogen_flag           STRING,
    wp_customer_sk            INT,
    wp_url                    STRING,
    wp_type                   STRING,
    wp_char_count             INT,
    wp_link_count             INT,
    wp_image_count            INT,
    wp_max_ad_count           INT
) USING iceberg
PARTITIONED BY (bucket(64, wp_web_page_sk))
TBLPROPERTIES (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy'
);

CREATE TABLE IF NOT EXISTS promotion (
    p_promo_sk                INT        NOT NULL,
    p_promo_id                STRING     NOT NULL,
    p_start_date_sk           INT,
    p_end_date_sk             INT,
    p_item_sk                 INT,
    p_cost                    DECIMAL(15,2),
    p_response_target         INT,
    p_promo_name              STRING,
    p_channel_dmail           STRING,
    p_channel_email           STRING,
    p_channel_catalog         STRING,
    p_channel_tv              STRING,
    p_channel_radio           STRING,
    p_channel_press           STRING,
    p_channel_event           STRING,
    p_channel_demo            STRING,
    p_channel_details         STRING,
    p_purpose                 STRING,
    p_discount_active         STRING
) USING iceberg
PARTITIONED BY (bucket(64, p_promo_sk))
TBLPROPERTIES (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy'
);

CREATE TABLE IF NOT EXISTS catalog_page (
    cp_catalog_page_sk        INT        NOT NULL,
    cp_catalog_page_id        STRING     NOT NULL,
    cp_start_date_sk          INT,
    cp_end_date_sk            INT,
    cp_department             STRING,
    cp_catalog_number         INT,
    cp_catalog_page_number    INT,
    cp_description            STRING,
    cp_type                   STRING
) USING iceberg
PARTITIONED BY (bucket(64, cp_catalog_page_sk))
TBLPROPERTIES (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy'
);

事实表DDL

CREATE TABLE IF NOT EXISTS inventory (
    inv_date_sk               INT        NOT NULL,
    inv_item_sk               INT        NOT NULL,
    inv_warehouse_sk          INT        NOT NULL,
    inv_quantity_on_hand      INT
) USING iceberg
PARTITIONED BY (bucket(64, inv_item_sk))
TBLPROPERTIES (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy'
);

CREATE TABLE IF NOT EXISTS store_returns (
    sr_returned_date_sk       INT,
    sr_return_time_sk         INT,
    sr_item_sk                INT        NOT NULL,
    sr_customer_sk            INT,
    sr_cdemo_sk               INT,
    sr_hdemo_sk               INT,
    sr_addr_sk                INT,
    sr_store_sk               INT,
    sr_reason_sk              INT,
    sr_ticket_number          INT        NOT NULL,
    sr_return_quantity        INT,
    sr_return_amt             DECIMAL(7,2),
    sr_return_tax             DECIMAL(7,2),
    sr_return_amt_inc_tax     DECIMAL(7,2),
    sr_fee                    DECIMAL(7,2),
    sr_return_ship_cost       DECIMAL(7,2),
    sr_refunded_cash          DECIMAL(7,2),
    sr_reversed_charge        DECIMAL(7,2),
    sr_store_credit           DECIMAL(7,2),
    sr_net_loss               DECIMAL(7,2)
) USING iceberg
PARTITIONED BY (bucket(64, sr_item_sk))
TBLPROPERTIES (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy'
);

CREATE TABLE IF NOT EXISTS catalog_returns (
    cr_returned_date_sk       INT,
    cr_returned_time_sk       INT,
    cr_item_sk                INT        NOT NULL,
    cr_refunded_customer_sk   INT,
    cr_refunded_cdemo_sk      INT,
    cr_refunded_hdemo_sk      INT,
    cr_refunded_addr_sk       INT,
    cr_returning_customer_sk  INT,
    cr_returning_cdemo_sk     INT,
    cr_returning_hdemo_sk     INT,
    cr_returning_addr_sk      INT,
    cr_call_center_sk         INT,
    cr_catalog_page_sk        INT,
    cr_ship_mode_sk           INT,
    cr_warehouse_sk           INT,
    cr_reason_sk              INT,
    cr_order_number           INT        NOT NULL,
    cr_return_quantity        INT,
    cr_return_amount          DECIMAL(7,2),
    cr_return_tax             DECIMAL(7,2),
    cr_return_amt_inc_tax     DECIMAL(7,2),
    cr_fee                    DECIMAL(7,2),
    cr_return_ship_cost       DECIMAL(7,2),
    cr_refunded_cash          DECIMAL(7,2),
    cr_reversed_charge        DECIMAL(7,2),
    cr_store_credit           DECIMAL(7,2),
    cr_net_loss               DECIMAL(7,2)
) USING iceberg
PARTITIONED BY (bucket(64, cr_item_sk))
TBLPROPERTIES (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy'
);

CREATE TABLE IF NOT EXISTS web_returns (
    wr_returned_date_sk       INT,
    wr_returned_time_sk       INT,
    wr_item_sk                INT        NOT NULL,
    wr_refunded_customer_sk   INT,
    wr_refunded_cdemo_sk      INT,
    wr_refunded_hdemo_sk      INT,
    wr_refunded_addr_sk       INT,
    wr_returning_customer_sk  INT,
    wr_returning_cdemo_sk     INT,
    wr_returning_hdemo_sk     INT,
    wr_returning_addr_sk      INT,
    wr_web_page_sk            INT,
    wr_reason_sk              INT,
    wr_order_number           INT        NOT NULL,
    wr_return_quantity        INT,
    wr_return_amt             DECIMAL(7,2),
    wr_return_tax             DECIMAL(7,2),
    wr_return_amt_inc_tax     DECIMAL(7,2),
    wr_fee                    DECIMAL(7,2),
    wr_return_ship_cost       DECIMAL(7,2),
    wr_refunded_cash          DECIMAL(7,2),
    wr_reversed_charge        DECIMAL(7,2),
    wr_account_credit         DECIMAL(7,2),
    wr_net_loss               DECIMAL(7,2)
) USING iceberg
PARTITIONED BY (bucket(64, wr_item_sk))
TBLPROPERTIES (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy'
);

CREATE TABLE IF NOT EXISTS web_sales (
    ws_sold_date_sk           INT,
    ws_sold_time_sk           INT,
    ws_ship_date_sk           INT,
    ws_item_sk                INT        NOT NULL,
    ws_bill_customer_sk       INT,
    ws_bill_cdemo_sk          INT,
    ws_bill_hdemo_sk          INT,
    ws_bill_addr_sk           INT,
    ws_ship_customer_sk       INT,
    ws_ship_cdemo_sk          INT,
    ws_ship_hdemo_sk          INT,
    ws_ship_addr_sk           INT,
    ws_web_page_sk            INT,
    ws_web_site_sk            INT,
    ws_ship_mode_sk           INT,
    ws_warehouse_sk           INT,
    ws_promo_sk               INT,
    ws_order_number           INT        NOT NULL,
    ws_quantity               INT,
    ws_wholesale_cost         DECIMAL(7,2),
    ws_list_price             DECIMAL(7,2),
    ws_sales_price            DECIMAL(7,2),
    ws_ext_discount_amt       DECIMAL(7,2),
    ws_ext_sales_price        DECIMAL(7,2),
    ws_ext_wholesale_cost     DECIMAL(7,2),
    ws_ext_list_price         DECIMAL(7,2),
    ws_ext_tax                DECIMAL(7,2),
    ws_coupon_amt             DECIMAL(7,2),
    ws_ext_ship_cost          DECIMAL(7,2),
    ws_net_paid               DECIMAL(7,2),
    ws_net_paid_inc_tax       DECIMAL(7,2),
    ws_net_paid_inc_ship      DECIMAL(7,2),
    ws_net_paid_inc_ship_tax  DECIMAL(7,2),
    ws_net_profit             DECIMAL(7,2)
) USING iceberg
PARTITIONED BY (bucket(64, ws_item_sk))
TBLPROPERTIES (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy'
);

CREATE TABLE IF NOT EXISTS catalog_sales (
    cs_sold_date_sk           INT,
    cs_sold_time_sk           INT,
    cs_ship_date_sk           INT,
    cs_bill_customer_sk       INT,
    cs_bill_cdemo_sk          INT,
    cs_bill_hdemo_sk          INT,
    cs_bill_addr_sk           INT,
    cs_ship_customer_sk       INT,
    cs_ship_cdemo_sk          INT,
    cs_ship_hdemo_sk          INT,
    cs_ship_addr_sk           INT,
    cs_call_center_sk         INT,
    cs_catalog_page_sk        INT,
    cs_ship_mode_sk           INT,
    cs_warehouse_sk           INT,
    cs_item_sk                INT        NOT NULL,
    cs_promo_sk               INT,
    cs_order_number           INT        NOT NULL,
    cs_quantity               INT,
    cs_wholesale_cost         DECIMAL(7,2),
    cs_list_price             DECIMAL(7,2),
    cs_sales_price            DECIMAL(7,2),
    cs_ext_discount_amt       DECIMAL(7,2),
    cs_ext_sales_price        DECIMAL(7,2),
    cs_ext_wholesale_cost     DECIMAL(7,2),
    cs_ext_list_price         DECIMAL(7,2),
    cs_ext_tax                DECIMAL(7,2),
    cs_coupon_amt             DECIMAL(7,2),
    cs_ext_ship_cost          DECIMAL(7,2),
    cs_net_paid               DECIMAL(7,2),
    cs_net_paid_inc_tax       DECIMAL(7,2),
    cs_net_paid_inc_ship      DECIMAL(7,2),
    cs_net_paid_inc_ship_tax  DECIMAL(7,2),
    cs_net_profit             DECIMAL(7,2)
) USING iceberg
PARTITIONED BY (bucket(64, cs_item_sk))
TBLPROPERTIES (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy'
);

CREATE TABLE IF NOT EXISTS store_sales (
    ss_sold_date_sk           INT,
    ss_sold_time_sk           INT,
    ss_item_sk                INT        NOT NULL,
    ss_customer_sk            INT,
    ss_cdemo_sk               INT,
    ss_hdemo_sk               INT,
    ss_addr_sk                INT,
    ss_store_sk               INT,
    ss_promo_sk               INT,
    ss_ticket_number          INT        NOT NULL,
    ss_quantity               INT,
    ss_wholesale_cost         DECIMAL(7,2),
    ss_list_price             DECIMAL(7,2),
    ss_sales_price            DECIMAL(7,2),
    ss_ext_discount_amt       DECIMAL(7,2),
    ss_ext_sales_price        DECIMAL(7,2),
    ss_ext_wholesale_cost     DECIMAL(7,2),
    ss_ext_list_price         DECIMAL(7,2),
    ss_ext_tax                DECIMAL(7,2),
    ss_coupon_amt             DECIMAL(7,2),
    ss_net_paid               DECIMAL(7,2),
    ss_net_paid_inc_tax       DECIMAL(7,2),
    ss_net_profit             DECIMAL(7,2)
) USING iceberg
PARTITIONED BY (bucket(64, ss_item_sk))
TBLPROPERTIES (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy'
);

文件内排序(WRITE LOCALLY ORDERED BY)

以下语句使用Spark的写入排序语法,为各表指定文件内排序键。

ALTER TABLE customer_address       WRITE LOCALLY ORDERED BY ca_address_sk;
ALTER TABLE customer_demographics  WRITE LOCALLY ORDERED BY cd_demo_sk;
ALTER TABLE date_dim               WRITE LOCALLY ORDERED BY d_date_sk;
ALTER TABLE warehouse              WRITE LOCALLY ORDERED BY w_warehouse_sk;
ALTER TABLE ship_mode              WRITE LOCALLY ORDERED BY sm_ship_mode_sk;
ALTER TABLE time_dim               WRITE LOCALLY ORDERED BY t_time_sk;
ALTER TABLE reason                 WRITE LOCALLY ORDERED BY r_reason_sk;
ALTER TABLE income_band            WRITE LOCALLY ORDERED BY ib_income_band_sk;
ALTER TABLE item                   WRITE LOCALLY ORDERED BY i_item_sk;
ALTER TABLE store                  WRITE LOCALLY ORDERED BY s_store_sk;
ALTER TABLE call_center            WRITE LOCALLY ORDERED BY cc_call_center_sk;
ALTER TABLE customer               WRITE LOCALLY ORDERED BY c_customer_sk;
ALTER TABLE web_site               WRITE LOCALLY ORDERED BY web_site_sk;
ALTER TABLE household_demographics WRITE LOCALLY ORDERED BY hd_demo_sk;
ALTER TABLE web_page               WRITE LOCALLY ORDERED BY wp_web_page_sk;
ALTER TABLE promotion              WRITE LOCALLY ORDERED BY p_promo_sk;
ALTER TABLE catalog_page           WRITE LOCALLY ORDERED BY cp_catalog_page_sk;
ALTER TABLE inventory              WRITE LOCALLY ORDERED BY inv_date_sk, inv_item_sk, inv_warehouse_sk;
ALTER TABLE store_returns          WRITE LOCALLY ORDERED BY sr_returned_date_sk, sr_item_sk, sr_ticket_number;
ALTER TABLE catalog_returns        WRITE LOCALLY ORDERED BY cr_returned_date_sk, cr_item_sk, cr_order_number;
ALTER TABLE web_returns            WRITE LOCALLY ORDERED BY wr_returned_date_sk, wr_item_sk, wr_order_number;
ALTER TABLE web_sales              WRITE LOCALLY ORDERED BY ws_sold_date_sk, ws_item_sk, ws_order_number;
ALTER TABLE catalog_sales          WRITE LOCALLY ORDERED BY cs_sold_date_sk, cs_item_sk, cs_order_number;
ALTER TABLE store_sales            WRITE LOCALLY ORDERED BY ss_sold_date_sk, ss_item_sk, ss_ticket_number;