离线cluster配置

cluster 配置中同时包含在线部分 (ha3) 配置和离线 (build_service) 两部分配置;离线配置中包含了对builder 和merger 的各种配置。

customized_merge_config

customized_merge_config:用户自定义配置。用户可以对不同场景定制不同的merge策略。比如增量阶段使用和缺省策略不同的merge策略;可以配置多个,多个merge策略会同时生效,相同时间如果触发了多个merge策略会按照配置中的顺序先后触发merge plan。 用户除了指定merge_config,还可以配置下面两个参数:

  • period: 索引的merge间隔时间,单位为s。或配置在某个固定时刻merge,比如在下午13pm触发merge,格式是"period": "daytime=13:00"

  • need_wait_alter_field:如果当前有动态加字段流程,是否要阻塞merge,默认是true。如果配成false,当前merge 策略将自动换成align_version merge策略生效。

  • merge_parallel_num: 每个partition参与merge的并发进程数量。

由于新增动态加字段功能。禁止用户自定义的merge策略命名为alter_field。

customized_merge_config默认值如下:

{
    "full":{
        "merge_config":{
            "keep_version_count":40,
            "merge_strategy":"optimize",
            "merge_strategy_param":"after-merge-max-segment-count=20",
            "merge_thread_count":4
        }
    },
    "large_segment_reclaim":{
        "merge_config":{
            "keep_version_count":40,
            "merge_strategy":"priority_queue",
            "merge_strategy_params":{
                "input_limits":"max-segment-size=20480",
                "output_limits":"max-merged-segment-size=13312;max-total-merged-size=15360",
                "strategy_conditions":"priority-feature=delete-doc-count#desc;conflict-segment-count=2;conflict-delete-percent=8"
            },
            "merge_thread_count":4
        },
        "period":"period=1800"
    },
    "segment_merge":{
        "merge_config":{
            "keep_version_count":40,
            "merge_strategy":"priority_queue",
            "merge_strategy_params":{
                "input_limits":"max-segment-size=12288",
                "output_limits":"max-merged-segment-size=13312;max-total-merged-size=15360",
                "strategy_conditions":"priority-feature=valid-doc-count#asc;conflict-segment-count=2;conflict-delete-percent=10"
            },
            "merge_thread_count":4
        },
        "period":"period=900"
    },
    "large_segment_merge":{
        "merge_config":{
            "keep_version_count":40,
            "merge_strategy":"priority_queue",
            "merge_strategy_params":{
                "input_limits":"max-segment-size=12288",
                "output_limits":"max-merged-segment-size=15360;max-total-merged-size=46080",
                "strategy_conditions":"priority-feature=valid-doc-count#desc;conflict-segment-count=2;conflict-delete-percent=10"
            },
            "merge_thread_count":4
        },
        "period":"period=3600"
    },
    "small_segment_merge":{
        "merge_config":{
            "keep_version_count":40,
            "merge_strategy":"priority_queue",
            "merge_strategy_params":{
                "input_limits":"max-segment-size=1536",
                "output_limits":"max-merged-segment-size=1536;max-total-merged-size=3072",
                "strategy_conditions":"priority-feature=valid-doc-count#asc;conflict-segment-count=2;conflict-delete-percent=20"
            },
            "merge_thread_count":4
        }
    }
}

segment_customize_metrics_updater

segment_customize_metrics_updater:用于生成各个segment的segment metric,可由用户自定义生成。indexlib内置存在lifecycle updater,配置该updater后,每个segment会生成配置字段的min,max值,并根据定义的lifecycle_param配置的打标方式,为每个sgement打标(merge阶段,build阶段按照默认tag打标)。打标可用于load策略和time_series_merge策略。默认值:[]。