The aliyun-codec plug-in, developed by Alibaba Cloud, is an index compression plug-in for Elasticsearch. It compresses underlying row-oriented (source), column-oriented (doc values), and inverted (postings) files. The plug-in also supports the source_reuse_doc_values feature. This makes it ideal for scenarios with high write volumes and high index storage costs, such as logging and time series analysis.
Usage notes
Install the plug-in before using it. For more information, see Install or uninstall built-in plug-ins.
The aliyun-codec plug-in is supported only on Elasticsearch 7.10.0 instances.
The index compression feature requires kernel version
1.5.0or later, and thesource_reuse_doc_valuesfeature requires kernel version1.6.0or later. If your current kernel version does not meet these requirements, you must first upgrade the kernel.
Performance reference
The following performance data for the aliyun-codec plug-in was recorded in a specific test environment and is for reference only.
Test environment: A single index with a size of 1.2 TB and 22 shards. row-oriented, column-oriented, and inverted compression are enabled, all using the zstd compression algorithm. The dataset consists of Alibaba Cloud Elasticsearch production logs.
Index compression: Compared to a cluster without index compression:
Write throughput: Unchanged.
Overall index size: Reduced by 40%.
Latency in I/O-intensive query scenarios: Reduced by 50%.
source_reuse_doc_values: Compared to a cluster without this feature enabled:
Write throughput: Unchanged.
Overall index size: Reduced by up to 40%. The reduction percentage depends on the proportion of fields with the
source_reuse_doc_valuesfeature enabled in the index.Latency in I/O-intensive query scenarios: Depends on factors such as the proportion of fields with the feature enabled and the node disk type.
Use the index compression feature
The index.codec setting is static. You must close the index before modifying this setting and reopen it afterward.
Assume that you have created an index named test. Run the following commands to enable compression for the index:
POST test/_close
PUT test/_settings
{
"index.codec" : "ali"
}
POST test/_openAfter applying this configuration, Elasticsearch uses the zstd algorithm by default to compress the row-oriented (source), column-oriented (doc values), and inverted (postings) files for this index.
You can also specify a compression algorithm for a specific file type. The following example disables compression for inverted files and uses the zstd algorithm for row-oriented and column-oriented files. To disable compression for a file type, set the corresponding parameter to "".
POST test/_close
PUT test/_settings
{
"index.codec":"ali",
"index.doc_value.compression.default":"zstd",
"index.postings.compression":"",
"index.source.compression":"zstd"
}
POST test/_openThe following table describes the index configuration parameters.
Parameter | Description |
index.doc_value.compression.default | * |
index.postings.compression |
|
index.source.compression | Specifies the compression algorithm for row-oriented (source) files. Valid values:
* |
index.postings.pfor.enabled | Specifies whether to enable encoding optimization for inverted index postings. Valid values are |
Use the source_reuse_doc_values feature
The Elasticsearch storage layer stores multiple copies of data in _source, inverted indexes, and doc_values. The source_reuse_doc_values feature reduces the overall index size by removing data from _source that is already present in doc_values.
Enable the source_reuse_doc_values feature
The source_reuse_doc_values feature can only be enabled during index creation and cannot be disabled later.
PUT test
{
"settings": {
"index": {
"ali_codec_service": {
"source_reuse_doc_values": {
"enabled": true
}
}
}
}
}Adjust source_reuse_doc_values settings
After you enable the source_reuse_doc_values feature, you can adjust the following settings.
Adjust the maximum number of fields
If the number of fields with source_reuse_doc_values enabled exceeds the specified value, Elasticsearch either throws an exception or automatically disables the feature. The default value is 50.
PUT _cluster/settings
{
"persistent": {
"apack.ali_codec_service.source_reuse_doc_values.max_fields": 100
}
}Set whether to enforce the maximum field limit
true: Elasticsearch throws an exception if the limit is exceeded.false: Elasticsearch automatically disables thesource_reuse_doc_valuesfeature if the limit is exceeded.
PUT _cluster/settings
{
"persistent": {
"apack.ali_codec_service.source_reuse_doc_values.strict_max_fields": true
}
}Adjust the concurrency for reading field values
When fetching an original document, the source_reuse_doc_values feature concurrently reads and assembles values from fields where this feature is enabled. The default concurrency is 5.
PUT test/_settings
{
"index": {
"ali_codec_service": {
"source_reuse_doc_values": {
"fetch_slice": 2
}
}
}
}Adjust the thread pool and queue size
By default, the thread pool size is the number of node cores, and the default queue size is 1000. You can change these settings only by modifying the YML file. For instructions, see Configure YML parameters.
apack.doc_values_fetch:
size: 8
queue_size: 1000