Use the analytic-search plug-in-Elasticsearch(ES)-阿里云帮助中心

The analytic-search plug-in, developed by the Alibaba Cloud Elasticsearch team, is a plug-in for log search and analysis. It provides two features: Kibana Discover query acceleration and concurrent query.

Usage notes

The analytic-search plug-in is a pre-installed system plug-in that cannot be uninstalled. You can view it on the plug-in configuration page. This plug-in requires your Alibaba Cloud Elasticsearch cluster to be version 7.10.0 or later with kernel version 1.7.0 or later.

Kibana Discover query acceleration

This feature reduces query latency for queries with no or a single condition by optimizing the index merge strategy and the Date_histogram execution plan. It is ideal for log retrieval in Kibana Discover.

Performance benchmarks

Test environment: 10 nodes, each with 16 vCPUs and 64 GB of memory. The dataset contained 60 billion business log documents per day, distributed across 12 indexes with 60 shards each.

Query condition	Standard SSD	Ultra disk	OpenStore
No conditions	Latency reduced by 96%	Latency reduced by 95%	Latency reduced by 94%
Single condition	Latency reduced by 88%	Latency reduced by 77%	Latency reduced by 85%
Multiple conditions	Latency reduced by 8%	Latency reduced by 11%	Latency reduced by 14%

Enable Kibana Discover query acceleration

To enable this feature, configure the index.sort settings for your index to sort by the timestamp field in descending order. The following example uses @timestamp as the field name; adjust it and the sort order to match your data.

PUT test_log
{
  "settings": {
    "index.points.same_sort_order_as_index_sort": true,
    "index.sort.field": [
      "@timestamp"
    ],
    "index.sort.order": [
      "desc"
    ]
  },
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      }
    }
  }
}

Concurrent query

This feature runs the query recall phase in parallel to improve resource utilization and reduce its average latency by 50%. It is best suited for scenarios with low QPS, high recall latency, and sufficient node compute resources.

Performance benchmarks

Test environment: Three OpenStore warm-hot shared computing nodes, each with 16 vCPUs and 64 GB of memory. The dataset was a single 1.6 TB index of business log data containing 6 billion documents across 60 shards. The query consists of 3 TermQuery(and) + TimeRange + Sort + Datehistogram, with 10 million hits per shard (a 10% hit rate).

Test results:

Single-shard query latency reduced by 65%.
Multi-shard query latency reduced by 53%.

Enable concurrent query

Run the following command to enable the concurrent query feature:

PUT _cluster/settings
{
  "persistent": {
    "apack.analytic_search.doc_concurrency.enabled": "true"
  }
}

Once enabled, new queries use the default concurrency policy. You can tune this behavior with the following parameters.

Cluster-level settings

Parameter	Default	Description
apack.analytic_search.doc_concurrency.enabled	false	Enables or disables the concurrent query feature.
apack.analytic_search.doc_concurrency.concurrent.policy	80%:4;90%:2	The concurrency policy. The format is `threshold1:concurrency1;threshold2:concurrency2`. For example, `80%:4;90%:2` means the node uses a concurrency of 4 when CPU utilization is less than 80%, a concurrency of 2 when CPU utilization is less than 90%, and 1 (disabled) at or above 90%.
apack.analytic_search.doc_concurrency.min_support_doc	10000	Disables concurrent query for indexes containing fewer documents than this value.
apack.analytic_search.doc_concurrency.min_support_processors	4	Disables concurrent query on nodes with fewer vCPUs than this value.
apack.analytic_search.doc_concurrency.max_support_heap_usage	80%	Disables concurrent query on nodes where JVM heap usage exceeds this percentage.
apack.analytic_search.doc_concurrency.max_support_cpu_usage	90	Disables concurrent query on nodes where CPU utilization exceeds this percentage.

Index-level settings

Parameter	Default	Description
index.apack.analytic_search.doc_concurrency.enabled	true	Enables or disables concurrent query for the index.