Engine metrics
The advanced monitoring and alert service provides a wide range of metrics for your Elasticsearch cluster. Engine metrics are custom metrics developed by Alibaba Cloud engineers based on extensive Operations and Maintenance (O&M) experience. These metrics cover areas such as cluster status, queries, writes, and cache. They help you troubleshoot performance and stability issues with the Elasticsearch engine. With the advanced monitoring and alert service, you can view a dashboard of engine metrics. You can also create custom alert rules to monitor cluster performance in real time and receive alert notifications. This topic describes each engine metric for Alibaba Cloud Elasticsearch.
Limits
The supported advanced monitoring metrics vary based on the version of your Alibaba Cloud Elasticsearch instance.
Only instances with the performance-enhanced kernel support advanced monitoring metrics related to index write and query QPS (Queries Per Second).
Version 6.7 instances with shared elastic storage enabled do not support metrics related to disk usage. For a list of supported metrics, see the console.
Overview
The advanced monitoring and alert service provides the following categories of engine metrics:
Description of labels (table headers)
Metric: The configurable engine metrics for advanced monitoring and alerts. When you create an alert rule, you must specify a metric. You can copy the metric and paste it into the search box, and the system automatically finds the corresponding metric. For more information, see Configure an alert rule.
Metric description: The description of the metric as displayed in the console.
Description: A detailed explanation of the metric.
Tags: The property tags supported by each metric when you create an alert rule.
NoteDifferent metrics support tags with different granularities. You can configure tags to further filter metric data.
The following tags provide a finer granularity than the common tag properties, such as instanceId and ip. Tags that are not mentioned, such as hostname, kmon_tenant_name, and kmon_service_name, are not monitored by Elasticsearch.
Aggregation operators:
Metric aggregation: The aggregation method used for metric values within the selected tags.
Sampling aggregation: The aggregation method used for data within a sampling period.
overview
Metric | Metric description | Description | Tags | Aggregation operators |
elasticsearch-server.search_total | End-to-end query QPS | The number of end-to-end queries per second. For example, if a client sends two index query requests per second, the value of search_total is 2. |
|
|
elasticsearch-server.search_time_in_millis.max | End-to-end query latency (max) | The end-to-end query latency. |
|
|
elasticsearch-server.bulk_total_operations | Bulk request TPS | The number of bulk operations per second at the shard dimension. |
|
|
elasticsearch-server.bulk_total_time_in_millis.max | Bulk request latency (max) | The total time taken by bulk operations at the shard dimension. |
|
|
elasticsearch-server.search_aggregation_total | End-to-end aggregate query QPS | The number of end-to-end aggregate queries per second. For example, if a client sends two aggregate query requests per second, the value of aggregation_total is 2. |
|
|
search
Metric | Metric description | Description | Tags | Aggregation operators |
elasticsearch-server.search_total | End-to-end query QPS (index) | The number of end-to-end queries per second across indexes. For example, if a client sends two index query requests per second, the value of search_total is 2. |
|
|
elasticsearch-server.search_time_in_millis.max | End-to-end query latency (max, index) | The end-to-end query latency across indexes. |
|
|
elasticsearch-server.search_aggregation_total | End-to-end aggregate query QPS (index) | The number of end-to-end aggregate queries per second across indexes. For example, if a client sends two aggregate query requests per second, the value of aggregation_total is 2. |
|
|
elasticsearch-server.search_total | Query QPS (coordinating node) | The number of queries per second on a coordinating node. For example, if a client sends two index query requests per second, the value of search_total on the coordinating node is 2. |
|
|
elasticsearch-server.search_time_in_millis.max | Query latency (max, coordinating node) | The query latency on a coordinating node. |
|
|
elasticsearch-server.search_aggregation_total | Aggregate query QPS (coordinating node) | The number of aggregate queries per second on a coordinating node. For example, if a client sends two aggregate query requests per second, the value of aggregation_total on the coordinating node is 2. |
|
|
elasticsearch-server.allocated_bytes.max | Large object allocation rate for aggregate queries (max, node) | The amount of memory allocated for an aggregate query. |
|
|
elasticsearch-server.query_total | Query phase QPS (node) | The total number of queries executed per second across all shards on a node. This value is related to the number of shards. For example, if an index has five primary shards, a query to that index results in five shard-level queries. |
|
|
elasticsearch-server.query_took_in_millis.max | Query phase latency (max, node) | The shard query phase latency at the node dimension. |
|
|
elasticsearch-server.fetch_total | Fetch phase QPS (node) | The number of queries per second in the shard fetch phase at the node dimension. |
|
|
elasticsearch-server.fetch_took_in_millis.max | Fetch phase latency (max, node) | The total time taken by the shard fetch phase at the node dimension. |
|
|
elasticsearch-server.query_total | Query phase QPS (shard) | The number of shard queries executed per second. This value is related to the number of shards. For example, if the index you query has five primary shards, the number of shard queries executed per second is 5. |
|
|
elasticsearch-server.query_took_in_millis.max | Query phase latency (max, shard) | The latency of the shard query phase at the shard dimension. |
|
|
elasticsearch-server.fetch_total | Fetch phase QPS (shard) | The number of queries per second in the shard fetch phase at the shard dimension. |
|
|
elasticsearch-server.fetch_took_in_millis.max | Fetch phase latency (max, shard) | The total time taken by the shard fetch phase at the shard dimension. |
|
|
Bulk (write)
Metric | Metric description | Description | Tags | Aggregation operators |
elasticsearch-server.bulk_total_operations | Bulk request TPS (index) | The number of bulk operations per second at the index dimension. |
|
|
elasticsearch-server.bulk_total_time_in_millis.max | Index Dimension: Max Bulk Request Latency | The total time taken by bulk operations at the index dimension. |
|
|
elasticsearch-server.bulk_avg_size_in_bytes | Average bulk size (index) | The average size of requests included in a single bulk command at the index dimension. |
|
|
elasticsearch-server.bulk_total_operations | Bulk request TPS (node) | The number of bulk operations per second at the node dimension. |
|
|
elasticsearch-server.bulk_total_time_in_millis.max | Bulk request latency (max, node) | The total time taken by bulk operations at the node dimension. |
|
|
elasticsearch-server.bulk_avg_size_in_bytes | Average bulk size (node) | The average size of requests included in a single bulk command at the node dimension. |
|
|
elasticsearch-server.bulk_total_operations | Bulk request TPS (shard) | The number of bulk operations per second at the shard dimension. |
|
|
elasticsearch-server.bulk_total_time_in_millis.max | Bulk request latency (max, shard) | The total time taken by bulk operations at the shard dimension. |
|
|
elasticsearch-server.bulk_avg_size_in_bytes | Average bulk size (shard) | The average size of requests included in a single bulk command at the shard dimension. |
|
|
Indexing Service
Metric | Metric description | Description | Tags | Aggregation operators |
elasticsearch-server.cube.follower_indices_throughput_in_bytes | Indexing Service write traffic | The traffic size of writes to an index when you write time series data to Elasticsearch. |
|
|
elasticsearch-server.cube.follower_indices_store_size_in_bytes | Indexing Service managed data volume | The amount of storage space occupied by an index when you write time series data to Elasticsearch. |
|
|
Openstore
The available metrics vary based on the kernel version. We recommend that you upgrade the minor version of the kernel to the latest version.
Metric | Metric description | Description | Tag | Aggregation operators |
aliyunes.elasticsearch.node.stats.fs.total.available.bytes | Free space on the metadata disk for Openstore hybrid storage. |
Important If the free space on the metadata disk is less than 20 GB (20,000,000,000), create an alert for the metadata disk using the advanced monitoring and alert service. Also, consider scaling out the cluster by adding more nodes. |
|
|
elasticsearch-server.openio.block_store_water_mark | Actual usage watermark of the data disk for Openstore hybrid storage. |
Important If the actual usage watermark of the data disk exceeds 85% for a long time, consider scaling out the cluster by adding more nodes. |
|
|
elasticsearch-server.openio.oss_upload_throughput | Upload bandwidth of shared storage (OpenStore hybrid storage instances). | This metric is supported only by kernel versions 1.10 and later. |
|
|
elasticsearch-server.openio.oss_download_throughput | Download bandwidth of shared storage (OpenStore hybrid storage instances). | This metric is supported only by kernel versions 1.10 and later. |
|
|
cache
Metric | Metric description | Description | Tags | Aggregation operators |
elasticsearch-server.query_cache_shard_hit_total | Query cache hit QPS (index) | The number of queries per second that hit the node cache during a shard query, observed at the index dimension. |
|
|
elasticsearch-server.query_cache_shard_cached_total | QPS of the index dimension query_cache | The number of new queries added to the node cache per second during a shard query, observed at the index dimension. |
|
|
elasticsearch-server.query_cache_shard_miss_total | Query cache miss QPS (index) | The number of queries per second that miss the node cache during a shard query, observed at the index dimension. |
|
|
elasticsearch-server.query_cache_shard_evictions_total | Query cache eviction QPS (index) | The number of queries evicted from the node cache per second during a shard query, observed at the index dimension. For example, when the cache is full, the least recently used query results are evicted to make space for new data. |
|
|
elasticsearch-server.query_cache_shard_hit_total | Query cache hit QPS (node) | The number of queries per second that hit the node cache during a shard query, observed at the node dimension. |
|
|
elasticsearch-server.query_cache_shard_cached_total | Query cache add QPS (node) | The number of new queries added to the node cache per second during a shard query, observed at the node dimension. |
|
|
elasticsearch-server.query_cache_shard_miss_total | Query cache miss QPS (node) | The number of queries per second that miss the node cache during a shard query, observed at the node dimension. |
|
|
elasticsearch-server.query_cache_shard_evictions_total | Query cache eviction QPS (node) | The number of queries evicted from the node cache per second during a shard query, observed at the node dimension. |
|
|
elasticsearch-server.query_cache_shard_cached_size_in_bytes.max | Query cache size (max, node) | The total size of new data added to the node cache during a shard query, observed at the node dimension. |
|
|
elasticsearch-server.query_cache_shard_hit_total | Query cache hit QPS (shard) | The number of queries per second that hit the node cache during a shard query. |
|
|
elasticsearch-server.query_cache_shard_cached_total | Query cache QPS per shard | The number of new queries added to the node cache per second during a shard query. |
|
|
elasticsearch-server.query_cache_shard_miss_total | Query cache miss QPS (shard) | The number of queries per second that miss the node cache during a shard query. |
|
|
elasticsearch-server.query_cache_shard_evictions_total | Query cache eviction QPS (shard) | The number of queries evicted from the node cache per second during a shard query. |
|
|
elasticsearch-server.query_cache_shard_cached_size_in_bytes.max | Query cache size (max, shard) | The total size of new data added to the cache during a shard query. |
|
|
Refresh (visibility)
Metric | Metric description | Description | Tags | Aggregation operators |
elasticsearch-server.refresh_total | Refresh QPS (index) | The number of queries per second for index refreshes. |
|
|
elasticsearch-server.refresh_interval_in_millis.max | Refresh interval (max, index) | The interval between refresh operations on an index. |
|
|
elasticsearch-server.refresh_took_in_millis.max | Refresh time (max, index) | The time taken by a refresh operation on an index. |
|
|
elasticsearch-server.refresh_total | Refresh QPS (node) | The number of refresh operations per second on a node. |
|
|
elasticsearch-server.refresh_interval_in_millis.max | Refresh interval (max, node) | The interval between refresh operations on a node. |
|
|
elasticsearch-server.refresh_took_in_millis.max | Refresh time (max, node) | The time taken by a refresh operation on a node. |
|
|
elasticsearch-server.refresh_total | Refresh QPS (shard) | The number of refresh operations per second on an index shard. |
|
|
elasticsearch-server.refresh_interval_in_millis.max | Refresh interval (max, shard) | The interval between refresh operations on an index shard. |
|
|
elasticsearch-server.refresh_took_in_millis.max | Refresh time (max, shard) | The time taken by a refresh operation on an index shard. |
|
|
merge
Metric | Metric description | Description | Tags | Aggregation operators |
elasticsearch-server.merge_total | Merge QPS (index) | Queries per second during the index merge phase. |
|
|
elasticsearch-server.merge_took_in_millis.max | Merge time (max, index) | The time taken to merge data for an index. |
|
|
elasticsearch-server.merge_size_in_bytes.max | Merge size (max, index) | The amount of memory occupied after merging data for an index. |
|
|
elasticsearch-server.merge_total | Merge QPS (node) | The number of queries processed per second by each node during the merge phase. |
|
|
elasticsearch-server.merge_took_in_millis.max | Merge time (max, node) | The time taken to merge data on each node. |
|
|
elasticsearch-server.merge_size_in_bytes.max | Merge size (max, node) | The amount of memory occupied after merging data on each node. |
|
|
elasticsearch-server.merge_total | Merge QPS (shard) | The number of queries per second processed by an index shard during the merge phase. |
|
|
elasticsearch-server.merge_took_in_millis.max | Merge time (max, shard) | The time taken to merge data for an index shard. |
|
|
elasticsearch-server.merge_size_in_bytes.max | Merge size (max, shard) | The amount of memory occupied after merging data for an index shard. |
|
|
cluster state
Metric | Metric description | Description | Tags | Aggregation operators |
elasticsearch-server.applied_cluster_state_count | Cluster state local apply QPS | The number of times a non-master node successfully receives and applies a cluster state update from the master node. |
|
|
elasticsearch-server.applied_cluster_state_took_in_millis.max | Cluster state local apply time (max) | The time taken by a non-master node to successfully receive and apply a cluster state update from the master node. |
|
|
elasticsearch-server.publish_time_in_millis.max | Cluster state broadcast time (max) | The time taken to broadcast a cluster state update. |
|
|
elasticsearch-server.failed_cluster_state_count | Cluster state local apply failure QPS | The number of times a non-master node fails to receive and apply a cluster state update from the master node. |
|
|
elasticsearch-server.failed_cluster_state_took_in_millis.max | Cluster state local apply failure time (max) | The time taken for a failed attempt by a non-master node to receive and apply a cluster state update from the master node. |
|
|
elasticsearch-server.task_execution_count | Master task processing QPS | The number of cluster state changes. For example, if operations such as frequent node changes or frequent updates to index mappings and settings occur in the cluster, the master node recalculates the number of cluster state changes. Important A higher number of state changes indicates frequent changes to the cluster or indexes, which may affect cluster stability. |
|
|
elasticsearch-server.task_execution_time_in_millis.max | Master task processing time (max) | The time taken by the master node to get the cluster state. |
|
|
elasticsearch-server.task_wait_time_in_millis.max | Task queue wait time (max) | The time that a task to get the cluster state waits in the master node's task queue. |
|
|
segment replication
Metric | Metric description | Description | Tags | Aggregation operators |
elasticsearch-server.segment_replication.refresh_copy_file_size | Average incremental copy size (index) | When physical replication is used, this is the size of incremental data copied from the primary replica during each refresh operation at the index dimension. |
|
|
elasticsearch-server.segment_replication.refresh_copy_file_size | Average incremental copy size (node) | When physical replication is used, this is the size of incremental data copied from the primary replica during each refresh operation at the node dimension. |
|
|
elasticsearch-server.segment_replication.refresh_copy_file_size | Incremental copy size (shard) | When physical replication is used, this is the size of incremental data copied from the primary replica during each refresh operation at the shard dimension. |
|
|
elasticsearch-server.segment_replication.refresh_latency_time | Incremental copy latency (avg, index) | When physical replication is used, this is the average time taken by each refresh operation at the index dimension. |
|
|
elasticsearch-server.segment_replication.refresh_latency_time | Incremental copy latency (max, index) | When physical replication is used, this is the maximum time taken by each refresh operation at the index dimension. |
|
|
elasticsearch-server.segment_replication.refresh_latency_time | Incremental copy latency (avg, node) | When physical replication is used, this is the average time taken by each refresh operation at the node dimension. |
|
|
elasticsearch-server.segment_replication.refresh_latency_time | Incremental copy latency (avg, shard) | When physical replication is used, this is the average time taken by each refresh operation. |
|
|
elasticsearch-server.segment_replication.refresh_latency_time | Incremental copy latency (max, shard) | When physical replication is used, this is the maximum time taken by each refresh operation at the shard dimension. |
|
|
elasticsearch-server.segment_replication.merge_copy_file_size | Average merge pre-copy size (index) | When physical replication is used, this is the average size of data copied to the replica after each merge phase at the index dimension. |
|
|
elasticsearch-server.segment_replication.merge_copy_file_size | Average merge pre-copy size (node) | When physical replication is used, this is the average size of data copied to the replica after each merge phase at the node dimension. |
|
|
elasticsearch-server.segment_replication.merge_copy_file_size | Merge pre-copy size (shard) | When physical replication is used, this is the average size of data copied to the replica after each merge phase at the shard dimension. |
|
|
elasticsearch-server.segment_replication.merge_latencc | Merge pre-copy latency (avg, index) | When physical replication is used, this is the average time taken to copy data to the replica after each merge phase at the index dimension. |
|
|
elasticsearch-server.segment_replication.merge_latencc | Merge pre-copy latency (max, index) | When physical replication is used, this is the maximum time taken to copy data to the replica after each merge phase at the index dimension. |
|
|
elasticsearch-server.segment_replication.merge_latencc | Merge pre-copy latency (avg, node) | When physical replication is used, this is the average time taken to copy data to the replica after each merge phase at the node dimension. |
|
|
elasticsearch-server.segment_replication.merge_latencc | Merge pre-copy latency (avg, shard) | When physical replication is used, this is the average time taken to copy data to the replica after each merge phase at the shard dimension. |
|
|
elasticsearch-server.segment_replication.merge_latencc | Merge pre-copy latency (max, shard) | When physical replication is used, this is the maximum time taken to copy data to the replica after each merge phase at the shard dimension. |
|
|
elasticsearch-server.segment_replication.replica_checkpoint_gap | Gap between replica and replication offset (index) | When physical replication is used, this is the number of gaps in the replication offset during the index checkpoint phase. |
|
|
elasticsearch-server.segment_replication.replica_checkpoint_gap | Gap between replica and replication offset (node) | When physical replication is used, this is the number of gaps in the replication offset during the node checkpoint phase. |
|
|
elasticsearch-server.segment_replication.replica_checkpoint_gap | Gap between replica and replication offset (shard) | When physical replication is used, this is the number of gaps in the replication offset during the shard checkpoint phase. |
|
|
elasticsearch-server.segment_replication.refresh_count | Incremental copy QPS (shard) | When physical replication is used, this is the QPS of incremental data copies during each refresh operation. |
|
|
elasticsearch-server.segment_replication.merge_error_count | Incremental copy failure QPS (shard) | When physical replication is used, this is the QPS of abnormal data copies. |
|
|
elasticsearch-server.segment_replication.merge_error_count | Merge pre-copy failure QPS (shard) | When physical replication is used, this is the QPS of abnormal data copies during the merge phase. |
|
|
elasticsearch-server.segment_replication.merge_count | Merge pre-copy QPS (shard) | The number of merge operations. |
|
|
elasticsearch-server.segment_replication.checkpoint_gap_count | Gap reporting QPS (shard) | You can use the physical replication feature to count the QPS during the checkpoint interval. |
|
|
elasticsearch-server.segment_replication.primary_checkpoint_gap.max | Gap between primary and replication offset (shard) | When physical replication is used, this is the size of the gap in the replica's checkpoint phase. |
|
|
Isolator (isolation pool)
Metric | Metric description | Description | Tags | Aggregation operators |
elasticsearch-server.isolator_tasks_isolated_total | Index dimension query: isolated_total | The number of queries per second for an index in the slow query isolation pool. |
|
|
elasticsearch-server.isolator_tasks_killed_total | Query cancellation QPS (index) | The number of queries per second for an index that trigger circuit breaking in the slow query isolation pool. |
|
|
elasticsearch-server.isolator_tasks_killed_mem_size_in_bytes.max | index, dimension, query, cancel, mem_size_in_bytes, and max | The amount of memory consumed by queries for an index that trigger circuit breaking in the slow query isolation pool. |
|
|
elasticsearch-server.isolator_tasks_isolated_total | Total isolated queries (node) | The number of queries per second for a node in the slow query isolation pool. |
|
|
elasticsearch-server.isolator_tasks_killed_total | Query cancellation QPS (node) | The number of queries per second for a node that trigger circuit breaking in the slow query isolation pool. |
|
|
elasticsearch-server.isolator_tasks_killed_mem_size_in_bytes.max | node dimension query cancel mem_size_in_bytes max | The amount of memory consumed by queries for a node that trigger circuit breaking in the slow query isolation pool. |
|
|
elasticsearch-server.isolator_tasks_isolated_total | Query `isolated_total` by shard dimension | The number of queries per second for an index shard in the slow query isolation pool. |
|
|
elasticsearch-server.isolator_tasks_killed_total | Query cancellation QPS (shard) | The number of queries per second for an index shard that trigger circuit breaking in the slow query isolation pool. |
|
|
elasticsearch-server.isolator_tasks_killed_mem_size_in_bytes.max | shard dimension query cancel mem_size_in_bytes max | The amount of memory consumed by queries for an index shard that trigger circuit breaking in the slow query isolation pool. |
|
|