Elasticsearch Insights

Data integration

Select the Elasticsearch instances and the data to collect in the target workspace. After the integration is complete, view the data through the Insights dashboard and Metric Explore.

Integrate from the Elasticsearch console

1. Log on to the Alibaba Cloud Elasticsearch console.

2. In the left-side navigation pane, choose Elasticsearch Insights.

3. When you access this feature for the first time, the feature introduction page is displayed. Confirm the authorization status of the service-linked roles, and then click Enable it now..

   Role name	Role description
   AliyunServiceRoleForCloudMonitor	Service-linked role for CloudMonitor
   AliyunServiceRoleForSLSAudit	Service-linked role for SLS audit

   If a role is not authorized, click Authorize for that role to complete the creation, and then enable the feature.

4. In the Enable dialog box, complete the configuration and click OK.

   Parameter	Description
   CloudMonitor 2.0 workspace	By default, the workspace named default-cms-{account ID}-{region ID} is used, and it is reused if it already exists. You can also select a custom workspace.
   Instance scope - Region	The region where the instances reside.
   Instance scope - Entity scope	Supported options: All Entities, By Resource Group, By Tag, Manual Selection. The page displays the number of matching entities in real time.
   Data integration scope	Enable the integration items listed in the following table as needed.

   Integration item	Purpose	Default status
   Elasticsearch Metrics	Integrates basic monitoring metrics so that you can view the running status of clusters and nodes.	Enabled
   Elasticsearch Advanced Monitoring Metrics	Integrates advanced metrics across the cluster, node, index, and throttling dimensions.	Enabled
   ElasticSearch Main Log	Integrates the main log into Simple Log Service.	Not enabled
   ElasticSearch Searching Slow Log	Integrates search slow logs into Simple Log Service.	Not enabled
   ElasticSearch Indexing Slow Log	Integrates indexing slow logs into Simple Log Service.	Not enabled
   ElasticSearch GC Log	Integrates GC logs into Simple Log Service.	Not enabled
   ElasticSearch Access Log	Integrates access logs into Simple Log Service.	Not enabled

   Metric integration items support further configuration. For basic metrics, you can configure Tag Key and Enable All Tags. For advanced metrics, you can configure Metric Collection Scope (all metrics or Select by Group, where the groups include cluster metrics, node metrics, index metrics, and throttling metrics), and you can add custom metrics to collect.

5. Wait for the activation to complete. The activation process performs the following steps in sequence: activating the service, initializing dependent resources, and creating the integration policy. After the process is complete, you are automatically redirected to the Elasticsearch Insights application. After activation, monitoring data takes approximately 10 minutes to be collected and displayed. During this period, the dashboard and Metric Explore may show no data, which is expected.

Integrate from the CloudMonitor 2.0 console

1. Log on to the CloudMonitor 2.0 console, and then go to the target workspace.

2. In the left-side navigation pane, click Integration Center, search for Elasticsearch, and then select Alibaba Cloud Service under the Alibaba Cloud Elasticsearch.

3. Select an integration policy. If a policy already exists, select it. If no policy exists, click Create, and then create one by following the on-screen instructions.

4. Set the instance scope and integration configuration to complete the integration.

Modify the integration configuration

1. Log on to the Alibaba Cloud Elasticsearch console.

2. In the left-side navigation pane, choose Elasticsearch Insights.

3. In the Elasticsearch Insights application, choose the InsightSettings tab.

4. In the Integration Settings section, modify the switch status of each integration item, and then click Save. The switches take effect after you save them.

5. To modify detailed settings such as the collection scope, click Settings for the corresponding integration item to go to the CloudMonitor 2.0 Integration Management page and make the changes.

After you enable this feature, Simple Log Service (SLS), CloudMonitor Enterprise Edition, and Managed Service for Prometheus are automatically activated. Writing metrics to Prometheus incurs fees, and writing logs to SLS incurs fees.

Disable Elasticsearch Insights

1. In the InsightSettings tab, go to the Close Insight section, and then click Close Insight.

2. In the confirmation dialog box, select I understand the scope of disabling and the billing impact, and then click OK.

After the feature is disabled, Insights data collection stops, but the current cloud services are not stopped. Data that has already been collected is retained according to the original storage policy. You can enable the feature again if you need to use it.

Application overview

In the Elasticsearch Insights application, select the corresponding feature based on your troubleshooting goal:

Feature	Entry	Use case
Resource Center	Resource Center > Elasticsearch Instance List	Find the target instance and view instance information and related observability data.
Insights	Insights > Elasticsearch Business Monitoring Dashboard	Troubleshoot exceptions from aspects such as cluster status, query and write performance, resource usage, and JVM.
Alert Management	Alert Management > AlertRule / AlertHistory	Configure alert conditions and notifications, and view historical alerts.
Data exploration	Metric Explore > Elasticsearch Basic Metrics / Elasticsearch High-Level Metrics	Filter data by metric dimension and use Metric Analysis or Metric Query for in-depth troubleshooting.
InsightSettings	InsightSettings	Modify integration switches, view data sources, and disable Insights.

Resource Center

Find Elasticsearch instances

1. In the Elasticsearch Insights navigation pane, choose Resource Center > Elasticsearch Instance List.

2. Set the time range at the top of the page, and then click Filter to set conditions, or enter an instance keyword and then click Search. The keyword must contain at least 4 characters.

3. View the following information in the instance list.

   Field	Description
   Instance name	The name of the Elasticsearch instance.
   Instance ID	Identifies the target instance.
   Region	The region where the instances reside.
   Domain name	The domain name used to access the instance.
   Port	The port used to access the instance.

View data for a single instance

Click an instance name to open the instance details drawer, view the monitoring dashboard of the instance, and continue the analysis through entries such as Entity Details, Associated Topology, Health, Metric Explore and Associated Items, and so on. Observability concepts such as Entity Details, Associated Topology, and Health are implemented based on CloudMonitor 2.0. For more information, see Product introduction of CloudMonitor 2.0.

The content displayed by the log and metric entries depends on the data that has been integrated and associated with the instance. When you investigate a single-instance issue, confirm the instance, region, and time range on the details page before you run a query.

Insights

View the Elasticsearch Business Monitoring Dashboard

1. In the Elasticsearch Insights navigation pane, choose Insights > Elasticsearch Business Monitoring Dashboard.

2. Set the time range and the following filter conditions, and then click Search.

   Filter	Description
   region_id	The region where the target instance resides.
   cluster_id	The Elasticsearch cluster to analyze.
   uid	The corresponding Alibaba Cloud account.
   source_node_host	Filter nodes. The initial value on the page is .*. When you troubleshoot a single node, filter by the target node information.

3. View monitoring charts across the following dimensions.

   Monitoring dimension	Main content	Troubleshooting use
   Cluster status	Cluster status, snapshot status, total number of nodes, number of disconnected nodes, number of indexes, number of shards, and number of primary shards	Check the cluster status and changes in nodes, indexes, and shards.
   Cluster performance	Query QPS, average search time, number of slow queries, write QPS, average write time, and number of HTTP requests with server errors	Compare with the time when the business exception occurred to determine whether query and write performance and error requests changed.
   Resource usage	Node CPU utilization, heap memory utilization, and disk usage	Find nodes with high resource pressure.
   JVM	Node Old GC frequency and Old GC duration	Analyze the temporal correlation between garbage collection activity and performance changes in combination with heap memory utilization.
   Thread pool	The number of tasks waiting in the query and write thread pools and the number of rejected requests or tasks	Check request queuing and rejection.

4. To further analyze a chart, use the Drill-down Analysis entry on the chart, or go to Metric Explore to view the corresponding data. To keep the page, use Save Page to Dashboard.

Analyze by exception symptom

Exception symptom	Recommended order
Increased query latency	Average search time and query QPS → number of slow queries → query thread pool waiting and rejection → node CPU, heap memory, and Old GC
Slowed or blocked writes	Write QPS and average time → write thread pool waiting and rejection → node CPU and disk usage
Abnormal cluster status	Cluster status → number of disconnected nodes → number of shards and primary shards → resource metrics of the related nodes and alert records

The preceding order is used to narrow the troubleshooting scope. An increase in a single metric cannot directly determine the root cause. You must make the judgment in combination with other metrics, alerts, and integrated logs from the same time period.

Alert Management

Configure alert rules

1. In the Elasticsearch Insights navigation pane, choose Alert Management > AlertRule.

2. Click Create Alert Rule, and then in the Alert Configuration step, set the following parameters.

   Parameter	Description
   Rule name	Use a name that makes it easy to identify the business and monitoring target.
   Alert object	The current application corresponds to Alibaba Cloud Elasticsearch
   Observation scope and entity filter conditions	Limits the instance scope to which the rule applies.
   Query metrics and detection conditions	Select the metric to monitor from the drop-down list, and then set the comparison condition and threshold. Add label filter conditions if needed.
   Duration	Set the alert to trigger immediately or after the condition is met continuously for a specified period.
   Alert detection period	Set the frequency at which the alert condition is checked.
   Severity	Select P1 Critical, P2 Error, P3 Warning, or P4 Info based on the business impact.

3. Click Next to go to the Notification Configuration step, and then configure the notifications by following the on-screen instructions to complete the rule creation.

4. Return to the alert rule list and verify the alert conditions, alert notifications, and enabled or disabled status of the rule.

We recommend that you set alert conditions such as query latency, write performance, and node resource usage based on your business baseline, and confirm that the rule is in the expected enabled state after creation.

View alert history

1. Select Alert Management > AlertHistory.

2. Set the time range and filter by rule name, alert status, or severity.

3. View the trigger time, alert content, associated resources, rule name or ID, duration, and number of events for each alert.

4. Return to the target instance based on the associated resources, adjust the dashboard time range to cover the period before and after the alert occurred, and analyze the issue by comparing the metric changes.

Data exploration

Analyze basic metrics

1. Select Metric Explore > Elasticsearch Basic Metrics.

2. Based on the analysis object, select Elasticsearch Cluster Dimension Metrics or Elasticsearch Cluster and Node Dimension Metrics.

3. In the Metric Analysis, filter by the metrics, labels, and time range provided on the page to view the related trends.

4. To run a custom query, switch to Metric Query, edit the query expression, and then click Execute Query. Query results can be displayed as a line chart or a table.

At the cluster dimension, you can view metrics such as cluster status, the number of nodes and shards, and query or write QPS. After you identify a specific node, use the cluster and node dimension metrics to further narrow the scope.

Analyze advanced metrics

Select Metric Explore > Elasticsearch High-Level Metrics, and then switch the metric category based on your analysis goal.

Metric category	Use case
Elasticsearch High-Level Metrics - Cluster Dimension Metrics	View the overall running status and service performance of the cluster.
Elasticsearch High-Level Metrics - Cluster and Index Dimension Metrics	Analyze the read and write performance, merge, refresh, and cache status of different indexes.
Elasticsearch High-Level Metrics - Cluster and Throttling Dimension Metrics	Analyze metrics such as throttler ingress, allowed requests, and rejected requests.
Elasticsearch High-Level Metrics - Cluster and Node Dimension Metrics	Analyze node resources, JVM, thread pools, and storage.

Advanced metrics require the Elasticsearch Advanced Monitoring Metrics integration item to be enabled. If the collection scope is configured as Select by Group, confirm that the group to which the target metric belongs is selected.

View integrated logs

After you enable the integration of the main log, Searching Slow Log, Indexing Slow Log, GC Log, or Access Log, the log data is written to a Logstore in Simple Log Service (SLS). In the InsightSettings tab, go to the Data Source section, and then click the name of the corresponding log data source to go to the Simple Log Service console and query the log data.

When you query data, first select the corresponding Logstore and the time range of the exception, and then narrow the scope based on the fields or keywords that actually exist in the logs. When you analyze slow queries, check the Searching Slow Log first. When you analyze write issues, check the Indexing Slow Log. When you analyze garbage collection issues, check the GC Log.

View data source storage locations

In the InsightSettings tab, go to the Data Source section to view the storage location of each integrated data type. Log data is stored in an SLS Logstore, and metric data is stored in a Prometheus instance. Click a data source name to go to the corresponding CloudMonitor 2.0 or Simple Log Service page and view the storage details.

Billing

When you enable Elasticsearch Insights, Simple Log Service (SLS), CloudMonitor Enterprise Edition, and Managed Service for Prometheus are automatically activated, which incurs the following fees:

• Metrics are written to an Alibaba Cloud Managed Service for Prometheus instance, which incurs CloudMonitor Enterprise Edition data export fees (excluding advanced metrics) and Prometheus metric write fees.

• Logs are written to an SLS Logstore, which incurs Simple Log Service fees.

For more information about the billing rules, see Billing overview of CloudMonitor 2.0 and Billing overview of Simple Log Service. Select the integration scope based on the metric groups and log types that you need to collect.

FAQ

Why can I see an instance in the instance list but no data on the dashboard?

Check the instance information and monitoring data separately:

1. In the InsightSettings tab, confirm that the region and entity scope of the target instance are included in the integration scope and that the corresponding metric integration is enabled.

2. Confirm that the region_id, cluster_id, uid, and node filter conditions of the dashboard match the target instance.

3. Extend the time range to include the data collection period, and then view the Metric Explore results.

4. If you have completed the integration but no data is available yet, first check the integration status and collection scope to avoid creating duplicate integrations.

Why are basic metrics available but advanced metrics are not?

Check whether Elasticsearch Advanced Monitoring Metrics is enabled. If the collection scope is set to Select by Group, confirm that the group to which the target metric belongs is selected, and check the query time range.

Why can I not query the integrated log data?

Check whether the integration of the corresponding log type is enabled, whether logs were generated during the target period, and whether the log data source is correct. You can confirm the SLS Logstore of the log data source in the Data Source section of the InsightSettings tab, and then query it in the Simple Log Service console to verify log collection and integration configuration separately. A certain delay exists between log collection and query availability, so no data may be available immediately after you enable the integration.

Is the collected data deleted after I disable Insights?

No. Disabling only stops data collection. Data that has already been collected is retained in Prometheus and SLS according to the original storage policy, and storage fees may still be incurred. To stop the billing, clear the storage in the corresponding service.