APM application performance monitoring-阿里云帮助中心

You can use the Alibaba Cloud-hosted Application Performance Monitoring (APM) service with Alibaba Cloud Elasticsearch to build an application performance monitoring system and achieve system observability. This topic describes the concepts, architecture, and features of the APM service for Alibaba Cloud Elasticsearch. It also explains how to quickly integrate and use the APM service.

Background information

The essence of observability is to measure your infrastructure, platform, and applications to understand how they run. Unlike traditional monitoring and O&M, mainstream monitoring now focuses more on detecting and alerting on issues. The ultimate goal of observability is to provide a reasonable explanation for everything that happens in a complex distributed system. Monitoring focuses on the service status during and after software delivery, whereas observability is responsible for the entire R&D and O&M lifecycle.

Logs, infrastructure metrics, and Application Performance Monitoring (APM) are the three pillars of observability. APM bridges the gap between metrics and logs. Logs and metrics often involve infrastructure and components. APM, however, focuses on the application. It allows IT and developer teams to monitor the application layer of their stack, including the end-user experience. Adding APM to your system monitoring lets you:

Understand where your service spends its time and why it crashes.
See how services interact with each other and visualize their performance bottlenecks.
Proactively find and fix performance bottlenecks and errors.
Improve the productivity of your developer teams.
Track the end-user experience in the browser.

APM is commonly used in the following scenarios:

Real User Monitoring: Improve the user experience by monitoring user behavior. For example, you can monitor user interactions with a web interface or client and record the time of each interaction occurrence.
Runtime application architecture: Understand dependencies between services and the network topology of application interactions within the architecture.
Business transaction: Generate meaningful Service-Level Agreement (SLA) reports and provide trend information about application performance from a business perspective.
Deep-dive component monitoring: This usually requires installing an agent and mainly targets the middle layer, which includes web servers, applications, and message servers. Robust monitoring should show a clear path of code execution. Because this dimension is closely related to the second dimension mentioned above, APM products often merge these two dimensions into a single feature.
Analytics/Reporting: Standardize and present the series of performance metrics collected from applications into a common view of application performance data.

For a comparison of the capabilities between the Alibaba Cloud-hosted Elastic APM and the popular open source APM system Apache SkyWalking, see Capability comparison.

Architecture and data model

The application performance monitoring service for Alibaba Cloud Elasticsearch lets you build a hosted APM Server service based on open source Elastic APM. You can connect it to your Alibaba Cloud Elasticsearch service for efficient application performance optimization and monitoring. The overall architecture of the functional modules is shown in the following figure. APM架构

Elastic APM consists of four components. For more information, see Introduction to the application performance monitoring and analysis service. The APM Agent data collector gathers different types of information and data, which are called events, from the applications it monitors. After the data collector gathers the data, it streams these events to the APM Server. The server then validates and processes the events. The supported event types are Spans, Transactions, Errors, and Metrics:

Spans: This event type contains information about a specific code path that was executed. Spans measure from the start to the end of an activity. They can have parent or child relationships with other spans.
Transaction: This event type is a special kind of span. It has no parent and can only have child spans. You can think of it as the root node of a tree data structure. It has extra metadata associated with it. You can consider transactions as the highest level of work measured in your service, such as a request to the service, a served HTTP request, or a specific background job.
Errors: This event type contains information about the original exception or the log message created when an exception occurred.
Metrics: This event type includes basic host-level metrics that the APM Agent automatically collects. These include system-level and process-level CPU and memory metrics. You can also obtain agent-specific metrics, such as JVM metrics from the Java Agent and Go runtime metrics from the Go agent.

Prerequisites

You must create an Alibaba Cloud Elasticsearch V7.10 instance. For more information, see Create an Alibaba Cloud Elasticsearch instance. When you purchase the instance, select the Advanced Edition. This edition meets the requirements for low-cost, high-volume data writes and storage in application performance monitoring scenarios. For more information, see Purchase page parameters (Advanced Edition).

Limits

Alibaba Cloud APM Server can only be integrated with Alibaba Cloud Elasticsearch V7.10.

Procedure

Step 1: Create an APM Server instance
Step 2: Configure the APM Agent
Step 3: Analyze application performance in Kibana

Step 1: Create an APM Server instance

Log on to the Alibaba Cloud Elasticsearch console.
Go to the Application Performance Monitoring (APM) console.
1. In the top navigation bar, select a region.
2. In the left-side navigation pane, clickAPMApplication Performance Monitoring.
On the APM Server Management page, create an APM Server instance as described in Create an APM Server instance.

The virtual private cloud (VPC) must be the same as the VPC that contains the client service (APM Agent) you want to monitor. After the instance is created, click Instance Management to the right of the instance to view basic information, such as the instance endpoint.

Step 2: Configure the APM Agent

On the APM Server Management page, click APM Data Ingestion.
In the Agent Data Ingestion dialog box, click the tab for your client's language and follow the instructions to configure data collection.

After the configuration is complete, the agent program works with your application to collect performance metrics and errors. It then sends all collected data to the server. The following example shows how to configure the agent for Java:
1. Configure your application to access Alibaba Cloud Elasticsearch and add the APM Agent dependency in the pom dependencies. For more information, see High Level REST Client (7.x).
```
<dependency>
  <groupId>co.elastic.apm</groupId>
  <artifactId>elastic-apm-agent</artifactId>
  <version>1.27.0</version>
</dependency>
```
2. Download a compatible agent file for your application from Maven Central.
3. Set the startup parameters and start the application using the javaagent parameter.
```
java -javaagent:/root/elastic-apm-agent-1.27.0.jar \
     -Delastic.apm.service_name=my-application \
     -Delastic.apm.server_urls=http://apm-cn-tl32fxqq****.apm.elasticsearch.aliyuncs.com::8200 \
     -Delastic.apm.secret_token= \
     -Delastic.apm.application_packages=org.example \
     -jar elastic.jar
```
  - apm-agent-java-0.1.2.jar: The name of the agent file that you downloaded in the previous step, including the .jar file extension.
  - Delastic.apm.server_urls: Set this parameter to the endpoint of your APM Server instance. After the instance is created, click Instance Management next to the instance to obtain the endpoint.

Step 3: Analyze application performance in Kibana

Log on to the Kibana console of the Elasticsearch instance that you associated with the APM Server in Step 1: Create an APM Server instance.

For more information about how to log on to the Kibana console, see Log on to the Kibana console.

Note
This topic uses Alibaba Cloud Elasticsearch V7.10 as an example. The operations may differ for other versions. The actual interface may vary.
Go to the Kibana homepage. In the upper-right corner, click Dev tools.
On the Console tab, you can run the following script to enable automatic creation of the APM onboarding index.
Note
- After an APM Server instance is created, its service starts automatically. After the service starts, Elasticsearch automatically creates APM-related indexes, which you can view in the Kibana console. However, the name of the APM onboarding index does not comply with the naming conventions for automatically created indexes in Alibaba Cloud Elasticsearch. Therefore, you must manually enable the automatic creation of the APM onboarding index.
- In the following script, 7.10.2 is the version number of your APM Server instance.
```
PUT _cluster/settings
{
  "persistent": {
    "action.auto_create_index":"+.*,+apm-7.10.2-onboarding-*,-*"
  }
}
```
View all services in APM.
1. In the upper-left corner of the page, click the icon to expand the navigation pane on the left.
2. In the left navigation pane, under Observability, click APM.
3. On the Services tab, you can view all services in APM.
Click a service to view its details.
The following figure shows the details of the petclinic-spring service. Each service has a similar layout.
- The upper-left corner shows the average, p95, and p99 values for response time. This indicates the specific time when an anomaly occurred. You can also hide curves for values that you are not interested in on the chart. This helps you better understand the impact of outliers on the entire service. When you move the mouse pointer over any chart, a pop-up window appears with a summary for that point in time. The chart shows that the spike in response time did not cause any server to return a 500 response error.
- By viewing the request details below, you can see that each request in the application comes from a different node. You can expand the default nodes using various agent APIs. You can sort by column heading, response time, or impact. The impact column considers the latency and popularity data for the corresponding request. Take the getOwners request with the highest impact as an example. Its average latency is not high, at 96 ms. Click the request name to view its details. The slowest request shown in the figure is less than one second.
- Scroll down to see the waterfall view of operations for this request. Many SELECT statements are in progress. With APM, you can see the actual queries that are being executed.
  
  Figure 1. Waterfall view of operations for the request
  
  Figure 2. Actual query being executed
- Because the APM Agent is installed on all multilayer microservices in this application, you can click View full trace. This shows the distributed trace of the request, which starts from the browser layer and includes the details of all components involved in its processing.
  
  Figure 3. Request details
  
  Figure 4. All components in the full trace
- Real User Monitoring: To obtain the most value from distributed tracing, you need to monitor as many components and services as possible. This includes using Real User Monitoring (RUM). A fast service response time does not mean that the request completes quickly in the browser. APM helps businesses measure the end-user experience in the browser. The distributed trace in the following figure shows four different services, including a web browser.

References

Monitoring applications with Elasticsearch and Elastic APM