The ARMS Go agent adds observable monitoring capabilities at compile time to manage application performance. This process inevitably introduces some performance overhead. The ARMS team uses multiple optimization techniques to reduce the agent's overhead and ensure stable application operation. This test report simulates a real-world scenario to test the performance overhead of the ARMS Go agent under different service traffic loads. Use this report to fully evaluate the performance impact before you integrate ARMS Go Application Monitoring.
Test scenario
The overall architecture is shown in the following figure:
The Go application is built on the Net/HTTP framework. It accesses MySQL and Redis services based on different requests from the stress testing source. Each request type accounts for 50% of the queries per second (QPS).
Test environment
The stress testing source is provided by Alibaba Cloud Performance Testing Service (PTS).
The Go application, MySQL, and Redis are deployed in the same Container Service for Kubernetes (ACK) cluster. The node instance type is ecs.c6.2xlarge. The operating system of the node is Alibaba Cloud Linux 3.2104 LTS 64-bit.
The Go application pod has 1 core, 2 GB of memory, and two replicas.
The Go agent version is Aliyun GoAgent 1.0.0.
Test procedure
Run three stress tests at 500, 1000, and 2000 QPS without the ARMS Go agent installed. Each test lasts for 1 hour. Before each test, warm up the Go application for 3 minutes with traffic at 100 QPS. The results are used as the baseline performance metrics.
Install the Go agent and set the sampling policy to a fixed rate of 10%. Repeat the stress tests from step 1. Compare the differences in CPU overhead, memory overhead, and response time (RT) for the Go application.
Install the Go agent and set the sampling policy to a fixed rate of 100%. Repeat the stress tests from step 1. Compare the differences in CPU overhead, memory overhead, and RT for the Go application.
All basic features of ARMS Go Application Monitoring are enabled, such as statistical metrics, call stacks, and Runtime. All plugin features are also enabled.
If performance is a concern, you can disable the Runtime collection switch in the application configuration.
Baseline performance metrics
Comparison Criteria | CPU | Memory | RT |
500 QPS | 2.42% | 0.71% | 30 ms |
1000 QPS | 4.21% | 0.91% | 30 ms |
2000 QPS | 8.5% | 1.41% | 30 ms |
The CPU metric is the percentage of the total CPU used by the pod.
The Memory metric is the percentage of the total memory used by the pod. Because pod memory usage naturally increases until it reaches the requested value, this report uses the actual memory usage at the end of the stress test.
The RT metric is the average response time of requests, measured in milliseconds (ms).
Performance metrics after installing the ARMS Go agent
Metric | 10% sample rate | 100% sample rate | ||||
CPU | Memory | RT | CPU | Memory | RT | |
500 QPS | 5.15% | 1.25% | 30 ms | 5.25% | 1.85% | 31 ms |
1000 QPS | 8.42% | 1.52% | 31 ms | 10.48% | 2.02% | 32 ms |
2000 QPS | 16.2% | 2.5% | 31 ms | 18.45% | 2.63% | 32 ms |
Probe performance overhead
Metric | 10% sample rate | 100% sample rate | ||||
CPU | Memory | RT | CPU | Memory | RT | |
500 QPS | +2.73% | +0.54% | - | +2.83% | +1.14% | +1 ms |
1000 QPS | +4.21% | +0.61% | +1 ms | +6.27% | +1.11% | +2 ms |
2000 QPS | +7.7% | +1.09% | +1 ms | +9.95% | +1.22% | +2 ms |
Analysis and conclusion
The additional CPU overhead from the ARMS Go agent is within 10%, and memory usage shows almost no increase.
The ARMS Go agent has a very small impact on RT, adding only 1 ms at 2000 QPS.
At a 100% fixed sample rate, the performance overhead is about 2% higher than at the 10% fixed sample rate.