Performance testing
Performance testing identifies bottlenecks and validates system capacity under high load, enabling informed capacity planning and optimization.
Performance testing simulates a large number of concurrent users accessing a system to evaluate its performance and stability under high load. It uncovers performance bottlenecks, assesses system capacity, and reveals potential risks. The implementation process is as follows:
-
Define the test objectives and load model. This step includes defining the test scenario, load type, load volume, and test duration.
-
Configure the staging environment. Set up the environment with the required hardware models, network configuration, and testing tools. You can use open source tools or existing cloud platforms, such as PTS.
-
Create a detailed test plan and test scripts to ensure repeatability and accuracy. The plan and scripts should include the stress testing scenario, test data, and methods for analyzing the results.
-
Run the test and analyze the data. Execute the test scripts and record the results. Monitor and record metrics such as system response time, throughput, concurrent users, and CPU and memory usage. Then, analyze and evaluate the test data to identify performance bottlenecks and develop optimization plans.
-
Optimize performance and retest. Optimize the system's hardware models, network environment, and applications based on the test results. Then, run the performance test again to confirm the performance improvements.
Performance testing covers both single-component and end-to-end scenarios. It helps identify potential issues before an application goes live and provides data that informs capacity planning and optimization.
Common performance testing tools include Alibaba Cloud PTS, Apache JMeter, ApacheBench (ab), and wrk. The following table compares these tools.
|
Comparison item |
Alibaba Cloud PTS |
Apache JMeter |
ApacheBench |
wrk |
|
|
Cost |
Learning curve |
Low |
Medium |
Low |
Low |
|
Deployment and O&M cost |
SaaS service, no deployment required |
Low for single-machine deployment, high for distributed deployment |
Low |
Low |
|
|
Pricing |
Yes. |
Open source, free |
Open source, free |
Open source, free |
|
|
Distributed capability |
Support for distributed stress testing |
Yes |
Yes, but with high deployment and O&M costs |
No |
No |
|
Stress testing engine capability |
Single-machine performance and stability |
Proprietary engine, high |
Low |
Medium |
High |
|
Support for multiple protocols |
Support |
Support |
No |
No |
|
|
Stress Level |
High. Up to millions of concurrent users and tens of millions of TPS. |
Low |
Low |
Low |
|
|
Stress testing scenario creation |
Support for client-side traffic recording |
Yes. Supports Chrome, iOS, and Android recorders. |
No |
No |
Not supported |
|
Support for flow orchestration |
Supported without coding. |
Support |
No |
No |
|
|
Support for response parameter fetching, assertions, logic controllers, etc. |
Support |
Support |
No |
No |
|
|
Stress test data creation |
Support for file data sources |
Support |
Support |
No |
Not supported |
|
Support for reading data from a DB as a data source |
Support |
No, requires custom implementation |
No |
No |
|
|
Support for using functions to generate or process test data |
Support |
Support |
No |
No |
|
|
Stress test model creation |
Support for concurrent model |
Support |
Support |
Support |
Support |
|
Support for throughput model |
Support |
No |
Support |
Support |
|
|
Support for traffic funnel model |
Support |
No |
No |
No |
|
|
Support for auto-increment and step-increment traffic models |
Support |
Support |
No |
Not supported |
|
|
Stress test traffic creation |
Support for custom traffic from multiple regions |
Support |
No, depends on your own deployment |
No, depends on your own deployment |
No, depends on your own deployment |
|
Support for IPv6 traffic |
Support |
No, depends on your own deployment |
No, depends on your own deployment |
No, depends on your own deployment |
|
|
Stress test traffic control |
Support for manual speed adjustment during testing |
Support |
No |
No |
No |
|
Support for dynamic scaling of max pressure and stress testing engines during a test |
Support |
No |
No |
No |
|
|
Stress test data visualization |
Support for real-time, multi-dimensional metric monitoring during testing |
Real-time, second-level data with multi-dimensional analysis |
Yes, but with limited analysis dimensions |
No |
No |
|
Support for test reports |
Yes, provides complete test reports |
Support is available, but the reports are basic. |
Basic reporting is supported. |
Yes, but reports are basic |
|
|
Support for associating monitoring data from the tested system |
Yes, can associate with Alibaba Cloud Cloud Monitor data |
No |
Not supported |
No |
|
|
Support for performance baseline feature |
Support |
No |
No |
No |
|
For more information, see the Performance Efficiency pillar.