System protection controls the inbound traffic of an application from a holistic perspective. It uses monitoring metrics such as application load, overall average response time (RT), inbound queries per second (QPS), and the number of concurrent threads to balance inbound traffic with the system's payload. This allows the system to operate at maximum throughput while ensuring overall stability.
Background information
Traditionally, adaptive system protection has relied on hard metrics, such as system load, to prevent overloads. When the system load exceeds a certain threshold, traffic is blocked or reduced. If the load recovers, traffic is allowed to resume. This approach has two unavoidable problems:
- Adjusting the traffic pass-through rate based on system load introduces latency. If an adjustment to the rate increases the load, the change is not observable for at least one second. Likewise, if an adjustment decreases the load, the system must wait one second before making further adjustments. This method wastes system processing capacity and causes the load curve to appear jagged.
- The pass-through rate recovers slowly. Consider a scenario where a downstream application is unreliable, causing long response times and high system load. If the downstream application recovers, its response time decreases, and the pass-through rate should increase significantly. However, because the system load is still high, the pass-through rate recovers slowly.
To solve these problems, the system protection feature in Application High Availability Service (AHAS) flow control and degradation uses a different approach. It uses the load per minute to initiate traffic control. It then uses request response times and the current request processing rate to determine the volume of allowed traffic. The goal is to increase the system's throughput rate without overwhelming it.
How it works
Imagine the process of a system handling requests as a water pipe. Incoming requests are like water flowing into this pipe. When the system processes requests smoothly, they do not need to be queued. They pass directly through the pipe, and their RT is at its minimum. Conversely, when requests are stacked, the processing time becomes the sum of the queuing time and the minimum processing time.
If T represents the number of concurrent requests in the system (water in the pipe), RT represents the processing time for a request, and QPS represents the number of incoming requests per second, then the number of requests in the system can be expressed as QPS * RT. When T ≈ QPS * Avg(RT), the system's processing capacity and the number of allowed incoming requests are balanced. The system's payload will not increase further. When the inbound traffic rate matches the maximum outbound traffic rate, the system's processing capacity is fully utilized.
System rules
System protection rules apply to the entire application, not to individual resources. They only take effect on inbound traffic. Inbound traffic refers to traffic entering the application (EntryType.IN), such as requests received by a web service or a Dubbo server. System rules support four threshold types:
- Load (only for Linux/Unix-like machines): System protection is triggered only when the system's 1-minute load average exceeds the threshold and the current number of concurrent threads exceeds the system's capacity.
- RT: System protection is triggered when the average RT of all inbound traffic on a single machine reaches the threshold.
- Concurrent threads: System protection is triggered when the number of concurrent threads for all inbound traffic on a single machine reaches the threshold.
- Inbound QPS: System protection is triggered when the QPS of all inbound traffic on a single machine reaches the threshold.
To configure system rules for an application, see Create a system rule.