Multi-Site High Availability (MSHA) is a commercial product for multi-active disaster recovery. It evolved from Alibaba's e-commerce platform and is a core module of Application High Availability Service (AHAS). MSHA helps you build disaster recovery architectures. It supports the entire lifecycle of the architecture, including publishing, operations and maintenance (O&M), drills, traffic switching, upgrades, and unpublishing. MSHA also manages the end-to-end path of your service traffic, from ingestion and service calls to asynchronous messages and data persistence.
What is multi-active?
Multi-active means that multiple sites concurrently provide services. The main difference from traditional disaster recovery is that all sites in a multi-active setup are active simultaneously. This not only addresses disaster recovery needs but also improves business continuity and expands capacity.
Problems solved by multi-active disaster recovery
Traditional disaster recovery is built on data-level recovery. A common method is to build an identical application system in a secondary data center. When a disaster occurs, the system is expected to resume operations within a specified time to minimize losses. However, real-world disaster recovery systems have several drawbacks:
The disaster recovery center is idle during normal operations. You cannot be sure if a switch to this center will succeed during a crisis.
Because the disaster recovery center is idle, its resources are unused. This leads to significant resource waste.
Because the disaster recovery center is idle, the primary data center operates in a single region. When service volume grows, this model cannot address the resource bottleneck of a single region.
To address the problems of traditional disaster recovery, you can implement multi-active disaster recovery. This approach provides the following architectural advantages:
Minute-level RTO: Recovery time is fast. The average recovery time for internal production systems at Alibaba is under 30 seconds. For external customers, the average is under 1 minute.
Full resource utilization: No resources are idle. Resources across multiple data centers are fully utilized, which prevents waste.
High switch success rate: Multi-active disaster recovery relies on a mature architecture and a visualized O&M platform. This results in a high switch success rate compared to traditional disaster recovery architectures. The success rate for thousands of internal traffic switches at Alibaba is over 99.9% per year.
Precise traffic control: Multi-active disaster recovery uses precise traffic routing to direct specific service traffic to the corresponding data center. This creates contained, end-to-end traffic flows. Enterprises can use this feature to develop capabilities such as full-domain grayscale releases and critical traffic protection.
Evaluation metrics for disaster recovery systems
The main purpose of a disaster recovery system is to prevent service interruptions during a disaster. When a disaster occurs, users are most concerned with the recovery process. The following list is based on the internationally recognized Share 78 standard for reviewing disaster recovery systems. You can use these points as metrics to measure and select a disaster recovery solution. The Share 78 standard defines the following scope for backup and recovery:
The readiness of the disaster recovery plan.
The distance between the application center and the backup center.
How the application center and the backup center are interconnected.
How data is transferred between the two centers.
What is the extent of data loss?
The process for ensuring data is synchronized to the backup center.
You can start backup jobs from the backup center.
Therefore, the design of a disaster recovery system mainly focuses on these user needs. Due to budget constraints, it is difficult to achieve a Level 6 disaster recovery rating with limited funds. The system you design aims to minimize downtime and recover as much data as possible within the given constraints. These are also the metrics for measuring the quality of your disaster recovery system. In the actual design process, we focus on two key metrics: RTO and RPO.
RPO (Recovery Point Objective)
The recovery point objective (RPO) is a time-based metric. It defines the point in time to which the system and data must be restored after a disaster. RPO indicates the maximum amount of data loss that a system can tolerate. The smaller the amount of data loss the system can tolerate, the smaller the RPO value.
RTO (Recovery Time Objective)
The recovery time objective (RTO) is a time-based metric. It defines the required time from when an information system or business function stops to when it must be restored after a disaster. RTO indicates the maximum service downtime that a system can tolerate. The more urgent the service requirements, the smaller the RTO value.
RPO relates to data loss, while RTO relates to service loss. You must determine RTO and RPO values after a risk analysis and business impact analysis based on different business needs. A good disaster recovery system should meet user needs as much as possible. However, the design of a disaster recovery system is often constrained by many factors, such as available technology, current network conditions, user preferences, and business services. Often, the most decisive factor is the cost of building the disaster recovery system.
Method for building a disaster recovery architecture
Based on the disaster recovery system construction model, the process is divided into three phases: analysis, design, and implementation.
The following sections describe each phase:
Analysis phase
After obtaining formal approval from management, secure the necessary personnel and resources. First, collect information about business processes, the technical infrastructure, and disaster types. Then, perform a business impact analysis and a risk analysis to determine the potential impact of interruptions and expected disasters. Use the results of the analysis to determine business criticality levels, recovery time objectives, and acceptable data loss.
Design phase
In this phase, use the results from the analysis and the company's disaster recovery investment plan to create short-term and long-term disaster recovery policies and goals. First, define a preliminary plan. Then, analyze various factors to eliminate unsuitable options from the candidate plans. Submit the remaining viable plans to an evaluation team. After a thorough review, the team selects the most suitable disaster recovery plan. By classifying services, you can set different disaster recovery levels for core and non-core services. We recommend building a multi-active system for critical infrastructure and core systems, such as those with Level 3 protection. For non-core services, you can build a standby disaster recovery system or perform only data disaster recovery.
Implementation phase
Based on the selected disaster recovery plan, integrate the company's resources, define the disaster recovery architecture, and create a detailed disaster recovery plan. Use technology and services to achieve the required disaster recovery goals. Any plan must be continuously tested and revised to meet the evolving needs of the business. Training and testing also help internal personnel understand their roles in the disaster recovery process. This ensures they can execute the recovery flow smoothly when a real disaster occurs. Testing can be divided into two methods: partial validation and full drills. As business needs, new technologies, and internal and external rules change, IT systems also change. To ensure the disaster recovery plan remains effective, you must regularly review and modify it.
Maintaining an effective disaster recovery architecture
Multi-active disaster recovery provides a solution to help businesses handle potential future disaster scenarios. But businesses continue to grow, and architectures evolve. Disaster recovery administration always addresses evolving challenges. Therefore, disaster recovery administration must not only continuously build more advanced disaster recovery technologies but also enhance the synergy between infrastructure, business systems, support tools, production policies, and emergency personnel. Only by continuously keeping capabilities up-to-date can you thrive in a rapidly changing and complex environment.
Disaster recovery drills are a management technique that can help businesses measure disaster recovery capabilities and expose potential risks. Drills can be divided into three types based on their purpose: tabletop exercises, simulation drills, and actual service takeover drills. Service takeover drills are sometimes combined with production fault drills. Although this may have some impact on the online environment, the results are often more realistic. A disaster recovery drill program typically evolves through the following four phases:
Phase one: Expose problems in a controlled way
Identify risk factors that affect availability in the infrastructure and business systems. Determine the specific impact of each risk factor, whether it is self-healing, and whether it is a zero-downtime factor. This phase requires small-scale production experiments to explore and validate.
Phase Two: Ensuring reliable convergence
After the validation in phase one, you can identify effective zero-downtime factors and expose some problems and risks. Next, continue to resolve the known risks and conduct regular drills to maintain readiness. The success rate of the drills is the key metric in this phase.
Phase three: Reliably quantify problems
After phase two, the infrastructure and business systems have achieved a basic level of certainty. Now, you need to focus on the impact of dynamic factors: support tools, production policies, and emergency personnel. This phase can use methods such as attack-and-defense simulations or surprise drills. Gradually establish a measurement system to collect and analyze metrics.
Phase Four: Identifying potential mining challenges
After the accumulation of structured data in phase three, you can use intelligent solutions to discover hidden and potential weaknesses and risks.