From unitization to elastic architecture: Handling explosive traffic spikes on the Internet

Diagram of an elastic architecture with unitization and cloud-native

Unitization is an architectural pattern that extends the concept of database sharding, a method for solving performance issues by partitioning data. The core idea of unitization is to shard user requests at the network entry point, not just at the data layer. At the network access layer, requests are sharded based on a dimension, such as user ID. This approach treats each data center as a massive, stateful database shard. For example, when a user sends a request, the access layer routes it to a specific regional data center based on the user's ID. Most processing then occurs within that data center. Some services may require cross-data center invocations, such as a money transfer between users in different data centers. These scenarios require a stateful design.

As we move into the cloud-native era, we design our large-scale architecture based on Kubernetes. In a unitized architecture, we deploy a Kubernetes cluster in each unit. We use a logically global Federated API Server deployment to manage multiple Kubernetes clusters and distribute control instructions. The control metadata is stored in an etcd cluster to maintain global data consistency. However, etcd only provides disaster recovery for two data centers within the same city and cannot ensure consistency across multiple cities and data centers. To address this, we are moving etcd to our OB KV engine. This approach retains the etcd storage format and semantics at the engine layer. The storage layer then achieves high availability across three regions and five data centers.

This architecture works well for Ant Group. However, when we offer this technology to external customers, we face new challenges. For example, customer data centers often have heterogeneous infrastructure. This requires us to implement multicloud adaptation based on Cloud Provider standards.

In addition, many financial institutions, including us, have legacy systems that were not designed with cloud-native principles. Many of these systems have stateful dependencies on the infrastructure, such as a dependency on IP addresses. This makes it difficult to fully adopt an immutable infrastructure model. Sometimes, strict business continuity requirements make it difficult to adopt the operations and maintenance (O&M) patterns of native Kubernetes workloads. For example, when using a native deployment for a canary release, the handling of applications and traffic is disruptive. This can cause business exceptions and service interruptions during O&M changes. To solve this problem, we extended the native deployment into the CAFE Deployment, which is better suited for financial services. This extension allows for more graceful large-scale cluster releases, canary releases, and rollbacks. It also aligns with our "three principles of technical risk."

Therefore, a financial-grade hybrid cloud must first address the challenges of elasticity and heterogeneity. It must also meet the stability requirements for large-scale, financial-grade O&M. After these problems are solved, the financial industry can focus on service innovation. The industry prioritizes stable innovation. This means continuing to support the business with traditional development and O&M models while simultaneously introducing new ones. Both models advance together in this dual-mode approach.