Canary switchover for clusters

Canary switchover

1. On the management page for your Recall Engine Edition instance, click O&M Center > Deployment Management in the navigation pane on the left.

The Deployment Management page displays the deployment topology. This topology includes components such as Query Access, clusters (for example, hz_pre_vpc_domain_1 and hz_pre_vpc_domain_2), the index table, and the data source, along with their data flow relationships.

2. Click a cluster name to start a canary switchover.

After you click a cluster name, select the Cluster Mount tab in the Cluster Configurations panel on the right.

3. After the operation, you can check its status on the Change History page.

On this page, the Change Process column shows the type of change, and the Status column uses a green checkmark or a red cross to indicate success or failure. Click a record to expand and view detailed information about the change nodes.

A cluster can be in one of the following three mount statuses:

Mount status	Traffic status	Actions
Unmounted	None	Mount
Mounted	online traffic	Unmount, Switch to Test Traffic
Mounted	test traffic	Unmount, Switch to Online Traffic

Use cases

Failover between two data centers

1. This example uses a deployment topology with two data centers: domain_1 and domain_2.

In the navigation pane on the left, go to O&M Center > Deployment Management to view the deployment topology. The topology shows cluster: hz_pre_vpc_domain_1 (the domain_1 data center) and cluster: hz_pre_vpc_domain_2 (the domain_2 data center). Each cluster contains data nodes and query nodes. The Query Access node is at the top, and the index table and data source are at the bottom. 2. If a failure occurs in the domain_1 data center, you can unmount it. Traffic then automatically routes to the domain_2 data center.

On the Deployment Management page, select the target cluster (for example, hz_pre_vpc_domain_1) in the deployment topology. In the Cluster Configurations panel on the right, click the Cluster Mount tab, and then click Unmount.

3. Ensure that the domain_2 data center can handle the traffic from both data centers. Before the failover, you can scale out the domain_2 data center by increasing the number of data replicas for its query nodes.

Select the query nodes of the cluster. In the Query Node Configuration panel on the right, click the Online Configuration tab. Here, you can view settings such as Number of data replicas (current value: 1) and Minimum service ratio (current value: 50). Click Edit in the upper-right corner to modify these settings.

New data center warm-up

Assume you have an existing data center, domain_1, that is already handling online traffic. When you deploy a new data center, perform the following steps to ensure the stability of your online query traffic:

In the navigation pane on the left, go to O&M Center > Deployment Management to view the current instance's deployment topology. The topology displays the Query Access node, the cluster (for example, hz_pre_vpc_domain_1, which contains data nodes and query nodes), the index table, and the data source. The data flows from the data source to the index table, then to the cluster, and finally to the Query Access node.

1. Mount the new data center and assign it to handle test traffic. This warms up the service by sending it a copy of online traffic.

On the Deployment Management page, click the target cluster (for example, hz_pre_vpc_domain_2) in the deployment topology. In the Cluster Configurations panel that appears on the right, click the Cluster Mount tab to view the current mount and traffic status.

2. Wait for the performance metrics of the new data center, such as query latency, to match those of the active domain_1 data center.

3. Switch the traffic for the domain_2 data center from test traffic to online traffic.

In the Actions column of the Cluster Mount tab, click Switch to Online Traffic.