Confluent cluster migration

Manual migration

LDAP migration

1. Log on to the ApsaraMQ for Kafka console. In the navigation pane on the left, click Instances.

2. On the Instances page, click the name of the target instance.

3. On the Instance Details page, click Users in the left-side navigation pane, and then click Add LDAP User.

4. In the Add LDAP User dialog box, set the username and password, confirm the password, and click OK.

5. Log on to the Control Center. For more information, see Log on to the Control Center.

6. In the right-side menu, select > MANAGE ROLE ASSIGNMENTS, click Assignments, and then select the destination cluster.

7. On the cluster details page, click Add role assignment in the upper-right corner to apply the LDAP permissions to the destination cluster.

Data migration

1. In the destination cluster, create the corresponding topics from the source cluster. For more information, see Create a topic.

2. Change the producer's endpoint to the destination cluster's endpoint.

3. Wait for the consumers to finish processing messages on the source cluster. Then, change the consumers' endpoints to point to the destination cluster.

4. Decommission the source cluster.

Schema migration

After creating topics in the destination cluster, create schemas for them.

In the left-side navigation pane of the Confluent Control Center, click Topics. Select the target topic to go to its details page. Go to the Schema tab, select the Value or Key subtab, and then click Set a schema to set a schema for the topic.

Assisted data migration

LDAP migration

Alibaba Cloud assists this migration by providing cURL commands.

1. Log on to the ApsaraMQ for Kafka console and add the LDAP user account and password to the destination cluster. For more information, see Add a user.

2. Alibaba Cloud provides you with a cURL command to retrieve your LDAP permission information.

3. Send the LDAP permission information to Alibaba Cloud technical support.

4. Run the cURL command provided by Alibaba Cloud to apply the permissions to the destination Confluent cluster.

   Example:

   curl -i -X POST \
      -H "Authorization:Basic XXXX==" \
      -H "Content-Type:application/json" \
      -d \
   '{"clusters":{"kafka-cluster":"XXXX"},"path":[]}' \
    'https://mds-XXXX.csp.aliyuncs.com/security/1.0/lookup/managed/rolebindings/principal/User:root'

Data migration with ClusterLink

Confluent ClusterLink losslessly migrates topics between clusters. It runs on Kafka brokers and creates a mirror topic that reads data from the source cluster. After the mirror topic catches up with the source topic, migrate producers and consumers to the destination cluster and promote the mirror topic to complete the migration.

1. Alibaba Cloud technical support creates a ClusterLink task and configures the mirror topic for synchronization. After the migration starts, wait for the mirroring lag to approach zero.

2. Stop the producers for the source topic. After the consumers have processed all remaining data, stop the consumers.

   Note

   If microservices produce and consume messages, you must also stop these microservices when you stop the producers.

3. When the mirroring lag is zero, Alibaba Cloud technical support promotes the mirror topic by setting its status to PROMOTE. A promoted topic becomes a standard, writable topic in the destination cluster.

4. Move all consumers and producers to the destination cluster. After the data migration is complete, Alibaba Cloud technical support updates DNS resolution to redirect the domain name to the destination cluster.

5. Restart the producers and consumers to complete their migration from the source cluster to the destination cluster.

6. Verify that the migrated topics work as expected.

Schema migration with SchemaLink

Provide Alibaba Cloud technical support with the topic names for SchemaLink migration. Alibaba Cloud creates a SchemaLink to synchronize schemas from the source cluster to the destination cluster's Schema Registry. Do not create new schemas or modify the topics during the migration.