AI 助理
文档备案控制台
官方文档
首页

云上迁移Topic和Group

更新时间：
复制 MD 格式
产品详情
我的收藏

当您想要将A地域的云消息队列 Kafka 版上的业务无损迁移至B地域，或者创建实例后想复制已有实例元数据（Topic、Group）配置时，您可以使用云消息队列 Kafka 版提供的迁移工具将某个云消息队列 Kafka 版实例的Topic、Group迁移至另一个云消息队列 Kafka 版实例中。

前提条件

重要

  • 迁移不会删除源云消息队列 Kafka 版实例的Topic/Group，只是在目标云消息队列 Kafka 版实例创建相同配置的Topic/Group。

  • 迁移内容仅为Topic/Group配置，不包含Topic/Group中存储的数据。

操作步骤

Topic

  1. 打开命令行工具。

  2. 使用cd命令将路径切换到迁移工具所在目录。

  3. 执行以下命令，对要迁移的Topic进行预检。

    java -jar kafka-migration-assessment.jar TopicMigrationFromAliyun  \
--sourceAk <yoursourceAccessKeyId>  \
--sourceSk <yoursourceAccessKeySecret>  \ 
--sourceRegionId <yoursourceRegionId>  \ 
--sourceInstanceId <yoursourceInstanceId>  \ 
--destAk <yourdestAccessKeyId>  \ 
--destSk <yourdestAccessKeySecret>  \ 
--destRegionId <yourdestRegionId>  \ 
--destInstanceId <yourdestInstanceId>

    上文中<>处内容需要替换为您的实例信息，配置规则请参见下表：

    参数

    描述

    sourceAk

    源云消息队列 Kafka 版实例所属阿里云账号的AccessKey ID。

    sourceSk

    源云消息队列 Kafka 版实例所属阿里云账号的AccessKey Secret。

    sourceRegionId

    源云消息队列 Kafka 版实例的地域ID。例如：cn-hangzhou。

    sourceInstanceId

    源云消息队列 Kafka 版实例的ID。

    destAk

    目标云消息队列 Kafka 版实例所属阿里云账号的AccessKey ID。

    destSk

    目标云消息队列 Kafka 版实例所属阿里云账号的AccessKey Secret。

    destRegionId

    目标云消息队列 Kafka 版实例的地域ID。例如：cn-hangzhou。

    destInstanceId

    目标云消息队列 Kafka 版实例的ID。

    返回结果示例如下： 

    [ecs-assist-user@xxx ~]$ java -jar kafka-migration-assessment.jar TopicMigrationFromAliyun --sourceAk xxx --sourceSk xxx --sourceRegionId cn-hangzhou --sourceInstanceId xxx --destAk xxx --destSk xxx --destRegionId cn-hangzhou --destInstanceId xxx
log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAddCookies).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[2024-06-05 10:37:39]- cmd:TopicMigrationFromAliyun, sourceTopicList:{"total":1,"code":200,"requestId":"xxx","success":true,"pageSize":10000,"currentPage":1,"message":"operation success.","topicList":[{"instanceId":"xxx","localTopic":false,"createTime":1717494901000,"regionId":"cn-hangzhou","statusName":"服务中","topic":"test11","remark":"657","partitionNum":12,"compactTopic":false,"status":0,"tags":[]}]}.
[2024-06-05 10:37:39]- cmd:TopicMigrationFromAliyun, destTopicList:{"total":0,"code":200,"requestId":"xxx","success":true,"pageSize":10000,"currentPage":1,"message":"operation success.","topicList":[]}.
[2024-06-05 10:37:39]- Dest instance topic list is empty, will create all topic in source instance.
[2024-06-05 10:37:39]- The quantities required are as follows:
+------------------+------------------+
| TopicNum         | PartitionNum     |
+------------------+------------------+
| 1                | 12               |
+------------------+------------------+
[2024-06-05 10:37:39]- Check instance capacity success.
[2024-06-05 10:37:39]- Will create topic:test11, isCompactTopic:false, partition number:12.

  4. 执行以下命令，迁移Topic。

    java -jar kafka-migration-assessment.jar TopicMigrationFromAliyun  \ 
--sourceAk <yoursourceAccessKeyId>  \ 
--sourceSk <yoursourceAccessKeySecret>  \ 
--sourceRegionId <yoursourceRegionId>  \ 
--sourceInstanceId <yoursourceInstanceId>  \ 
--destAk <yourdestAccessKeyId>  \ 
--destSk <yourdestAccessKeySecret>  \ 
--destRegionId <yourdestRegionId>  \ 
--destInstanceId <yourdestInstanceId>  \ 
--commit

    参数

    描述

    commit

    提交迁移请求。

    返回结果示例如下： 

    [2024-06-05 11:13:44]- Create Topic: test11  success.
Request: {"PartitionNum":"12","Action":"CreateTopic","CompactTopic":"false","ServiceCode":"alikafka","Format":"JSON","Version":"2019-09-16","InstanceId":"xxx","Topic":"test11","Remark":"657"}.
Response: {"code":200,"message":"operation success","requestId":"xxx","success":true}.
[2024-06-05 11:13:45]- Report meta data success, migrationType:topic_from_aliyun, total:1, succeed:1, failed:0, progress:1/1.
[2024-06-05 11:13:45]- Migrate Topic to Alibaba Cloud Kafka. Migration Report:
+------------------------------+----------------------+------------------------+-------------------------------------------+
| Topic                        | PartitionsToSubtract | MigrationSuccessFlag   | RequestId                                 |
+------------------------------+----------------------+------------------------+-------------------------------------------+
| test11                       | 12                   | true                   | xxx                                       |
+------------------------------+----------------------+------------------------+-------------------------------------------+

    MigrationSuccessFlag列为true代表Topic已成功在目标实例中创建。

Group

  1. 打开命令行工具。

  2. 使用cd命令将路径切换到迁移工具所在目录。

  3. 执行以下命令，对要迁移的Group进行预检。

    java -jar kafka-migration-assessment.jar ConsumerGroupMigrationFromAliyun  \
--sourceAk <yoursourceAccessKeyId>  \ 
--sourceSk <yoursourceAccessKeySecret>  \ 
--sourceRegionId <yoursourceRegionId>  \ 
--sourceInstanceId <yoursourceInstanceId>  \ 
--destAk <yourdestAccessKeyId>  \ 
--destSk <yourdestAccessKeySecret>  \ 
--destRegionId <yourdestRegionId>  \ 
--destInstanceId <yourdestInstanceId>

    上文中<>处内容需要替换为您的实例信息，配置规则请参见下表：

    参数

    描述

    sourceAk

    源云消息队列 Kafka 版实例所属阿里云账号的AccessKey ID。

    sourceSk

    源云消息队列 Kafka 版实例所属阿里云账号的AccessKey Secret。

    sourceRegionId

    源云消息队列 Kafka 版实例的地域ID。例如：cn-hangzhou。

    sourceInstanceId

    源云消息队列 Kafka 版实例的ID。

    destAk

    目标云消息队列 Kafka 版实例所属阿里云账号的AccessKey ID。

    destSk

    目标云消息队列 Kafka 版实例所属阿里云账号的AccessKey Secret。

    destRegionId

    目标云消息队列 Kafka 版实例的地域ID。例如：cn-hangzhou。

    destInstanceId

    目标云消息队列 Kafka 版实例的ID。

    返回结果示例如下：

    [ecs-assist-user@xxx                 ~]$ java -jar kafka-migration-assessment.jar ConsumerGroupMigrationFromAliyun --sourceAk xxx
--sourceSk xxx                              --sourceRegionId cn-hangzhou --sourceInstanceId xxx                              --destAk xxx
-destSk xxx                                 --destRegionId cn-hangzhou --destInstanceId xxx
log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAddCookies).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[2024-06-05 11:28:21]- cmd: ConsumerGroupMigrationFromAliyun, request: null, response: {"code":200,"requestId":"xxx","succ
ess":true,"consumerList":[{"instanceId":"xxx","regionId":"cn-hangzhou","consumerId":"Group2","tags":[]},{"instanceId":"alikafka_p
xxx","regionId":"cn-hangzhou","consumerId":"Group1","tags":[]}],"message":"operation success."}.
[2024-06-05 11:28:21]- cmd: ConsumerGroupMigrationFromAliyun, request: null, response: {"code":200,"requestId":"xxx","succ
ess":true,"consumerList":[],"message":"operation success."}.
[2024-06-05 11:28:21]- Dest instance consumer list is empty, will create all topic in source instance.
[2024-06-05 11:28:21]- The quantities required are as follows:
+-----------------+
| GroupNum        |
+-----------------+
| 2               |
+-----------------+
[2024-06-05 11:28:21]- Check instance capacity success.
[2024-06-05 11:28:21]- Will create consumer groups:[Group2, Group1].

  4. 执行以下命令，迁移Group。

    java -jar kafka-migration-assessment.jar ConsumerGroupMigrationFromAliyun  \
--sourceAk <yoursourceAccessKeyId>  \ 
--sourceSk <yoursourceAccessKeySecret>  \ 
--sourceRegionId <yoursourceRegionId>  \ 
--sourceInstanceId <yoursourceInstanceId>  \ 
--destAk <yourdestAccessKeyId>  \ 
--destSk <yourdestAccessKeySecret>  \ 
--destRegionId <yourdestRegionId>  \ 
--destInstanceId <yourdestInstanceId>  \
--commit

    参数

    说明

    commit

    提交迁移请求。

    返回结果示例如下：

    [2024-06-05 11:39:24]- Report meta data success, migrationType:consumer_from_aliyun, total:2, succeed:0, failed:0, progress:0/2.
[2024-06-05 11:39:25]- ConsumerCreate success, consumer group=Group2
[2024-06-05 11:39:28]- ConsumerCreate success, consumer group=Group1
[2024-06-05 11:39:28]- Report meta data success, migrationType:consumer_from_aliyun, total:2, succeed:2, failed:0, progress:2/2.
[2024-06-05 11:39:28]- Migrate Group to Alibaba Cloud Kafka. Migration Report:
+----------+-------------------+------------------------+--------------+
| Group    | GroupToSubtract   | MigrationSuccessFlag   | RequestId    |
+----------+-------------------+------------------------+--------------+
| Group2   | 1                 | true                   | xxx          |
+----------+-------------------+------------------------+--------------+
| Group1   | 1                 | true                   | xxx          |
+----------+-------------------+------------------------+--------------+

    MigrationSuccessFlag列为true代表Topic已成功在目标实例中创建。

结果验证

  1. 登录云消息队列 Kafka 版控制台，在概览页面的资源分布区域，选择地域。

  2. 在实例列表页面，单击目标实例名称。

    • Topic：在实例详情页面，单击左侧导航栏的Topic 管理，在Topic列表中，查看成功迁移的Topic。

    • Group：在实例详情页面，单击左侧导航栏的Group 管理，在Group列表中，查看成功迁移的Group。

后续操作

由于迁移过程中涉及实例切换，请在迁移结束后检查客户端实例接入点是否需要更新，详情请参见查看接入点。

上一篇：借助工具迁移下一篇：使用Prometheus Exporter监控Kafka客户端