Metadata import

Prerequisites

• Java Development Kit (JDK) 8 is downloaded and installed.

• Purchase and deploy an ApsaraMQ for Kafka instance:

  • Purchase and deploy a VPC instance

  • Purchase and deploy an internet- and VPC-connected instance

  Note

  For information about the instance specifications to purchase, see Step 1: Evaluate specifications.

Limitations

Serverless instances do not support metadata import tasks.

Background information

Kafka cluster metadata consists of the topic and group configurations of a Kafka cluster. This metadata is stored in ZooKeeper, and each node in the cluster retrieves the latest metadata from ZooKeeper. As a result, the metadata exported from any node is always up-to-date and consistent. You can export the cluster metadata to a JSON file and then import it into another Kafka cluster to back up the metadata of your self-managed Kafka cluster.

To migrate metadata, you export it from your self-managed cluster and import it into an ApsaraMQ for Kafka instance. ApsaraMQ for Kafka then parses the imported metadata, allowing you to visually configure topics and groups. When the import task is complete, the corresponding topics and groups are created in the destination ApsaraMQ for Kafka instance.

Step 1: Evaluate specifications

Use the specification evaluation feature to get recommendations for instance specifications based on your self-managed Kafka cluster's data.

1. Log on to the ApsaraMQ for Kafka console. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

2. In the left-side navigation pane, click Migration. On the page that appears, click the Metadata Import tab.

3. On the Metadata Import tab, click Evaluation.

4. In the Evaluation panel, enter your self-managed cluster information to get instance specification recommendations for ApsaraMQ for Kafka.

   Parameter	Description	Example
   Major Version	The Kafka version of your self-managed cluster.	0.10.x
   Migration from IDC	Whether the migration is from an on-premises data center (IDC).	Yes
   Maximum Cluster Traffic	The peak traffic for message production or consumption. Unit: MB/s.	20
   Public Network Traffic	If you do not need internet access, enter 0. Unit: MB/s.	0
   SSD	Whether the disk type is SSD.	Yes
   Disk Capacity	The total storage capacity of the disk. Unit: GB.	500
   Partitions	The total number of partitions for the topics to be migrated.	800
   Replicas	The number of replicas for each topic.	3
   Scenario	The scenario for message production and consumption.	Business Message
   Response Latency	The response time (RT) requirement for message production and consumption.	No special requirements

   After you enter the parameters, click OK. The recommended instance specifications are displayed at the top of the Evaluation panel. An example of the recommended specifications is as follows: Specification Type is Standard (High-write), Instance Type is VPC instance, Peak Traffic is 20 MB/s, Disk Type is SSD, Disk Capacity is 900 GB, Partitions is 0, and Server Version is 0.10.2. In the instance list, click Purchase Instance and purchase an ApsaraMQ for Kafka instance based on the recommendations.

Step 2: Export metadata

Use the metadata export tool to export the metadata file from your self-managed Kafka cluster.

1. Download the metadata export tool: kafka-migration-assessment.jar.

2. Upload the tool to a server in your self-managed Kafka cluster.

3. Run the following command to grant executable permissions on the JAR file:

   chmod 777 kafka-migration-assessment.jar

4. Run the following command to export the metadata:

   java -jar kafka-migration-assessment.jar MigrationFromZk --sourceZkConnect <yourHost:yourPort> --sourceBootstrapServers <yourHost:yourPort> --targetDirectory <yourDirectoryName> --fileName <yourFileName> --commit

   Parameter	Description	Example
   sourceZkConnect	The IP address and port number of the ZooKeeper service for your self-managed Kafka cluster. If you do not specify this parameter, the tool attempts to automatically discover them.	192.168.XX.XX:2181
   sourceBootstrapServers	The IP address and port number of your self-managed Kafka cluster. If you do not specify this parameter, the tool attempts to automatically discover them.	192.168.XX.XX:9092
   targetDirectory	The directory where the exported metadata file is stored. If not specified, the current directory is used by default.	../home/
   fileName	The name of the exported metadata file. If not specified, the default filename is kafka-metadata-export.json.	metadata.json
   commit	Required to run the export command.	commit
   installTsar	Specifies whether to automatically install Tsar. By default, Tsar is not installed. Installing Tsar provides more accurate information about the current machine's specifications, recent storage usage, traffic, and Kafka cluster configuration. However, automatic installation can be time-consuming and may fail due to environmental differences.	None
   evaluate	Specifies whether to retrieve information about the current machine's specifications, recent storage usage, traffic, and Kafka cluster configuration. This information is used during migration to recommend a suitable ApsaraMQ for Kafka instance specification. The default value is true. If you do not require a specification evaluation, set this parameter to false.	None

   A successful command execution generates a JSON file in the specified directory.

5. Locate the exported metadata file in the configured directory and download it to your local machine.

Step 3: Create a metadata import task

1. Log on to the ApsaraMQ for Kafka console. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

2. In the left-side navigation pane, click Migration. On the page that appears, click the Metadata Import tab.

3. On the Metadata Import tab, click Create Task.

4. In the Create Metadata Import Task panel, configure the parameters and click Create.

   1. On the Create Task tab, set the Task Name and Destination Instance, and then upload the metadata file that you obtained in Step 2: Export metadata.

   2. On the Edit Topic tab, you can perform the following operations:

      • To add a topic, click Add Topic and enter the Name, Description, Partitions, and Message Type for the topic.

      • To edit a topic, click Modify in the Actions column and change the Description, Partitions, Message Type, or Log Cleanup Policy.

      • To delete a topic, click Delete in the Actions column, and then click OK in the Note dialog box.

   3. On the Edit Group tab, you can perform the following operations:

      • To add a group, click Add Group and enter the Group ID and Description.

      • To edit a group, click Modify in the Actions column and change the Description.

      • To delete a group, click Delete in the Actions column, and then click OK in the Note dialog box.

   After the task is created, you can view its progress on the Migration page. The Topic Migration Progress and Group Migration Progress columns show the status. To view more information, click Details in the Actions column.

Step 4: Verify migration results

1. Log on to the ApsaraMQ for Kafka console. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

2. In the left-side navigation pane, click Instances.

3. On the Instances page, click the name of the destination instance.

4. Verify the imported resources.

   • In the left-side navigation pane, click Topics. On the Topics page, verify that the imported topics appear on the page.

   • In the left-side navigation pane, click Groups. On the Groups page, verify that the imported groups appear on the page.