For cloud migration and testing scenarios, you can enable automatic topic creation. This lets you create topics programmatically without using the console, simplifying operations.
Prerequisites
You have purchased and deployed a Message Queue for Apache Kafka instance that meets the following requirements:
Item | Description |
Status | Running |
Version | The major version is 2.2.0 or later, and the minor version is the latest. |
You can view the status and version of your instance in the Basic Information section on the Instance Details page of the Message Queue for Apache Kafka console.
Usage notes
Important: This feature is intended for migration and testing scenarios only. We strongly recommend that you do not enable automatic topic creation in production environments. If you must enable it for a production environment, do not use it for critical production services.
For a Professional Edition instance with ACL enabled, you cannot automatically create a topic by sending a message.
Procedure
Log on to the ApsaraMQ for Kafka console. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.
On the Instances page, click the name of the instance that you want to manage.
On the Instance Details page, in the Configuration Information section, turn on the Automatic Topic Creation switch.
NoteNaming conventions for automatically created topics:
Must be 3 to 64 characters in length.
Can contain only letters, digits, hyphens (-), and underscores (_), and must contain at least one letter or digit.
On a Standard Edition instance with this feature enabled, sending a message to a non-existent topic automatically creates it. The new topic then appears in the console.
FAQ
Can topics be created automatically?
Yes, but we strongly recommend against it. In Message Queue for Apache Kafka, you should manage topics using the Message Queue for Apache Kafka console and OpenAPI. We advise against enabling automatic topic creation in a production environment for the following reasons:
Automatic topic creation does not support fine-grained permission control or resource operation auditing, which poses security and governance risks.
It can lead to uncontrolled resource consumption and poor resource governance, compromising system stability.
Automatic topic creation is not guaranteed to succeed, and failures can disrupt your services. For more information, see What should I do if automatic topic creation fails?
When should I enable automatic topic creation?
This feature is not generally recommended. However, you can enable it to support specific use cases such as rapid cloud migration and testing. If you enable this feature, use it only for these scenarios.
Handling automatic topic creation failures
If automatic topic creation fails, you can try the following solutions:
If you only need to create a few topics, we recommend creating them manually in the console or using OpenAPI.
Check the logs or troubleshoot the issue based on the following possible causes:
Invalid operations: An invalid operation can prevent the topic from being created or used correctly, affecting your services. Examples include:
Invalid topic name: The topic name specified by the application does not meet the naming conventions. For example, the name contains invalid characters or exceeds the length limit.
Invalid topic configuration: The broker engine is improperly selected, or the quotas for topics or partitions are insufficient.
ACL enabled: If the ACL feature is enabled, you cannot automatically create a topic by sending a message.
Uncontrollable factors: Probabilistic failures due to external factors can cause long recovery times and severely impact your services. For example, network jitter, server upgrades, management platform maintenance, or ECS instance failures can significantly increase topic creation time and lead to failures.
Service conflicts: Automatic topic creation can cause naming conflicts between different services. These conflicts cause data from different services to mix, which makes issues difficult to detect. Once an issue is detected, recovery is extremely difficult and can lead to irreversible data loss.
Latency issues: Creating a topic on demand takes significantly longer than using an existing topic. This can cause brief service interruptions that are unacceptable for most applications.
Quota limits: topic creation fails if the quota for topics or partitions is exceeded.
References
For detailed steps on cloud migration, see Cloud migration.
If you want to create topics using the console or API, see Create a topic and CreateTopic.