Procedure
Implement schema validation for messages in an ApsaraMQ for Confluent instance to ensure that messages sent by producers conform to a predefined data structure, improving data consistency and system reliability.
Process overview
Step 1: Purchase and deploy an instance
Purchase an instance
Log on to the ApsaraMQ for Confluent console. In the left-side navigation pane, click Instances.
In the top navigation bar, select a region and click Buy Instance.
-
In the panel that appears, set the Instance Version parameter to Confluent and then click OK.Select Instance Billing Method
-
Configure the following parameters and click Create Cluster. Complete the payment as prompted.
Parameter
Description
Plan Edition
Cluster edition: Professional Edition or Enterprise Edition. Compare features in Editions.
Duration
Subscription duration. Example: 1 year.
Region and Zone
Region and zone where the cluster runs. Example: China (Hangzhou).
Computing Resources
Number of capacity units (CUs). Size based on throughput and retention needs. Evaluate cluster resources.
Component Resources
Storage and component configuration. Adjust based on components you plan to use (ksqlDB, Connect). Evaluate cluster resources.
NoteOne capacity unit (CU) = 1 CPU core + 4 GB of memory.
Deploy the instance
Log on to the ApsaraMQ for Confluent console. In the left-side navigation pane, click Instances.
-
In the top navigation bar, select the region. Find the instance whose status is Not Deployed and click Deploy in the Actions column.
-
In the Deploy Instance panel, configure the following parameters and click OK.
Deployment parameters:
Parameter
Description
Example
Deployment Mode
Single-zone or multi-zone deployment.
Single zone
Zone
Select a zone.
Zone A
VPC
VPC in which the SLB instance resides.
vpc-bp17fapfdj0dwzjkd****
vSwitch
vSwitch in the selected zone. If none exists, create one in the corresponding zone. Each vSwitch must have more than 64 available IP addresses.
vsw-bp1gbjhj53hdjdkg****
SLB
By default, SLB is activated.
None
Assign Public Network IP
Whether to enable public Internet access for the cluster.
Yes
Logon Username
Control Center logon username. Default: root.
root
Logon Password
Password for Control Center logon.
******
Confirm Password
Re-enter the password.
******
After you configure the parameters, the instance enters the Deploying state. Deployment takes approximately 10 to 30 minutes.
Step 2: Log on to Control Center
Log on to the ApsaraMQ for Confluent console. In the left-side navigation pane, click Instances.
-
In the top navigation bar, select a region. On the Instances page, click the name of your target instance.
-
In the upper-right corner of the Instance Details page, click Log on to Console to open Control Center.
NoteLog on to Control Center with the root username and password that you configured when you deployed the instance.
-
After logging on, you are redirected to the Home page in Control Center.
The page displays the cluster health status, indicated by the Healthy clusters and Unhealthy clusters badges, and cluster cards. Each card contains the cluster name, running status, and overview information (Brokers, partitions, Topics, Production, Consumption), as well as the number of connected ksqlDB clusters and Connect clusters.
(Optional) Step 3: Install a connector
To use connector features, you must purchase Connect component resources and install a connector.
-
Enterprise Edition supports custom connectors that are not in the connector catalog. To install a custom connector, you must submit a ticket and provide a validated connector package.
-
ApsaraMQ for Confluent supports different types of connectors based on the instance edition. For more information, see Manage connectors.
-
For more information, see Connector use cases.
Step 4: Create a topic
-
Log on to Control Center. On the Home page, click the controlcenter.clusterk card to go to the Cluster overview page.
The left-side navigation menu contains options such as Cluster overview, Brokers, Topics, and Consumers. The right side of the page displays an overview of the cluster's brokers and topics.
-
In the left-side navigation pane, click Topics. On the topic list page, click + Add topic.
-
On the New topic page, set a topic name and the number of partitions, and then click Create with defaults.
For example, set Topic name to
transactionsand Number of partitions to1. -
After you create the topic, you are redirected to its details page.
The page includes four tabs: Overview, Messages, Schema, and Configuration. The Overview tab shows metrics such as Production (production rate), Consumption (consumption rate), and Availability (availability).
Step 5: Enable schema validation
-
On the topic details page, click the Configuration tab, and then click Edit settings.
-
Then, click Switch to expert mode.
-
Set the confluent_value_schema_validation field to true and then click Save changes to enable schema validation for message content. After this feature is enabled, message formats are validated during production and consumption.
Step 6: Manage users and grant permissions
To use other LDAP users, add them in the ApsaraMQ for Confluent console and grant them permissions. For details, see Manage users and grant permissions.
Step 7: Network access and security settings
To produce or consume messages, provide the service endpoint and grant the required permissions to the LDAP user.
-
Endpoints
Select an internal or public endpoint based on your requirements. If you want to use a public endpoint, you must enable public network access.
-
Permissions
Grant the required service permissions to the LDAP user. For more information, see Manage users and grant permissions.
NoteThe root user has all permissions. For production environments, we recommend using dedicated users with only the minimum required permissions.
-
For more information about security settings, see Network access and security settings.
Step 8: Produce and consume messages
1. Set up the environment
-
This section shows how to use sample code on a Linux server to connect to ApsaraMQ for Confluent to produce and consume messages.
-
Install Java 8 or 11. For more information about supported Java versions, see Java Support in Confluent Platform.
-
Install Maven 3.8 or a later version. For more information, see Installing Apache Maven.
-
-
Run the following commands to clone the sample code and switch to the
7.9.0-postbranch.git clone https://github.com/confluentinc/examples.git cd examples/clients/avro git checkout 7.9.0-post -
Create a client configuration file named
java.configin the$HOME/.confluent/directory, where$HOMEis your user's home directory. Add the following settings to the file.# Required connection configs for Kafka producer, consumer, and admin bootstrap.servers={{ BROKER_ENDPOINT }} security.protocol=SASL_SSL sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='{{ CLUSTER_API_KEY }}' password='{{ CLUSTER_API_SECRET }}'; sasl.mechanism=PLAIN # Required for correctness in Apache Kafka clients prior to 2.6 client.dns.lookup=use_all_dns_ips # Best practice for higher availability in Apache Kafka clients prior to 3.0 session.timeout.ms=45000 # Best practice for Kafka producer to prevent data loss acks=all # Required connection configs for Confluent Cloud Schema Registry schema.registry.url=https://{{ SR_ENDPOINT }} basic.auth.credentials.source=USER_INFO basic.auth.user.info={{ SR_API_KEY }}:{{ SR_API_SECRET }}Parameter
Description
Example value
BROKER_ENDPOINT
The endpoint of the KAFKA service.
Obtain the endpoint from the Access Links and Ports page of the ApsaraMQ for Confluent console. To use the public endpoint, you must enable public network access. For details on other security configurations, see Network access and security settings.
pub-kafka-xxxxxxxxxxx.csp.aliyuncs.com:9092
CLUSTER_API_KEY
The username and password of an LDAP user from the Users page in the ApsaraMQ for Confluent console.
During testing, you can use the root account and its password. To use other users, create them in the ApsaraMQ for Confluent console and grant them the required permissions for the Kafka cluster. For details, see Manage users and grant permissions.
root
CLUSTER_API_SECRET
**
SR_ENDPOINT
The endpoint of the SCHEMA_REGISTRY service.
Obtain the endpoint from the Access Links and Ports page of the ApsaraMQ for Confluent console. To use the public endpoint, you must enable public network access. For details on other security configurations, see Network access and security settings.
pub-schemaregistry-xxxxxxxxxxx.csp.aliyuncs.com:443
SR_API_KEY
The username and password of an LDAP user from the Users page in the ApsaraMQ for Confluent console.
During testing, you can use the root account and its password. To use other users, create them in the ApsaraMQ for Confluent console and grant them the required permissions for Schema Registry. For details, see Manage users and grant permissions.
root
SR_API_SECRET
**
-
The sample code uses the topic
transactions. If you use a different topic, you must update the topic name in the code. The following code provides samples. For more information, see Confluent Platform Code Samples.
2. Create a schema
-
Go to the
examples/clients/avroproject directory and run the following command to view the content of thePayment.avscfile.cat src/main/resources/avro/io/confluent/examples/clients/basicavro/Payment.avscOutput:
{ "namespace": "io.confluent.examples.clients.basicavro", "type": "record", "name": "Payment", "fields": [ {"name": "id", "type": "string"}, {"name": "amount", "type": "double"} ] } -
In the Control Center console, go to the topic details page, click the Schema tab, and then click Set a schema.
-
On the Schema tab, click Avro, paste the content of the
Payment.avscfile into the editor, and then click Create.
3. Produce messages
-
Go to the
examples/clients/avroproject directory and run the following command to compile the project.mvn clean compile package -
After the compilation completes, run the following command to produce messages.
mvn exec:java -Dexec.mainClass=io.confluent.examples.clients.basicavro.ProducerExample \ -Dexec.args="$HOME/.confluent/java.config"The following output indicates that the messages were produced successfully.
... Successfully produced 10 messages to a topic called transactions [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ ... -
You can view the produced messages in the Control Center console.
On the Topics page, select the transactions topic and click the Messages tab. You will see 10 messages with keys ranging from
"id0"to"id9", and the value is in JSON format, for example,{"id":"id9","amount":1000}.
4. Consume messages
-
If you have not yet compiled the project, go to the
examples/clients/avroproject directory and run the following command:mvn clean compile package -
Run the following command to consume messages.
mvn exec:java -Dexec.mainClass=io.confluent.examples.clients.basicavro.ConsumerExample \ -Dexec.args="$HOME/.confluent/java.config" -
The following output indicates that the messages were consumed successfully.
... key = id0, value = {"id": "id0", "amount": 1000.0} key = id1, value = {"id": "id1", "amount": 1000.0} key = id2, value = {"id": "id2", "amount": 1000.0} key = id3, value = {"id": "id3", "amount": 1000.0} key = id4, value = {"id": "id4", "amount": 1000.0} key = id5, value = {"id": "id5", "amount": 1000.0} key = id6, value = {"id": "id6", "amount": 1000.0} key = id7, value = {"id": "id7", "amount": 1000.0} key = id8, value = {"id": "id8", "amount": 1000.0} key = id9, value = {"id": "id9", "amount": 1000.0} ...