Procedure

更新时间: 2026-07-17 17:39:22

Implement schema validation for messages in an ApsaraMQ for Confluent instance to ensure that messages sent by producers conform to a predefined data structure, improving data consistency and system reliability.

Process overview

Step 1: Purchase and deploy an instance

Purchase an instance

  1. Log on to the ApsaraMQ for Confluent console. In the left-side navigation pane, click Instances.

  2. In the top navigation bar, select a region and click Buy Instance.

  3. In the panel that appears, set the Instance Version parameter to Confluent and then click OK.Select Instance Billing Method

  4. Configure the following parameters and click Create Cluster. Complete the payment as prompted.

    Parameter

    Description

    Plan Edition

    Cluster edition: Professional Edition or Enterprise Edition. Compare features in Editions.

    Duration

    Subscription duration. Example: 1 year.

    Region and Zone

    Region and zone where the cluster runs. Example: China (Hangzhou).

    Computing Resources

    Number of capacity units (CUs). Size based on throughput and retention needs. Evaluate cluster resources.

    Component Resources

    Storage and component configuration. Adjust based on components you plan to use (ksqlDB, Connect). Evaluate cluster resources.

    Note

    One capacity unit (CU) = 1 CPU core + 4 GB of memory.

Deploy the instance

  1. Log on to the ApsaraMQ for Confluent console. In the left-side navigation pane, click Instances.

  2. In the top navigation bar, select the region. Find the instance whose status is Not Deployed and click Deploy in the Actions column.

  3. In the Deploy Instance panel, configure the following parameters and click OK.

    Deployment parameters:

    Parameter

    Description

    Example

    Deployment Mode

    Single-zone or multi-zone deployment.

    Single zone

    Zone

    Select a zone.

    Zone A

    VPC

    VPC in which the SLB instance resides.

    vpc-bp17fapfdj0dwzjkd****

    vSwitch

    vSwitch in the selected zone. If none exists, create one in the corresponding zone. Each vSwitch must have more than 64 available IP addresses.

    vsw-bp1gbjhj53hdjdkg****

    SLB

    By default, SLB is activated.

    None

    Assign Public Network IP

    Whether to enable public Internet access for the cluster.

    Yes

    Logon Username

    Control Center logon username. Default: root.

    root

    Logon Password

    Password for Control Center logon.

    ******

    Confirm Password

    Re-enter the password.

    ******

    After you configure the parameters, the instance enters the Deploying state. Deployment takes approximately 10 to 30 minutes.

Step 2: Log on to Control Center

  1. Log on to the ApsaraMQ for Confluent console. In the left-side navigation pane, click Instances.

  2. In the top navigation bar, select a region. On the Instances page, click the name of your target instance.

  3. In the upper-right corner of the Instance Details page, click Log on to Console to open Control Center.

    Note

    Log on to Control Center with the root username and password that you configured when you deployed the instance.

  4. After logging on, you are redirected to the Home page in Control Center.

    The page displays the cluster health status, indicated by the Healthy clusters and Unhealthy clusters badges, and cluster cards. Each card contains the cluster name, running status, and overview information (Brokers, partitions, Topics, Production, Consumption), as well as the number of connected ksqlDB clusters and Connect clusters.

(Optional) Step 3: Install a connector

To use connector features, you must purchase Connect component resources and install a connector.

  • Enterprise Edition supports custom connectors that are not in the connector catalog. To install a custom connector, you must submit a ticket and provide a validated connector package.

  • ApsaraMQ for Confluent supports different types of connectors based on the instance edition. For more information, see Manage connectors.

  • For more information, see Connector use cases.

Step 4: Create a topic

  1. Log on to Control Center. On the Home page, click the controlcenter.clusterk card to go to the Cluster overview page.

    The left-side navigation menu contains options such as Cluster overview, Brokers, Topics, and Consumers. The right side of the page displays an overview of the cluster's brokers and topics.

  2. In the left-side navigation pane, click Topics. On the topic list page, click + Add topic.

  3. On the New topic page, set a topic name and the number of partitions, and then click Create with defaults.

    For example, set Topic name to transactions and Number of partitions to 1.

  4. After you create the topic, you are redirected to its details page.

    The page includes four tabs: Overview, Messages, Schema, and Configuration. The Overview tab shows metrics such as Production (production rate), Consumption (consumption rate), and Availability (availability).

Step 5: Enable schema validation

  1. On the topic details page, click the Configuration tab, and then click Edit settings.

  2. Then, click Switch to expert mode.

  3. Set the confluent_value_schema_validation field to true and then click Save changes to enable schema validation for message content. After this feature is enabled, message formats are validated during production and consumption.

Step 6: Manage users and grant permissions

To use other LDAP users, add them in the ApsaraMQ for Confluent console and grant them permissions. For details, see Manage users and grant permissions.

Step 7: Network access and security settings

To produce or consume messages, provide the service endpoint and grant the required permissions to the LDAP user.

  • Endpoints

    Select an internal or public endpoint based on your requirements. If you want to use a public endpoint, you must enable public network access.

  • Permissions

    Grant the required service permissions to the LDAP user. For more information, see Manage users and grant permissions.

    Note

    The root user has all permissions. For production environments, we recommend using dedicated users with only the minimum required permissions.

  • For more information about security settings, see Network access and security settings.

Step 8: Produce and consume messages

1. Set up the environment

  1. This section shows how to use sample code on a Linux server to connect to ApsaraMQ for Confluent to produce and consume messages.

  2. Run the following commands to clone the sample code and switch to the 7.9.0-post branch.

    git clone https://github.com/confluentinc/examples.git
    cd examples/clients/avro
    git checkout 7.9.0-post
  3. Create a client configuration file named java.config in the $HOME/.confluent/ directory, where $HOME is your user's home directory. Add the following settings to the file.

    # Required connection configs for Kafka producer, consumer, and admin
    bootstrap.servers={{ BROKER_ENDPOINT }}
    security.protocol=SASL_SSL
    sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='{{ CLUSTER_API_KEY }}' password='{{ CLUSTER_API_SECRET }}';
    sasl.mechanism=PLAIN
    # Required for correctness in Apache Kafka clients prior to 2.6
    client.dns.lookup=use_all_dns_ips
    # Best practice for higher availability in Apache Kafka clients prior to 3.0
    session.timeout.ms=45000
    # Best practice for Kafka producer to prevent data loss
    acks=all
    # Required connection configs for Confluent Cloud Schema Registry
    schema.registry.url=https://{{ SR_ENDPOINT }}
    basic.auth.credentials.source=USER_INFO
    basic.auth.user.info={{ SR_API_KEY }}:{{ SR_API_SECRET }}

    Parameter

    Description

    Example value

    BROKER_ENDPOINT

    The endpoint of the KAFKA service.

    Obtain the endpoint from the Access Links and Ports page of the ApsaraMQ for Confluent console. To use the public endpoint, you must enable public network access. For details on other security configurations, see Network access and security settings.

    pub-kafka-xxxxxxxxxxx.csp.aliyuncs.com:9092

    CLUSTER_API_KEY

    The username and password of an LDAP user from the Users page in the ApsaraMQ for Confluent console.

    During testing, you can use the root account and its password. To use other users, create them in the ApsaraMQ for Confluent console and grant them the required permissions for the Kafka cluster. For details, see Manage users and grant permissions.

    root

    CLUSTER_API_SECRET

    **

    SR_ENDPOINT

    The endpoint of the SCHEMA_REGISTRY service.

    Obtain the endpoint from the Access Links and Ports page of the ApsaraMQ for Confluent console. To use the public endpoint, you must enable public network access. For details on other security configurations, see Network access and security settings.

    pub-schemaregistry-xxxxxxxxxxx.csp.aliyuncs.com:443

    SR_API_KEY

    The username and password of an LDAP user from the Users page in the ApsaraMQ for Confluent console.

    During testing, you can use the root account and its password. To use other users, create them in the ApsaraMQ for Confluent console and grant them the required permissions for Schema Registry. For details, see Manage users and grant permissions.

    root

    SR_API_SECRET

    **

  4. The sample code uses the topic transactions. If you use a different topic, you must update the topic name in the code. The following code provides samples. For more information, see Confluent Platform Code Samples.

    Producer sample code

    import io.confluent.kafka.serializers.AbstractKafkaSchemaSerDeConfig;
    import org.apache.kafka.clients.producer.KafkaProducer;
    import org.apache.kafka.clients.producer.ProducerConfig;
    import org.apache.kafka.clients.producer.ProducerRecord;
    import org.apache.kafka.common.serialization.StringSerializer;
    import io.confluent.kafka.serializers.KafkaAvroSerializer;
    import org.apache.kafka.common.errors.SerializationException;
    import java.util.Properties;
    import java.io.IOException;
    import java.nio.file.Files;
    import java.nio.file.Paths;
    import java.io.FileInputStream;
    import java.io.InputStream;
    public class ProducerExample {
        private static final String TOPIC = "transactions";
        private static final Properties props = new Properties();
        private static String configFile;
        @SuppressWarnings("InfiniteLoopStatement")
        public static void main(final String[] args) throws IOException {
            if (args.length < 1) {
              // Backwards compatibility, assume localhost
              props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
              props.put(AbstractKafkaSchemaSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, "http://localhost:8081");
            } else {
              // Load properties from a local configuration file
              // Create the configuration file (e.g. at '$HOME/.confluent/java.config') with configuration parameters
              // to connect to your Kafka cluster, which can be on your local host, Confluent Cloud, or any other cluster.
              // Documentation at https://docs.confluent.io/platform/current/tutorials/examples/clients/docs/java.html
              configFile = args[0];
              if (!Files.exists(Paths.get(configFile))) {
                throw new IOException(configFile + " not found.");
              } else {
                try (InputStream inputStream = new FileInputStream(configFile)) {
                  props.load(inputStream);
                }
              }
            }
            props.put(ProducerConfig.ACKS_CONFIG, "all");
            props.put(ProducerConfig.RETRIES_CONFIG, 0);
            props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
            props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class);
            try (KafkaProducer<String, Payment> producer = new KafkaProducer<String, Payment>(props)) {
                for (long i = 0; i < 10; i++) {
                    final String orderId = "id" + Long.toString(i);
                    final Payment payment = new Payment(orderId, 1000.00d);
                    final ProducerRecord<String, Payment> record = new ProducerRecord<String, Payment>(TOPIC, payment.getId().toString(), payment);
                    producer.send(record);
                    Thread.sleep(1000L);
                }
                producer.flush();
                System.out.printf("Successfully produced 10 messages to a topic called %s%n", TOPIC);
            } catch (final SerializationException e) {
                e.printStackTrace();
            } catch (final InterruptedException e) {
                e.printStackTrace();
            }
        }
    }

    Consumer sample code

    import io.confluent.kafka.serializers.AbstractKafkaSchemaSerDeConfig;
    import org.apache.kafka.clients.consumer.ConsumerConfig;
    import org.apache.kafka.clients.consumer.ConsumerRecord;
    import org.apache.kafka.clients.consumer.ConsumerRecords;
    import org.apache.kafka.clients.consumer.KafkaConsumer;
    import io.confluent.kafka.serializers.KafkaAvroDeserializer;
    import io.confluent.kafka.serializers.KafkaAvroDeserializerConfig;
    import org.apache.kafka.common.serialization.StringDeserializer;
    import java.time.Duration;
    import java.util.Collections;
    import java.util.Properties;
    import java.io.IOException;
    import java.nio.file.Files;
    import java.nio.file.Paths;
    import java.io.FileInputStream;
    import java.io.InputStream;
    public class ConsumerExample {
        private static final String TOPIC = "transactions";
        private static final Properties props = new Properties();
        private static String configFile;
        @SuppressWarnings("InfiniteLoopStatement")
        public static void main(final String[] args) throws IOException {
            if (args.length < 1) {
              // Backwards compatibility, assume localhost
              props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
              props.put(AbstractKafkaSchemaSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, "http://localhost:8081");
            } else {
              // Load properties from a local configuration file
              // Create the configuration file (e.g. at '$HOME/.confluent/java.config') with configuration parameters
              // to connect to your Kafka cluster, which can be on your local host, Confluent Cloud, or any other cluster.
              // Documentation at https://docs.confluent.io/platform/current/tutorials/examples/clients/docs/java.html
              configFile = args[0];
              if (!Files.exists(Paths.get(configFile))) {
                throw new IOException(configFile + " not found.");
              } else {
                try (InputStream inputStream = new FileInputStream(configFile)) {
                  props.load(inputStream);
                }
              }
            }
            props.put(ConsumerConfig.GROUP_ID_CONFIG, "test-payments");
            props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true");
            props.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000");
            props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
            props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
            props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, KafkaAvroDeserializer.class);
            props.put(KafkaAvroDeserializerConfig.SPECIFIC_AVRO_READER_CONFIG, true); 
            try (final KafkaConsumer<String, Payment> consumer = new KafkaConsumer<>(props)) {
                consumer.subscribe(Collections.singletonList(TOPIC));
                while (true) {
                    final ConsumerRecords<String, Payment> records = consumer.poll(Duration.ofMillis(100));
                    for (final ConsumerRecord<String, Payment> record : records) {
                        final String key = record.key();
                        final Payment value = record.value();
                        System.out.printf("key = %s, value = %s%n", key, value);
                    }
                }
            }
        }
    }

2. Create a schema

  1. Go to the examples/clients/avro project directory and run the following command to view the content of the Payment.avsc file.

    cat src/main/resources/avro/io/confluent/examples/clients/basicavro/Payment.avsc

    Output:

    {
     "namespace": "io.confluent.examples.clients.basicavro",
     "type": "record",
     "name": "Payment",
     "fields": [
         {"name": "id", "type": "string"},
         {"name": "amount", "type": "double"}
     ]
    }
    
  2. In the Control Center console, go to the topic details page, click the Schema tab, and then click Set a schema.

  3. On the Schema tab, click Avro, paste the content of the Payment.avsc file into the editor, and then click Create.

3. Produce messages

  1. Go to the examples/clients/avro project directory and run the following command to compile the project.

    mvn clean compile package
  2. After the compilation completes, run the following command to produce messages.

    mvn exec:java -Dexec.mainClass=io.confluent.examples.clients.basicavro.ProducerExample \
      -Dexec.args="$HOME/.confluent/java.config"

    The following output indicates that the messages were produced successfully.

    ...
    Successfully produced 10 messages to a topic called transactions
    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD SUCCESS
    [INFO] ------------------------------------------------------------------------
    ...
  3. You can view the produced messages in the Control Center console.

    On the Topics page, select the transactions topic and click the Messages tab. You will see 10 messages with keys ranging from "id0" to "id9", and the value is in JSON format, for example, {"id":"id9","amount":1000}.

4. Consume messages

  1. If you have not yet compiled the project, go to the examples/clients/avro project directory and run the following command:

    mvn clean compile package
  2. Run the following command to consume messages.

    mvn exec:java -Dexec.mainClass=io.confluent.examples.clients.basicavro.ConsumerExample \
      -Dexec.args="$HOME/.confluent/java.config"
  3. The following output indicates that the messages were consumed successfully.

    ...
    key = id0, value = {"id": "id0", "amount": 1000.0}
    key = id1, value = {"id": "id1", "amount": 1000.0}
    key = id2, value = {"id": "id2", "amount": 1000.0}
    key = id3, value = {"id": "id3", "amount": 1000.0}
    key = id4, value = {"id": "id4", "amount": 1000.0}
    key = id5, value = {"id": "id5", "amount": 1000.0}
    key = id6, value = {"id": "id6", "amount": 1000.0}
    key = id7, value = {"id": "id7", "amount": 1000.0}
    key = id8, value = {"id": "id8", "amount": 1000.0}
    key = id9, value = {"id": "id9", "amount": 1000.0}
    ...
上一篇: User Guide 下一篇: Estimate cluster resources for ApsaraMQ for Confluent
阿里云首页 云消息队列 Kafka 版 相关技术圈