Forward data to an AMQP consumer group

更新时间:
复制 MD 格式

By default, an AMQP server-side subscription receives data from all devices that belong to a product. To gain more control, you can use the Data Forwarding feature. This feature lets you use an SQL statement to process and filter messages from all or specific devices before forwarding them to an AMQP server-side subscription consumer group. The messages are then consumed by an AMQP client.

How it works

image

Data Forwarding can forward messages from a specified topic for all or specific devices under the same product to one or more consumer groups in real time. Each consumer group can include multiple consumers, which are AMQP clients. When a message is forwarded to a consumer group, a random consumer in that group receives the message. Different consumer groups are identified by their consumer group IDs. In the preceding figure:

  1. Consumer: an AMQP client that uses an AMQP SDK to register with a consumer group to receive messages that IoT Platform forwards.

  2. IoT Platform: configure the data source topic, SQL statement, and data forwarding destination for a Data Forwarding rule, and then start the rule to forward device messages to the AMQP server-side subscription consumer group.

    • Data source: For information about the supported topic types, see Data formats.

      Examples:

      • Data source Topic_Product A: forwards messages from all devices that belong to product A to a consumer group.

      • Data source Topic_Device B: forwards messages from a specified device, Device B, that belongs to product B to a consumer group.

    • SQL statement: write an SQL statement to parse and process the JSON data from device messages. For more information about the syntax of SQL statements, see SQL statements.

      Note

      Binary data is not parsed and is transmitted transparently.

    • Data forwarding destination: the AMQP server-side subscription consumer group that receives device data. For more information, see Manage AMQP consumer groups.

  3. AMQP client: After you start the rule, IoT Platform automatically forwards device messages to the AMQP client. You do not need to subscribe to topics in the AMQP client code. The AMQP client receives messages as long as it is online.

Note

Data Forwarding does not support sending messages from an AMQP client to a device. To send commands to a device, call the message communication APIs.

Use cases

Business servers receive device messages: Data Forwarding provides a flexible way to forward device messages to an AMQP server-side subscription consumer group.

  • Forward messages from specified devices.

  • Forward messages from specified topics.

  • Filter or process messages before forwarding.

To receive real-time messages from all devices that belong to a specified product on your AMQP client, you can directly configure an AMQP server-side subscription. For more information, see Configure an AMQP server-side subscription.

Limitations

  • If a client fails to authenticate within 15 seconds of connecting, the server closes the connection.

  • The maximum number of transactions per second (TPS) for a single connection of an AMQP client is 1,000. The TPS limit for message forwarding is determined by the message forwarding TPS specification of your instance. There is no limit on the message size. For more information about the limits of AMQP server-side subscriptions, see Limits on server-side subscriptions.

  • You cannot use the legacy Data Forwarding feature for products and devices on Cloud Gateway or MQTT instances. To forward data from these products and devices, you must use the new Data Forwarding feature. For more information, see Data Forwarding (New).

Prerequisites

Step 1: Configure a data forwarding destination

  1. Log on to the IoT Platform console.

  2. On the Overview page, find the instance that you want to manage and click the instance ID or instance name.

  3. In the navigation pane on the left, choose Message Forwarding > Data Forwarding.

  4. Find the rule that you want to manage and click View in the Actions column. The Data Forwarding Rule page appears.

    Important

    If the new version of the data forwarding page is displayed, click Back to Old Version in the upper-right corner. Then, find the destination rule and click View.

  5. In the Forward Data section, click Add Operation.

  6. In the Add Operation dialog box, for Select Operation, select Publish to AMQP Subscribed Consumer Group. Configure the other parameters as prompted and click OK.

    Parameter

    Description

    Select operation

    Select Publish to AMQP Subscribed Consumer Group.

    Consumer group

    Select an existing consumer group as the data forwarding destination. You can click Create Consumer Group to create a consumer group.

    Tag

    If you set a tag, this operation adds the tag to all messages forwarded to the AMQP server-side subscription consumer group.

    The tag must be 1 to 128 characters in length and can be a constant or a variable.

    • A constant can contain Chinese characters, letters, and digits.

    • A variable must be in the ${key} format. It represents the value of the key parameter in the JSON data returned by the SQL statement. If the value cannot be retrieved, no tag is added to the message.

  7. Return to the Data Forwarding page. Find the rule and click Start to enable the rule.

Step 2: Run the AMQP client

Important
  • Use the AMQP SDK access sample that is provided by Alibaba Cloud IoT Platform. Alibaba Cloud does not provide technical support for self-developed AMQP SDKs.

  • This example uses Java. For examples in other languages, see AMQP client access instructions.

  • This example uses an ECS instance with the Alibaba Cloud Linux operating system as the development environment for the AMQP client:

  1. Log on to the ECS instance. For more information about logon methods, see Select an ECS remote connection method.

  2. Run the following command to download the demo file.

    wget https://linkkit-export.oss-cn-shanghai.aliyuncs.com/amqp/amqp-demo.zip
  3. Run the following command to decompress the demo file.

    unzip amqp-demo.zip
  4. In the AmqpClient.java file in the src/main/java/com.aliyun.iotx.demo directory, modify the AMQP access information as described in the following table.

    Important

    The demo code in this example includes code to terminate the program (Thread.sleep(60 * 1000);). This means the program terminates one minute after it starts successfully. In a real-world scenario, you can set the runtime as needed.

    Parameter

    Description

    accessKey

    The AccessKey ID and AccessKey secret of an Alibaba Cloud account or a Resource Access Management (RAM) user.

    Log on to the IoT Platform console, move the pointer over your profile picture, and then click AccessKey Management to get the AccessKey ID and AccessKey secret.

    Important

    To avoid the security risks of hard coding an AccessKey into your business code, use environment variables to manage your AccessKey.

    Set the ALIBABA_CLOUD_ACCESS_KEY_ID and ALIBABA_CLOUD_ACCESS_KEY_SECRET environment variables in your local operating system to your AccessKey ID and AccessKey secret, respectively.

    In the sample code, you can retrieve them using the following methods:

    • System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID")

    • System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET")

    accessSecret

    consumerGroupId

    The ID of the consumer group in the corresponding instance of IoT Platform.

    Log on to the IoT Platform console. In the corresponding instance, choose Message Forwarding > Server-side Subscription > Consumer Group List to view your consumer group ID.

    iotInstanceId

    The instance ID. You can view the ID of the current instance on the Instance Overview page of the IoT Platform console.

    • If an ID value exists, you must pass this ID value.

    • If there is no Instance Overview page or ID value, pass an empty value, which is iotInstanceId = "".

    clientId

    The client ID. It is user-defined and cannot exceed 64 characters. Use a unique identifier such as the UUID, MAC address, or IP address of the server where your AMQP client is located.

    After the AMQP client is connected and started, log on to the IoT Platform console. In the corresponding instance, choose Message Forwarding > Server-side Subscription > Consumer Group List. Click View for the consumer group. The Consumer Group Details page will display this parameter, making it easy for you to identify and distinguish different clients.

    connectionCount

    The number of connections for starting the AMQP client. The maximum is 128. This is used for scaling out real-time message pushes.

    The Consumer Group Details page displays connected clients in the format ${clientId}+"-"+number. The minimum value for the number is 0.

    host

    The AMQP access domain name.

    For information about the AMQP access domain name corresponding to ${YourHost}, see View and configure instance endpoint information (Endpoint).

  5. The relevant Maven dependencies have been added to the pom.xml file. In the amqp-demo root directory, run the following command to reload Maven changes and build the project.

    mvn clean package
  6. In the amqp-demo/target directory, run the following command to run the generated JAR package.

    java -jar demo-0.0.1-SNAPSHOT-jar-with-dependencies.jar
  7. After you run the sample code, the following information is returned, which indicates that the AMQP client has connected to IoT Platform and is successfully receiving messages.

    Important

    Device messages can be received on the server only when the AMQP client is online.

    10:42:43.254 [main] INFO com.aliyun.iotx.demo.AmqpClient - amqp demo is started successfully, and will exit after 60s 
    10:59:46.405 [AmqpProvider :(3):[amqps://iot-cn-6******.amqp.iothub.aliyuncs.com:5671]] DEBUG org.apache.qpid.jms.provider.amqp.AmqpConsumer - Dispatching received message: JmsInboundMessageDispatch { sequence = 1, messageId = 1731508564705******, consumerId = ID:4e6d6f2b-1423-4c44-91ca-37202a******:1:1:1 }
    10:59:46.409 [JmsSession [ID:4e6d6f2b-1423-4c44-91ca-37202a******:1:1] delivery dispatcher] DEBUG org.apache.qpid.jms.provider.failover.FailoverProvider - Executing Failover Task: message acknowledge -> JmsInboundMessageDispatch { sequence = 1, messageId = 1731508564705******, consumerId = ID:4e6d6f2b-1423-4c44-91ca-37202******:1:1:1 } ackType: DELIVERED (5)
    10:59:46.432 [AmqpProvider :(3):[amqps://iot-cn-6******.amqp.iothub.aliyuncs.com:5671]] DEBUG org.apache.qpid.jms.provider.amqp.AmqpConsumer - Delivered Ack of message: JmsInboundMessageDispatch { sequence = 1, messageId = 1731508564705******, consumerId = ID:4e6d6f2b-1423-4c44-91ca-37202a******:1:1:1 }
    10:59:46.441 [JmsSession [ID:4e6d6f2b-1423-4c44-91ca-37202******:1:1] delivery dispatcher] DEBUG org.apache.qpid.jms.provider.failover.FailoverProvider - Executing Failover Task: message acknowledge -> JmsInboundMessageDispatch { sequence = 1, messageId = 1731508564705******, consumerId = ID:4e6d6f2b-1423-4c44-91ca-37202a******:1:1:1 } ackType: ACCEPTED (6)
    10:59:46.442 [AmqpProvider :(3):[amqps://iot-cn-6******.amqp.iothub.aliyuncs.com:5671]] DEBUG org.apache.qpid.jms.provider.amqp.AmqpConsumer - Accepted Ack of message: JmsInboundMessageDispatch { sequence = 1, messageId = 1731508564705******, consumerId = ID:4e6d6f2b-1423-4c44-91ca-37202a******:1:1:1 }
    10:59:46.452 [pool-1-thread-1] INFO com.aliyun.iotx.demo.AmqpClient - receive message,
     topic = /g18******/device01/thing/event/property/post,
     messageId = 1731508564705******,
     content = {"temperature":10,"humidity":56}

    The online AMQP clients are displayed in the corresponding consumer group. In amqp-demo, connectionCount = 4 indicates four clients.

    image.png

What to do next

  1. If the AMQP client is offline, messages for the AMQP server-side subscription accumulate. After the AMQP client comes back online, IoT Platform pushes the messages again. If you do not need to consume the stacked messages, you can clear them before the AMQP client goes online.

  2. After all configurations are complete and the device reports subscription data that is received by the AMQP client, you can log on to the IoT Platform console and go to the corresponding instance to view message operational logs.

    • On the Monitoring and O&M > Simple Log Service > IoT Platform Logs tab, you can view the log records of device data reporting, data forwarding from IoT Platform to the AMQP client, and acknowledgments (ACKs) returned by the AMQP client. For more information, see Query IoT Platform logs.

    • On the Message Forwarding > Server-side Subscription > Consumer Group List tab, click View in the Actions column of the target consumer group. On the consumer group details page, you can view the message consumption rate, message accumulation, consumption logs, and more. For more information, see View and monitor consumer groups.

FAQ