Forward data to an AMQP consumer group
By default, an AMQP server-side subscription receives data from all devices that belong to a product. To gain more control, you can use the Data Forwarding feature. This feature lets you use an SQL statement to process and filter messages from all or specific devices before forwarding them to an AMQP server-side subscription consumer group. The messages are then consumed by an AMQP client.
How it works
Data Forwarding can forward messages from a specified topic for all or specific devices under the same product to one or more consumer groups in real time. Each consumer group can include multiple consumers, which are AMQP clients. When a message is forwarded to a consumer group, a random consumer in that group receives the message. Different consumer groups are identified by their consumer group IDs. In the preceding figure:
-
Consumer: an AMQP client that uses an AMQP SDK to register with a consumer group to receive messages that IoT Platform forwards.
-
IoT Platform: configure the data source topic, SQL statement, and data forwarding destination for a Data Forwarding rule, and then start the rule to forward device messages to the AMQP server-side subscription consumer group.
-
Data source: For information about the supported topic types, see Data formats.
Examples:
-
Data source Topic_Product A: forwards messages from all devices that belong to product A to a consumer group.
-
Data source Topic_Device B: forwards messages from a specified device, Device B, that belongs to product B to a consumer group.
-
-
SQL statement: write an SQL statement to parse and process the JSON data from device messages. For more information about the syntax of SQL statements, see SQL statements.
NoteBinary data is not parsed and is transmitted transparently.
-
Data forwarding destination: the AMQP server-side subscription consumer group that receives device data. For more information, see Manage AMQP consumer groups.
-
-
AMQP client: After you start the rule, IoT Platform automatically forwards device messages to the AMQP client. You do not need to subscribe to topics in the AMQP client code. The AMQP client receives messages as long as it is online.
Data Forwarding does not support sending messages from an AMQP client to a device. To send commands to a device, call the message communication APIs.
Use cases
Business servers receive device messages: Data Forwarding provides a flexible way to forward device messages to an AMQP server-side subscription consumer group.
-
Forward messages from specified devices.
-
Forward messages from specified topics.
-
Filter or process messages before forwarding.
To receive real-time messages from all devices that belong to a specified product on your AMQP client, you can directly configure an AMQP server-side subscription. For more information, see Configure an AMQP server-side subscription.
Limitations
-
If a client fails to authenticate within 15 seconds of connecting, the server closes the connection.
-
The maximum number of transactions per second (TPS) for a single connection of an AMQP client is 1,000. The TPS limit for message forwarding is determined by the message forwarding TPS specification of your instance. There is no limit on the message size. For more information about the limits of AMQP server-side subscriptions, see Limits on server-side subscriptions.
-
You cannot use the legacy Data Forwarding feature for products and devices on Cloud Gateway or MQTT instances. To forward data from these products and devices, you must use the new Data Forwarding feature. For more information, see Data Forwarding (New).
Prerequisites
-
A consumer group is created for the data forwarding destination. You can use the default consumer group (DEFAULT_GROUP) provided by IoT Platform or create a new one.
-
A Data Forwarding rule is created and an SQL statement is written to process the data.
Step 1: Configure a data forwarding destination
Log on to the IoT Platform console.
On the Overview page, find the instance that you want to manage and click the instance ID or instance name.
In the navigation pane on the left, choose .
Find the rule that you want to manage and click View in the Actions column. The Data Forwarding Rule page appears.
ImportantIf the new version of the data forwarding page is displayed, click Back to Old Version in the upper-right corner. Then, find the destination rule and click View.
In the Forward Data section, click Add Operation.
-
In the Add Operation dialog box, for Select Operation, select Publish to AMQP Subscribed Consumer Group. Configure the other parameters as prompted and click OK.
Parameter
Description
Select operation
Select Publish to AMQP Subscribed Consumer Group.
Consumer group
Select an existing consumer group as the data forwarding destination. You can click Create Consumer Group to create a consumer group.
Tag
If you set a tag, this operation adds the tag to all messages forwarded to the AMQP server-side subscription consumer group.
The tag must be 1 to 128 characters in length and can be a constant or a variable.
-
A constant can contain Chinese characters, letters, and digits.
-
A variable must be in the
${key}format. It represents the value of thekeyparameter in the JSON data returned by the SQL statement. If the value cannot be retrieved, no tag is added to the message.
-
Return to the Data Forwarding page. Find the rule and click Start to enable the rule.
Step 2: Run the AMQP client
Use the AMQP SDK access sample that is provided by Alibaba Cloud IoT Platform. Alibaba Cloud does not provide technical support for self-developed AMQP SDKs.
This example uses Java. For examples in other languages, see AMQP client access instructions.
This example uses an ECS instance with the Alibaba Cloud Linux operating system as the development environment for the AMQP client:
Log on to the ECS instance. For more information about logon methods, see Select an ECS remote connection method.
Run the following command to download the demo file.
wget https://linkkit-export.oss-cn-shanghai.aliyuncs.com/amqp/amqp-demo.zipRun the following command to decompress the demo file.
unzip amqp-demo.zipIn the
AmqpClient.javafile in thesrc/main/java/com.aliyun.iotx.demodirectory, modify the AMQP access information as described in the following table.ImportantThe demo code in this example includes code to terminate the program (
Thread.sleep(60 * 1000);). This means the program terminates one minute after it starts successfully. In a real-world scenario, you can set the runtime as needed.Parameter
Description
accessKey
The AccessKey ID and AccessKey secret of an Alibaba Cloud account or a Resource Access Management (RAM) user.
Log on to the IoT Platform console, move the pointer over your profile picture, and then click AccessKey Management to get the AccessKey ID and AccessKey secret.
ImportantTo avoid the security risks of hard coding an AccessKey into your business code, use environment variables to manage your AccessKey.
Set the ALIBABA_CLOUD_ACCESS_KEY_ID and ALIBABA_CLOUD_ACCESS_KEY_SECRET environment variables in your local operating system to your AccessKey ID and AccessKey secret, respectively.
In the sample code, you can retrieve them using the following methods:
System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID")System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET")
accessSecret
consumerGroupId
The ID of the consumer group in the corresponding instance of IoT Platform.
Log on to the IoT Platform console. In the corresponding instance, choose to view your consumer group ID.
iotInstanceId
The instance ID. You can view the ID of the current instance on the Instance Overview page of the IoT Platform console.
If an ID value exists, you must pass this ID value.
If there is no Instance Overview page or ID value, pass an empty value, which is
iotInstanceId = "".
clientId
The client ID. It is user-defined and cannot exceed 64 characters. Use a unique identifier such as the UUID, MAC address, or IP address of the server where your AMQP client is located.
After the AMQP client is connected and started, log on to the IoT Platform console. In the corresponding instance, choose . Click View for the consumer group. The Consumer Group Details page will display this parameter, making it easy for you to identify and distinguish different clients.
connectionCount
The number of connections for starting the AMQP client. The maximum is 128. This is used for scaling out real-time message pushes.
The Consumer Group Details page displays connected clients in the format
${clientId}+"-"+number. The minimum value for the number is 0.host
The AMQP access domain name.
For information about the AMQP access domain name corresponding to
${YourHost}, see View and configure instance endpoint information (Endpoint).The relevant Maven dependencies have been added to the
pom.xmlfile. In theamqp-demoroot directory, run the following command to reload Maven changes and build the project.mvn clean packageIn the
amqp-demo/targetdirectory, run the following command to run the generated JAR package.java -jar demo-0.0.1-SNAPSHOT-jar-with-dependencies.jarAfter you run the sample code, the following information is returned, which indicates that the AMQP client has connected to IoT Platform and is successfully receiving messages.
ImportantDevice messages can be received on the server only when the AMQP client is online.
10:42:43.254 [main] INFO com.aliyun.iotx.demo.AmqpClient - amqp demo is started successfully, and will exit after 60s 10:59:46.405 [AmqpProvider :(3):[amqps://iot-cn-6******.amqp.iothub.aliyuncs.com:5671]] DEBUG org.apache.qpid.jms.provider.amqp.AmqpConsumer - Dispatching received message: JmsInboundMessageDispatch { sequence = 1, messageId = 1731508564705******, consumerId = ID:4e6d6f2b-1423-4c44-91ca-37202a******:1:1:1 } 10:59:46.409 [JmsSession [ID:4e6d6f2b-1423-4c44-91ca-37202a******:1:1] delivery dispatcher] DEBUG org.apache.qpid.jms.provider.failover.FailoverProvider - Executing Failover Task: message acknowledge -> JmsInboundMessageDispatch { sequence = 1, messageId = 1731508564705******, consumerId = ID:4e6d6f2b-1423-4c44-91ca-37202******:1:1:1 } ackType: DELIVERED (5) 10:59:46.432 [AmqpProvider :(3):[amqps://iot-cn-6******.amqp.iothub.aliyuncs.com:5671]] DEBUG org.apache.qpid.jms.provider.amqp.AmqpConsumer - Delivered Ack of message: JmsInboundMessageDispatch { sequence = 1, messageId = 1731508564705******, consumerId = ID:4e6d6f2b-1423-4c44-91ca-37202a******:1:1:1 } 10:59:46.441 [JmsSession [ID:4e6d6f2b-1423-4c44-91ca-37202******:1:1] delivery dispatcher] DEBUG org.apache.qpid.jms.provider.failover.FailoverProvider - Executing Failover Task: message acknowledge -> JmsInboundMessageDispatch { sequence = 1, messageId = 1731508564705******, consumerId = ID:4e6d6f2b-1423-4c44-91ca-37202a******:1:1:1 } ackType: ACCEPTED (6) 10:59:46.442 [AmqpProvider :(3):[amqps://iot-cn-6******.amqp.iothub.aliyuncs.com:5671]] DEBUG org.apache.qpid.jms.provider.amqp.AmqpConsumer - Accepted Ack of message: JmsInboundMessageDispatch { sequence = 1, messageId = 1731508564705******, consumerId = ID:4e6d6f2b-1423-4c44-91ca-37202a******:1:1:1 } 10:59:46.452 [pool-1-thread-1] INFO com.aliyun.iotx.demo.AmqpClient - receive message, topic = /g18******/device01/thing/event/property/post, messageId = 1731508564705******, content = {"temperature":10,"humidity":56}The online AMQP clients are displayed in the corresponding consumer group. In
amqp-demo,connectionCount = 4indicates four clients.
What to do next
If the AMQP client is offline, messages for the AMQP server-side subscription accumulate. After the AMQP client comes back online, IoT Platform pushes the messages again. If you do not need to consume the stacked messages, you can clear them before the AMQP client goes online.
After all configurations are complete and the device reports subscription data that is received by the AMQP client, you can log on to the IoT Platform console and go to the corresponding instance to view message operational logs.
On the tab, you can view the log records of device data reporting, data forwarding from IoT Platform to the AMQP client, and acknowledgments (ACKs) returned by the AMQP client. For more information, see Query IoT Platform logs.
On the tab, click View in the Actions column of the target consumer group. On the consumer group details page, you can view the message consumption rate, message accumulation, consumption logs, and more. For more information, see View and monitor consumer groups.
FAQ
How do I implement device connection and status synchronization on IoT Platform?
Can hexadecimal data reported by a device be subscribed to through an AMQP server-side subscription?
Does one consumer group correspond to one AMQP message queue?
Can hexadecimal data reported by a device be subscribed to through an AMQP server-side subscription?
Why is there a delay when IoT Platform forwards messages through a consumer group?
Why doesn't the device receive messages published by the AMQP client?