Search for messages

Billing

The ApsaraMQ for Kafka message search feature is in public preview and is independent of ApsaraMQ for Kafka instances. Therefore, you are not charged for this feature in ApsaraMQ for Kafka. The automatically created Tablestore instance and data table are also free of charge during the public preview period. Alibaba Cloud does not provide a Service-Level Agreement (SLA) for the message search feature during the public preview. For the SLAs and billing details of other products that the message search feature depends on, see the documentation for those products.

Notes

• When you enable message search for the first time, the Tablestore service is automatically enabled only in the same region. Currently, message search is available only in the China (Hangzhou) and China (Shenzhen) regions.
• When you enable message search for the first time, ApsaraMQ for Kafka automatically creates the service-linked role AliyunServiceRoleForAlikafkaConnector for the connector feature. If the service-linked role already exists, ApsaraMQ for Kafka does not create it again. For more information about service-linked roles, see Service-linked Role.
• By default, a single instance supports message search for up to three topics simultaneously. A maximum of 10 search results are displayed for each topic.
• Each retrieved message displays a maximum of 1 KB of content in the ApsaraMQ for Kafka console. Content that exceeds 1 KB is automatically truncated. To view the full message content, download the message.
• Currently, Tablestore does not support single fields larger than 2 MB. Messages that exceed 2 MB are not synchronized. Therefore, you cannot retrieve these messages on the Message Retrieval page in the ApsaraMQ for Kafka
• The data retention period in Tablestore is the same as the message retention period (time to live (TTL)) of the ApsaraMQ for Kafka instance. When data exceeds the message retention period, it is automatically purged, and the related indexes are removed. For more information about the configuration and description of the message retention period for ApsaraMQ for Kafka, see Change message configurations.

  Because the data expiration policies of Tablestore and ApsaraMQ for Kafka are not completely consistent, the actual data that you can retrieve may vary.

Enable message search

Enable the message search feature for a topic in an instance to search for messages in that topic as needed.

1. Log on to the ApsaraMQ for Kafka console.

2. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

3. In the left-side navigation pane, click Message Retrieval.
4. On the Message Retrieval page, select the instance that contains the topic from the Select Instance drop-down list, and then click Enable Message Retrieval.
5. In the Enable Message Retrieval panel, enter the parameters, and then click OK.
   Note If this is the first time you enable the message search feature for the current instance, click Enable Message Retrieval. In the You have not enabled the connector feature for the instance. dialog box, click Confirm. Then, enter the parameters in the Enable Message Retrieval panel.

   Table 1. Enablement parameters

   Parameter	Description	Example
   Data Source Topic	The topic for which to enable message search.	test
   Consumer Offset	The position from which to start consumption. The default value is Latest Offset. The following values are available: Earliest Offset: Start consumption from the earliest offset. Latest Offset: Start consumption from the latest offset.	Latest Offset
   Record Time	The time when the message is recorded. This is the time range for message search. The default value is Instant Time. The following values are available: Native Time: The message creation time recorded in ApsaraMQ for Kafka. This is the message creation time in the ProducerRecord that is included with the client or specified by you. Instant Time: The time when data is synchronized to Tablestore. This value indicates only the time when the message is synchronized to Tablestore, not the production time of the message. Therefore, the message time in the search results may be inconsistent with the message production time.	Native Time

   On the Message Retrieval page, you can view the topic for which you enabled the search feature.

Test sending messages

After you enable message search, you can send messages to the data source topic in ApsaraMQ for Kafka to test if the message search task was created successfully.

1. Log on to the ApsaraMQ for Kafka console.

2. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

3. In the left-side navigation pane, click Message Retrieval.
4. On the Message Retrieval page, find the target topic and perform an operation based on the task status.
   • If the task is in a state other than Running or Paused, click Test in the Actions column.
   • If the task is in the Running or Paused state, choose More > Test in the Actions
5. In the Start to Send and Consume Message panel, send a test message.
   1. In the Message Key text box, enter a key for the message, such as demo.
   2. In the Message Content text box, enter the content for the test message, such as {"key": "test"}.
   3. For Send to Specified Partition, specify whether to send the message to a specific partition.
      • Click Yes, and enter the partition ID in the Partition ID text box, such as 0. To find the partition ID, see View partition status.
      • Click No to not specify a partition.

Search for messages

1. Log on to the ApsaraMQ for Kafka console.

2. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

3. In the left-side navigation pane, click Message Retrieval.
4. On the Message Retrieval page, find the target topic and click Search in the Actions column.
5. In the Search panel, set the search criteria. Select a search item from the drop-down list, click Add Search Item, enter a value in the Value column, and then click OK.

   Search Item	Description
   Partition	The partition where the message content of the topic is located.
   Offset Range	The offset range of the message content in the topic.
   Key	The message key or value of the topic message content to search for. This is the key or value to match. The following describes how to use keywords for search: The search is a phrase match query. For example, if the message key is "Message Queue for Apache Kafka is a distributed, high-throughput, and scalable message queue service provided by Alibaba Cloud.", you can set the search keyword to "distributed", or a combination of "Alibaba Cloud" and "distributed". If the search keyword contains an asterisk (*) or a question mark (?), a wildcard query is used. The asterisk (*) represents any sequence of characters, and the question mark (?) represents any single character. The search is not case-sensitive for English letters. For example, if the message content is "AliKafkaTest001qaz", you can set the keyword to "AliKafkaTe*". Do not use a wildcard character as the first character of a keyword. This reduces search efficiency. A string with a wildcard character cannot exceed 20 characters in length. For more information about the usage and limits of tokenization, match phrase queries, and wildcard queries, see Tokenization, Match phrase query, and Wildcard query.
   Value
   Time Range	The time range to search for topic messages. You can set a time range within the last three days or a custom time. The minimum time granularity for the search is one minute.
   Actions	Click to remove the search condition. To remove all conditions, click Clear All above the list of search conditions.

   Note When you set multiple search criteria, the search returns results that match all criteria (an intersection).
   The retrieved messages are displayed on the Topic Search page.

   Table 2. Search result parameters

   Parameter	Description
   Partition	The topic partition of the message.
   Offset	The offset where the message is located.
   Key	The key of the message (forcefully converted to the string type).
   Value	The value of the message (forcefully converted to the string type), which is the specific content of the message.
   Created At	The time when the message was created or synchronized to Tablestore. The message creation time refers to the time specified in the ProducerRecord that is included with the client or specified by you when the message is sent. Note If this field is configured, the configured value is displayed. If this field is not configured, the system time when the message was sent is used by default. If the displayed value is 1970/x/x x:x:x, the sending time is configured as 0 or another incorrect value.
   Actions	Click Download Key: Downloads the key of the message. Click Download Value: Downloads the specific content of the message. Important Each retrieved message displays a maximum of 1 KB of content in the console. Content that exceeds 1 KB is automatically truncated. To view the full message content, download the message. Only Professional Edition instances support downloading messages. The maximum size of a downloadable message is 10 MB. If a message exceeds 10 MB, only the first 10 MB of content is downloaded.

View message search task details

After you enable message search, you can view details such as the automatically created topics, Groups, Tablestore instance name, and Tablestore data table name. You can also navigate to the Tablestore data table from the details page.

1. Log on to the ApsaraMQ for Kafka console.

2. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

3. In the left-side navigation pane, click Message Retrieval.
4. On the Message Retrieval page, find the target topic and click Details in the Actions column.
   On the task details page, you can view the details of the message search task for the target topic. You can also click Tablestore next to Destination Service in the Basic Information section to navigate to the data table details page.

View consumption details

You can view the consumption progress of online Groups that subscribe to the current topic in each partition to monitor message consumption and backlog.

1. Log on to the ApsaraMQ for Kafka console.

2. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

3. In the left-side navigation pane, click Message Retrieval.
4. On the Message Retrieval page, find the desired topic, and click Consumption Progress in the Actions column.
   On the consumption details page, you can view the consumption status of each partition in the topic.

Pause a message search task

To temporarily stop a running message search task for a topic, you can pause the task.

1. Log on to the ApsaraMQ for Kafka console.

2. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

3. In the left-side navigation pane, click Message Retrieval.
4. On the Message Retrieval page, find the target topic and choose More > Suspend in the Actions
5. In the dialog box that appears, click Confirm.

Enable a message search task

You can enable a paused message search task as needed.

1. Log on to the ApsaraMQ for Kafka console.

2. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

3. In the left-side navigation pane, click Message Retrieval.
4. On the Message Retrieval page, find the target topic and choose More > Enable in the Actions
5. In the dialog box that appears, click Confirm.

Delete a message search task

When you delete a message search task for a topic, the corresponding Tablestore data table and search index are also deleted. The message search feature is then no longer available for the topic. To use message search for this topic again, you must create a new task and wait for the data to synchronize.

1. Log on to the ApsaraMQ for Kafka console.

2. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

3. In the left-side navigation pane, click Message Retrieval.
4. On the Message Retrieval page, find the target topic and perform an operation based on the task status.
   • If the task is in a state other than Running or Paused, click Delete in the Actions column.
   • If the task is in the Running or Paused state, choose More > Delete in the Actions
5. In the dialog box that appears, click Confirm.
   The deleted topic is no longer visible on the Message Retrieval page.