Message queue configuration
This topic explains how to configure event notifications for your source data storage service to send object change events to a message queue. Once configured, the migration service can receive these events from the message queue to create and run queue-based sync tasks.
This document contains information about third-party products for reference only. Alibaba Cloud makes no express or implied commitment regarding the performance and reliability of third-party products, or any potential impact caused by their operations.
Alibaba Cloud OSS
The migration service currently supports only a queue as an event destination. When you create a queue, ensure that it is in the same region as the OSS bucket.
Select event types based on your requirements. The migration service supports the following event types.
Group | Type | Description |
ObjectCreated (Create object) | ObjectCreated:PutObject | An object is uploaded using the |
ObjectCreated:PostObject | An object is uploaded using the | |
ObjectCreated:AppendObject | An object is created by appending data. | |
ObjectCreated:PutSymlink | A symbolic link is created for an object. | |
ObjectCreated:CopyObject | An object is copied. | |
ObjectCreated:CompleteMultipartUpload | A multipart upload is completed. | |
ObjectReplication (Replicate object) | ObjectReplication:ObjectCreated | A write operation occurred during data replication. |
ObjectReplication:ObjectModified | An overwrite operation occurred during data replication. | |
ObjectModified (Modify object) | ObjectModified:UpdateObjectMeta | The metadata of an object is modified by calling the |
For detailed instructions on how to create an event notification in OSS, see Use event notifications to process OSS object changes in real time.
The migration service requires the following permissions to access and consume messages from Message Service (MNS):
mns:GetQueueAttributesmns:ReceiveMessagemns:DeleteMessage
When you create a queue, note the following:
We recommend setting the message retention period to 7 days. This ensures that messages in the queue are not deleted by the queue policy before they are fully consumed.
We recommend setting the delivery delay to 10 seconds or more. This helps prevent
LastModifiedTimeupdate failures when uploading objects to OSS due to clock skew.If the data to be migrated contains large objects, we recommend setting an appropriate visibility timeout based on the bandwidth of your sync task.
The event notification rule takes about 10 minutes to take effect.
After creating the queue and configuring the event notification, upload a test object to the OSS bucket. This verifies that messages are correctly sent to the message queue.
After the migration service successfully consumes a message, it deletes the message from the queue. The migration service also deletes messages from the queue if their format, bucket, or prefix does not match the sync task.
Amazon S3
The migration service currently supports only event notifications from Amazon S3 to an Amazon SQS queue. When you create the queue, ensure that it is in the same region as the Amazon S3 bucket.
Select event types based on your requirements. The migration service supports the following event types.
Group | Type | Description |
ObjectCreated (Create object) | ObjectCreated:Put | An object is uploaded using the Put method. |
ObjectCreated:Post | An object is uploaded using the Post method. | |
ObjectCreated:Copy | An object is copied. | |
ObjectCreated:CompleteMultipartUpload | A multipart upload is completed. |
For detailed instructions on how to create an event notification in Amazon S3, see Amazon S3 Event Notifications.
The migration service requires the following permissions to access and consume messages from Amazon SQS:
sqs:ReceiveMessagesqs:DeleteMessagesqs:GetQueueAttributes
When you create a queue, note the following:
We recommend selecting the standard queue type.
We recommend setting the message retention period to 7 days or more. This ensures that messages in the queue are not deleted by the queue policy before they are fully consumed.
We recommend setting the delivery delay to 10 seconds or more. This helps prevent issues caused by clock skew during object uploads.
If the data to be migrated contains large objects, we recommend setting an appropriate visibility timeout based on the bandwidth of your sync task.
The migration service supports only SSE-SQS to encrypt messages in transit.
After creating the queue and configuring the event notification, upload a test object to the Amazon S3 bucket. This verifies that messages are correctly sent to the message queue.
After the migration service successfully consumes a message, it deletes the message from the queue. The migration service also deletes messages from the queue if their format, bucket, or prefix does not match the sync task.