Message queue configuration
Configure event notifications for your source data storage service to send object change events to a message queue. The migration service then receives these events to create and run queue-based sync tasks.
This document contains information about third-party products for reference only. Alibaba Cloud makes no express or implied commitment regarding the performance and reliability of third-party products, or any potential impact caused by their operations.
Alibaba Cloud OSS
The migration service currently supports only a queue as an event destination. When you create a queue, make sure that it is in the same region as the OSS bucket.
Select event types based on your requirements. The following event types are supported.
|
Group |
Type |
Description |
|
ObjectCreated (Create object) |
ObjectCreated:PutObject |
An object is uploaded by using the |
|
ObjectCreated:PostObject |
An object is uploaded by using the |
|
|
ObjectCreated:AppendObject |
An object is created by appending data. |
|
|
ObjectCreated:PutSymlink |
A symbolic link is created for an object. |
|
|
ObjectCreated:CopyObject |
An object is copied. |
|
|
ObjectCreated:CompleteMultipartUpload |
A multipart upload is completed. |
|
|
ObjectReplication (Replicate object) |
ObjectReplication:ObjectCreated |
A write operation occurs during data replication. |
|
ObjectReplication:ObjectModified |
An overwrite operation occurs during data replication. |
|
|
ObjectModified (Modify object) |
ObjectModified:UpdateObjectMeta |
Object metadata is modified by calling the |
For more information about creating event notifications in OSS, see Use event notifications to process OSS object changes in real time.
The migration service requires the following permissions to access and consume messages from Message Service (MNS):
-
mns:GetQueueAttributes -
mns:ReceiveMessage -
mns:DeleteMessage
-
When you create a queue, note the following:
-
Set the message retention period to 7 days. This ensures that messages in the queue are not deleted by the queue policy before they are fully consumed.
-
Set the delivery delay to 10 seconds or more. This helps prevent
LastModifiedTimeupdate failures when uploading objects to OSS due to clock skew. -
For data that contains large objects, set an appropriate visibility timeout based on the bandwidth of your sync task.
-
-
The event notification rule takes about 10 minutes to take effect.
-
After creating the queue and configuring the event notification, upload a test object to the OSS bucket to verify that messages are correctly sent to the message queue.
-
After the migration service successfully consumes a message, it deletes the message from the queue. The migration service also deletes messages whose format, bucket, or prefix does not match the sync task.
Amazon S3
The migration service currently supports only event notifications from Amazon S3 to an Amazon SQS queue. When you create the queue, make sure that it is in the same region as the Amazon S3 bucket.
Select event types based on your requirements. The following event types are supported.
|
Group |
Type |
Description |
|
ObjectCreated (Create object) |
ObjectCreated:Put |
An object is uploaded by using the Put method. |
|
ObjectCreated:Post |
An object is uploaded by using the Post method. |
|
|
ObjectCreated:Copy |
An object is copied. |
|
|
ObjectCreated:CompleteMultipartUpload |
A multipart upload is completed. |
For more information about creating event notifications in Amazon S3, see Amazon S3 Event Notifications.
The migration service requires the following permissions to access and consume messages from Amazon SQS:
-
sqs:ReceiveMessage -
sqs:DeleteMessage -
sqs:GetQueueAttributes
-
When you create a queue, note the following:
-
Select the standard queue type.
-
Set the message retention period to 7 days or more. This ensures that messages in the queue are not deleted by the queue policy before they are fully consumed.
-
Set the delivery delay to 10 seconds or more. This helps prevent issues caused by clock skew during object uploads.
-
For data that contains large objects, set an appropriate visibility timeout based on the bandwidth of your sync task.
-
The migration service supports only SSE-SQS for encrypting messages in transit.
-
-
After creating the queue and configuring the event notification, upload a test object to the Amazon S3 bucket to verify that messages are correctly sent to the message queue.
-
After the migration service successfully consumes a message, it deletes the message from the queue. The migration service also deletes messages whose format, bucket, or prefix does not match the sync task.