This topic answers frequently asked questions about same-account and cross-account data replication (including same-region and cross-region replication) and describes troubleshooting methods.
Can I specify a custom path or prefix for the destination bucket?
No. Same-region and cross-region replication do not support custom storage paths for the destination bucket. The destination path is determined by the data path of the source bucket configured in the data replication rule.
Can I replicate objects based on their last modified time?
No. You cannot configure a data replication rule to replicate objects based on a specific last-modified time range. However, you can use the File Modified Time feature of the online migration service to do this. If you specify a time range, only the files modified within that period are migrated.
Why am I unable to create a data replication rule?
-
Check whether the required permissions are granted.
-
Missing permissions for a RAM user
-
A RAM user cannot click OK when creating a data replication rule in the console.
-
Cause: The
oss:PutBucketReplicationpermission is not granted. -
Solution: Grant the
oss:PutBucketReplicationpermission to the RAM user.
-
-
When a RAM user creates a data replication rule in the console, custom RAM roles are not displayed in the RAM role list.
-
Cause: The
ram:ListRolespermission is not granted. -
Solution: Grant the
ram:ListRolespermission to the RAM user.
-
For more information, see Grant custom policies to a RAM user.
-
-
Missing permissions for a RAM role
-
Same-account replication
For same-account replication, you must grant permissions to the RAM role to perform replication operations between the source and destination buckets. For more information about how to create and authorize a RAM role, see Role types.
-
Cross-account replication
For cross-account replication, you must use Account A to grant a RAM role permissions to replicate data from the source bucket, and use Account B to grant a RAM role permissions to receive replicated objects in the destination bucket. For more information about how to create and authorize a RAM role, see Role authorization.
-
-
-
Check whether the source and destination buckets have the same versioning status.
The source and destination buckets for data replication must have the same versioning status: either both have versioning enabled, or both are unversioned.
-
Check whether the endpoint or AccessKey information is correct.
If you use an SDK or ossutil to create a data replication rule, check the following configurations:
-
Check whether the region endpoints for the source and destination buckets are correct. For more information, see Regions and endpoints.
-
Check whether the AccessKey pair is correct. For more information, see View the AccessKey information of a RAM user.
-
Why are objects not replicated to the destination bucket?
If object replicas do not appear in the destination bucket after you configure a data replication rule, check the following possible causes.
-
The source bucket is incorrectly configured.
-
Check whether the data replication status is Enabled.
-
The prefix is incorrect.
-
To replicate specific objects from the source bucket to the destination bucket, set the prefix to
log, only objects with thelogprefix, such aslog/date1.txtandlog/date2.txt, are replicated. Objects that do not match the specified prefix, such asdate3.txt, are not replicated.NoteDo not add an asterisk (*) at the end of the prefix, such as
log/*. The prefix cannot contain the bucket name. -
To replicate all objects from the source bucket to the destination bucket, leave the prefix field empty.
-
-
-
Check whether replication of historical objects is enabled and whether the missing objects are historical objects.
-
Check whether the objects in the source bucket are replicas from another bucket.
If an object in a bucket is a replica created by another replication configuration, OSS does not replicate that object. For example, if you configure replication from bucket A to bucket B and from bucket B to bucket C, OSS does not replicate the object replicas from bucket B to bucket C.
-
Check whether the objects are encrypted with KMS. If so, you must enable replication for KMS-encrypted objects.
If the source object or destination bucket uses server-side encryption with KMS-managed keys (SSE-KMS) and a specified CMK ID, you must select Copy and configure the following parameters to replicate the object to the destination bucket:
-
CMK ID: Specify the KMS key to encrypt the destination object.
You must first create a KMS key in the same region as the destination bucket. For more information, see Create a key.
-
RAM Role Name: Select a RAM role to perform KMS encryption on the destination object.
-
New RAM Role: Create a new RAM role to perform KMS encryption on the destination object. The role name must be in the format of
kms-replication-SourceBucketName-DestinationBucketName. -
AliyunOSSRole: Use the AliyunOSSRole role to perform KMS encryption on the destination object. If you have not created this role, OSS automatically creates it when you select this option.
NoteIf you create a role or modify the permissions of a role, make sure to grant the role the
AliyunOSSFullAccesspermission. Otherwise, data replication may fail. -
You can call the HeadObject and GetBucketEncryption operations to query the encryption status of the source object and the destination bucket, respectively.
-
-
Check whether the replication progress is 100%.
Data replication is an asynchronous (near real-time) process. Replicating data to the destination bucket can take several minutes to several hours, depending on the data size. If you are replicating a large object, allow some time for the process to complete. After the replication progress reaches 100%, check whether the object appears in the destination bucket.
Why are object deletions not replicated?
-
Cause 1: A retention policy (WORM) is configured for the destination bucket.
Before the retention period expires, no user, including the resource owner, can delete objects from the bucket.
-
Cause 2: Whether objects are deleted from the destination bucket depends on the versioning status of the source bucket and the configured data replication policy.
Source bucket versioning
Request method
Data replication policy
Result
Unversioned
A Delete request is initiated.
Replicate additions and modifications
Only the object in the source bucket is deleted. The object in the destination bucket is not deleted.
Replicate additions, modifications, and deletions
The objects in both the source and destination buckets are deleted.
Versioning enabled
A Delete request is initiated without an object version ID.
Replicate additions and modifications
No object is deleted from either bucket. OSS creates a delete marker in the source bucket and replicates the delete marker to the destination bucket.
Replicate additions, modifications, and deletions
A Delete request is initiated with an object version ID.
Replicate additions and modifications
Only the object version in the source bucket is deleted. The object version in the destination bucket is not deleted.
Replicate additions, modifications, and deletions
The specified object versions in both the source and destination buckets are deleted.
Verify data consistency
You can run the following code to verify data consistency between the source and destination buckets after replication is complete.
import com.aliyun.oss.OSSClient;
import com.aliyun.oss.common.auth.*;
import com.aliyun.oss.model.*;
import com.aliyun.oss.OSSException;
import com.aliyuncs.exceptions.ClientException;
public class Demo {
public static void main(String[] args) throws ClientException {
// Obtain access credentials from environment variables. Before you run this sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are set.
EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider();
// Set srcEndpoint to the endpoint of the region where the source bucket is located.
String srcEndpoint = "https://oss-cn-hangzhou.aliyuncs.com";
OSSClient srcClient = new OSSClient(srcEndpoint , credentialsProvider);
// Specify the name of the source bucket.
String srcBucketName = "src-replication-bucket";
// Set destEndpoint to the endpoint of the region where the destination bucket is located.
String destEndpoint = "https://oss-cn-beijing.aliyuncs.com";
OSSClient destClient = new OSSClient(destEndpoint, credentialsProvider);
// Specify the name of the destination bucket.
String destBucketName = "dest-replication-bucket";
// If the source and destination buckets do not have versioning enabled, call listObjectsV2 to list the replicated objects from the source bucket.
// If versioning is enabled or suspended for the source and destination buckets, call listVersions to list the replicated objects from the source bucket.
ListObjectsV2Result result;
ListObjectsV2Request request = new ListObjectsV2Request(srcBucketName);
do {
result = srcClient.listObjectsV2(request);
for (OSSObjectSummary summary : result.getObjectSummaries())
{
String objectName = summary.getKey();
ObjectMetadata srcMeta;
try {
// Get the metadata of the source object.
srcMeta = srcClient.headObject(srcBucketName, objectName);
} catch (OSSException ossException) {
if (ossException.getErrorCode().equals("NoSuchKey")) {
continue;
} else {
System.out.println("head src-object failed: " + objectName);
}
continue;
}
ObjectMetadata destMeta;
try {
// Get the metadata of the object replicated to the destination bucket.
destMeta = destClient.headObject(destBucketName, objectName);
} catch (OSSException ossException) {
if (ossException.getErrorCode().equals("NoSuchKey")) {
System.out.println("dest-object not exist: " + objectName);
} else {
System.out.println("head dest-object failed: " + objectName);
}
continue;
}
// Check whether the CRC values of the source and destination objects are consistent.
Long srcCrc = srcMeta.getServerCRC();
String srcMd5 = srcMeta.getContentMD5();
if (srcCrc != null) {
if (destMeta.getServerCRC() != null) {
if (!destMeta.getServerCRC().equals(srcCrc)) {
System.out.println("crc not equal: " + objectName
+ " | srcCrc: " + srcCrc + " | destCrc: " + destMeta.getServerCRC());
}
continue;
}
}
// Check whether the MD5 values of the source and destination objects are consistent.
if (srcMd5!= null) {
if (destMeta.getContentMD5() != null) {
if (!destMeta.getContentMD5().equals(srcMd5)) {
System.out.println("md5 not equal: " + objectName
+ " | srcMd5: " + srcMd5 + " | destMd5: " + destMeta.getContentMD5());
}
continue;
}
}
// Check whether the ETag values of the source and destination objects are consistent.
if (srcMeta.getETag() == null || !srcMeta.getETag().equals(destMeta.getETag())) {
System.out.println("etag not equal: " + objectName
+ " | srcEtag: " + srcMeta.getETag() + " | destEtag: " + destMeta.getETag());
}
}
request.setContinuationToken(result.getNextContinuationToken());
request.setStartAfter(result.getStartAfter());
} while (result.isTruncated());
}
}
Is chained replication supported?
No. For example, if you configure a data replication rule from bucket A to bucket B and another rule from bucket B to bucket C, data from bucket A is replicated only to bucket B, not to bucket C.
To replicate data from bucket A to bucket C, you must also configure a separate data replication rule from bucket A to bucket C.
If historical data replication is enabled for both bucket A and bucket B and is still in progress, new data written to bucket A might be scanned by the historical replication task and replicated to bucket C.
Does two-way replication between buckets create a replication loop?
No. If you configure two-way replication between bucket A and bucket B, replicated data is never sent back to its origin. For example, data replicated from A to B is not then replicated back to A.
Are lifecycle rule deletions replicated?
-
If the data replication policy for the source bucket is set to Replicate additions and modifications, objects deleted from the source bucket by a lifecycle rule are not deleted from the destination bucket.
-
If the data replication policy for the source bucket is set to Replicate additions, modifications, and deletions, objects deleted from the source bucket by a lifecycle rule are also deleted from the destination bucket.
NoteIf you find objects in the destination bucket that share names with objects deleted from the source bucket by a lifecycle rule, this does not indicate a policy failure. It is possible that an object with the same name was manually written to the destination bucket.
Why is historical replication progress stuck at 0%?
-
The replication progress is not updated in real time.
The replication progress for historical data is updated only after all files are scanned. If your bucket contains a very large number of objects (for example, hundreds of millions), it may take several hours for the progress to update. A progress of 0% does not necessarily mean that historical data is not being replicated.
You can confirm whether historical data replication has started by checking for changes in the destination bucket's storage capacity and traffic. For more information about how to view the storage capacity of and traffic to and from the destination bucket, see Query bucket-level usage.
-
The authorization policy of the source bucket is incorrect.
When you create a data replication rule, OSS does not check the authorization policy of the source bucket. This allows the rule to be created successfully even if the permissions are incorrect. However, data will not be replicated, and the replication progress will remain at 0%.
For information about how to configure the correct permission policy, see Permissions required for data replication.
What to do if data replication is slow?
-
Increase bandwidth
Data replication is an asynchronous (near real-time) process. The time required to transfer data from the source bucket to the destination bucket can range from several minutes to several hours, depending on the data volume. If the replication process takes too long, check if a bandwidth limit is the cause. If so, contact technical support to request a bandwidth increase to improve replication efficiency.
-
Enable Replication Time Control (RTC)
With RTC enabled, OSS replicates most objects within seconds and 99.99% of objects within 10 minutes. After you enable RTC, you are charged for the data replication traffic generated by RTC-enabled tasks. For more information, see Use Replication Time Control (RTC).
How to track replication operations?
You can configure an event notification rule and set the event type to ObjectReplication:ObjectCreated, ObjectReplication:ObjectRemoved, and ObjectReplication:ObjectModified to receive notifications of object changes in the source and destination buckets, such as creations, updates, deletions, and overwrites. For more information, see Use event notifications to monitor OSS object changes in real time.
Is replication supported on suspended-versioning buckets?
No. Data replication can be enabled only between two buckets that are unversioned or two buckets that have versioning enabled.
Are KMS API calls for replication billable?
Yes. If the destination bucket uses KMS encryption, you are charged for the corresponding KMS API calls. For more information about billing, see Billing of KMS 1.0.
Can I disable a replication rule?
Yes. You can click Disable Replication next to the rule to stop data replication.
After you disable the rule, existing replicated data remains in the destination bucket, but new data written to the source bucket is no longer replicated.
Is replication from a no-region bucket billable?
Yes. When you perform cross-region replication for a bucket with no region attribute, you are charged for cross-region traffic, requests, and other fees.
Does replication change the order of files?
Replication does not change the modification order of files.