Cross-region replication within the same account

更新时间:
复制 MD 格式

Cross-region replication within the same account automatically and asynchronously (near-real-time) replicates object operations, such as creation, updates, and deletions, from a source bucket in one region to a destination bucket in another region under the same account. This topic describes how to configure cross-region replication within the same account.

Prerequisites

  • You have created a bucket (Bucket A) in a specific region to serve as the source bucket. Record the account ID, the name of Bucket A, and its region.

  • You have created another bucket (Bucket B) in a different region under the same account to serve as the destination bucket. Record the name of Bucket B and its region.

Role types

When you configure cross-region replication, you must specify a role that Object Storage Service (OSS) can assume to replicate objects between the source and destination buckets. You can use any of the following role types.

Important

A RAM user must have the ram:CreateRole, ram:GetRole, ram:ListPoliciesForRole, and ram:AttachPolicyToRole permissions to create a role. However, granting role-related permissions such as ram:CreateRole and ram:GetRole to a RAM user poses a security risk. For better security, use your Alibaba Cloud account to create the RAM role and grant the necessary permissions. The RAM user can then directly use the role.

(Recommended) Create a new role

When you create a cross-region replication rule, you can choose to create a new role. A role is automatically created with a name in the format oss-replication-{uuid}. The permission policy attached to this role depends on whether you choose to replicate objects encrypted with Key Management Service (KMS).

  • Replicate KMS-encrypted objects

    After the role is created, follow the on-screen instructions to grant permissions. Once authorized, the role has a fine-grained permission policy for replicating from the source bucket to the destination bucket, as well as the AliyunKMSCryptoUserAccess policy for managing Key Management Service (KMS).

  • Do not replicate KMS-encrypted objects

    After the role is created, follow the on-screen instructions to grant permissions. Once authorized, the role has a fine-grained permission policy for replicating from the source bucket to the destination bucket.

AliyunOSSRole

When you create a cross-region replication rule, you can select AliyunOSSRole to complete the replication task. If you select this role, different permission policies are attached based on whether you choose to replicate KMS-encrypted objects.

  • Replicate KMS-encrypted objects

    If you select AliyunOSSRole, the AliyunOSSFullAccess (for managing Object Storage Service) and AliyunKMSCryptoUserAccess (for managing Key Management Service) policies are automatically attached to the role.

    Warning

    This role has full access to all buckets and KMS keys under your account. Because the permissions are extensive, use this role with caution.

  • Do not replicate KMS-encrypted objects

    If you select AliyunOSSRole, the AliyunOSSFullAccess policy (for managing Object Storage Service) is automatically attached to the role.

    Warning

    This role has full access to all buckets under your account. Because the permissions are extensive, use this role with caution.

Custom role

When you create a cross-region replication rule, you can use a custom role to complete the replication task. You must create a custom role in the RAM console and grant the required permissions to the role.

  1. Create a regular service role.

    During role creation, select Alibaba Cloud Service as the trusted entity and OSS as the trusted service. For more information, see Create a regular service role.

  2. Grant permissions to the role.

    You can grant permissions to the role in either of the following ways.

    Attach system policy

    Warning

    You can attach the AliyunOSSFullAccess system policy to the RAM role. By default, AliyunOSSFullAccess provides full permissions on all buckets under the current account. Use this policy with caution.

    If you want to replicate KMS-encrypted objects to the destination bucket, you must also attach the AliyunKMSFullAccess system policy to the role.

    For more information, see Manage permissions for a RAM role.

    Attach custom policy

    Use a RAM policy to grant the minimum permissions required for replication from the source bucket (src-bucket) to the destination bucket (dest-bucket).

    Note

    Replace the names of the source and destination buckets with your actual bucket names.

    {
       "Version":"1",
       "Statement":[
          {
             "Effect":"Allow",
             "Action":[
                "oss:ReplicateList",
                "oss:ReplicateGet"
             ],
             "Resource":[
                "acs:oss:*:*:src-bucket",
                "acs:oss:*:*:src-bucket/*"
             ]
          },
          {
             "Effect":"Allow",
             "Action":[
                "oss:ReplicateList",
                "oss:ReplicateGet",
                "oss:ReplicatePut",
                "oss:ReplicateDelete"
             ],
             "Resource":[
                "acs:oss:*:*:dest-bucket",
                "acs:oss:*:*:dest-bucket/*"
             ]
          }
       ]
    }

    For more information, see Manage permissions for a RAM role.

    Note

    If you want to replicate KMS-encrypted objects to the destination bucket, you must also attach the AliyunKMSFullAccess system policy to the role.

Important

When replicating data across regions within the same account, OSS validates only the permission policy of the RAM role used for replication. It does not check the bucket policies of the source or destination buckets.

Procedure

OSS console

  1. Log on to the OSS console.

  2. In the left-side navigation pane, click Buckets. Then, click the name of the source bucket.

  3. In the left-side navigation pane, choose Data Management > CRR.

  4. On the CRR tab, click CRR.

  5. In the CRR panel, configure the parameters as described in the following table.

    Section

    Parameter

    Description

    Configure Destination Bucket

    Source Bucket

    The region and name of the source bucket.

    Destination Bucket

    Select Select a bucket that belongs to the Alibaba Cloud account. Then, from the drop-down lists, select the region and name of the destination bucket.

    Objects to Replicate

    Select the source objects to replicate.

    • Synchronize all files: Replicate all objects from the source bucket to the destination bucket.

    • Objects with Specified Prefix: Replicate objects with a specified prefix to the destination bucket. You can add up to 10 prefixes. To increase this limit, contact Technical Support to increase the limit to a maximum of 100.

    Object Tagging

    Note

    To configure this parameter, you must meet the following conditions:

    • You have configured object tags.

    • The Replicate Deletion Markers and Replicate Deletions of Specific Object Versions check boxes are cleared.

    Select the Configure Rules check box to replicate objects that have specific tags to the destination bucket. You can add up to 10 tags (key-value pairs). After you add tags, select one of the following filtering policies:

    • Include all tags: Replicate an object only if all of its tags are included in the set of tags defined in the filter rule.

    • Include any one tag: Replicate an object if at least one of its tags is included in the set of tags defined in the filter rule.

      Note

      Currently, tag-based filtering is not supported in the finance cloud regions of China (Shenzhen) and China (Shanghai).

    Replicate KMS-Encrypted Source Objects

    Select whether to replicate KMS-encrypted objects to the destination bucket.

    • Yes: Select this option to replicate objects to the destination bucket if the source objects or the destination bucket are encrypted using a KMS-managed key (SSE-KMS with a specified CMK ID).

      Note

      You can call the HeadObject and GetBucketEncryption operations to query the encryption status of the source object and the destination bucket, respectively.

    • No: Do not replicate KMS-encrypted objects to the destination bucket.

    CMK ID

    Specify the KMS key to encrypt the destination objects.

    You must first create a KMS key in the KMS console in the same region as the destination bucket. For more information, see Create a key.

    RAM Role

    We recommend that you select New RAM Role. From the drop-down list, select this option and follow the on-screen instructions to grant permissions.

    You can also select AliyunOSSRole or a custom role. For more information about these role types, see Role types.

    Configure Replication Policy

    Replicate Historical Data

    Select whether to replicate existing objects that were in the source bucket before the cross-region replication rule took effect.

    • Yes: Replicate existing historical data to the destination bucket.

      Important

      When you replicate historical data, objects from the source bucket might overwrite objects with the same name in the destination bucket. To prevent data loss, we recommend that you enable versioning for both the source and destination buckets.

    • No: Replicate only objects that are uploaded or updated after the cross-region replication rule takes effect.

    Copy Delete Operation

    Select whether to replicate delete operations from the source bucket to the destination bucket.

    Note

    This option appears if versioning is not enabled on the source bucket. If versioning is enabled on the source bucket, this option is replaced by two separate settings: Copy Delete Marker and Copy Delete Operation of Specified Version.

    • Yes (for scenarios where you need to share and access the same dataset): Replicate create, update, and delete operations on objects in the source bucket to the destination bucket.

      Important

      With this policy, all create, update, and delete operations are replicated. When you delete an object from the source bucket, either manually or through a lifecycle rule, the corresponding object in the destination bucket is also deleted and cannot be recovered.

    • No (for disaster recovery scenarios): Replicate only create and update operations on objects in the source bucket. Delete operations do not affect the destination bucket.

      Note

      This prevents accidental data loss in the destination bucket caused by manual deletions or automated lifecycle rule deletions in the source bucket.

    Copy Delete Marker

    Select whether to replicate delete markers from the source bucket to the destination bucket.

    • Replicate: When an object is deleted from the source bucket without a version ID specified, OSS creates a delete marker in the source bucket, which is then replicated to the destination bucket. This is suitable for scenarios where you share and access the same dataset and need to ensure that the source and destination buckets are in a consistent state.

      Important

      With this policy, when you delete an object from the source bucket, either manually or through a lifecycle rule, a delete marker is replicated to the destination bucket, making the data inaccessible there as well.

    • Do not replicate (for disaster recovery scenarios): Delete markers created in the source bucket are not replicated to the destination bucket. This effectively prevents data loss in the destination bucket caused by accidental deletions or automated lifecycle rule deletions in the source bucket.

    Copy Delete Operation of Specified Version

    Select whether to replicate the permanent deletion of a specific object version from the source bucket to the destination bucket.

    Note

    After you create a data replication rule, changes to the storage class of objects in the source bucket that are caused by lifecycle rules or CopyObject operations are not synchronized to the destination bucket. Additionally, the last access time (x-oss-last-access-time) attribute of objects in the source bucket is not synchronized.

    • Replicate: When a specific version of a source object, including the current and noncurrent versions, is permanently deleted, the corresponding version in the destination bucket is also permanently deleted. This is suitable for scenarios that require source and destination data to be completely identical.

      Important

      With this policy, an object version that is permanently deleted from the source bucket cannot be recovered from the destination bucket. Choose this option with caution.

    • Do not replicate (for disaster recovery scenarios): When a specific version of a source object is permanently deleted, the corresponding version in the destination bucket is not deleted. This protects the data in the destination bucket from permanent delete operations in the source bucket.

    If an Object is uploaded to the source Bucket by using multipart upload, the upload operation for each part is replicated to the destination Bucket. Finally, the Object that is created after the CompleteMultipartUpload operation is performed on all parts is also replicated to the destination Bucket.

    For more information about replication behavior when versioning is used, see Replication with versioning.

    Configure Replication Speed

    Acceleration Type

    Only Transfer Acceleration is supported. Transfer acceleration improves the speed of cross-region replication between regions inside and outside the Chinese mainland. If you enable transfer acceleration, you are charged an additional fee. For more information about billing, see Transfer acceleration fees.

    RTC

    Note

    You can enable RTC for CRR tasks only between the following regions in the Chinese mainland: China (Hangzhou), China (Shanghai), China (Qingdao), China (Beijing), China (Zhangjiakou), and China (Shenzhen).

    You can enable RTC for CRR tasks only between the US (Silicon Valley) and US (Virginia) regions.

    For tasks that do not involve replicating historical data, RTC takes effect within 15 minutes of being enabled. For tasks that do involve replicating historical data, RTC takes effect about one hour after the historical data replication is complete. Once it takes effect, RTC ensures that 99.99% of newly written objects are replicated within 10 minutes. Enabling RTC incurs RTC fees.

  6. Click OK, and then in the dialog box that appears, click Enable.

    • After a cross-region replication rule is created, you cannot edit or delete it.

    • The replication task starts 3 to 5 minutes after the cross-region replication rule is configured. You can view the replication progress on the CRR tab of the source bucket.

    • Because cross-region replication is asynchronous, the time required to replicate data to the destination bucket depends on the data size, typically ranging from a few minutes to several hours.

Alibaba Cloud SDK

Only the Java, Python, and Go SDKs support cross-region replication within the same account.

Java

import com.aliyun.oss.*;
import com.aliyun.oss.common.auth.*;
import com.aliyun.oss.common.comm.SignVersion;
import com.aliyun.oss.model.AddBucketReplicationRequest;

public class Demo {

    public static void main(String[] args) throws Exception {
        // The endpoint of the China (Hangzhou) region is used in this example. Replace it with the actual endpoint of your region.
        String endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
        // Specify the region that corresponds to the endpoint, for example, cn-hangzhou.
        String region = "cn-hangzhou";
        // We strongly recommend that you do not hard-code your access credential in your code. This can lead to credential leaks and compromise the security of all resources in your account. This example uses environment variables to obtain the access credential. Before running this code, configure the environment variables.
        EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider();
        // Specify the name of the source bucket.
        String bucketName = "src-bucket";
        // Specify the destination bucket to which you want to replicate data. The destination and source buckets must belong to the same account.
        String targetBucketName = "dest-bucket";
        // Specify the region where the destination bucket is located. The destination and source buckets must be in different regions.
        String targetBucketLocation = "oss-cn-shanghai";

        // Create an OSSClient instance.
        // Call the shutdown method to release resources when the OSSClient instance is no longer used.
        ClientBuilderConfiguration clientBuilderConfiguration = new ClientBuilderConfiguration();
        // Explicitly declare the use of the V4 signature algorithm.
        clientBuilderConfiguration.setSignatureVersion(SignVersion.V4);
        OSS ossClient = OSSClientBuilder.create()
                .endpoint(endpoint)
                .credentialsProvider(credentialsProvider)
                .clientConfiguration(clientBuilderConfiguration)
                .region(region)
                .build();

        try {
            AddBucketReplicationRequest request = new AddBucketReplicationRequest(bucketName);
            request.setTargetBucketName(targetBucketName);
            request.setTargetBucketLocation(targetBucketLocation);
            // By default, historical data is replicated. Set this to false to disable historical data replication.
            request.setEnableHistoricalObjectReplication(false);
            // Specify the name of the role that authorizes OSS to perform data replication. This role must have been granted permissions to perform cross-region replication on the source bucket and to receive replicated objects in the destination bucket.
            request.setSyncRole("yourRole");
            // Specify whether OSS replicates objects created with SSE-KMS encryption.
            //request.setSseKmsEncryptedObjectsStatus("Enabled");
            // Specify the SSE-KMS key ID. This element is required if Status is set to Enabled.
            //request.setReplicaKmsKeyID("3542abdd-5821-4fb5-a425-90adca***");
            //List prefixes = new ArrayList();
            //prefixes.add("image/");
            //prefixes.add("video");
            //prefixes.add("a");
            //prefixes.add("A");
            // Specify the prefix of the objects to be replicated. If you specify a prefix, only objects with that prefix are replicated to the destination bucket.
            //request.setObjectPrefixList(prefixes);
            //List actions = new ArrayList();
            //actions.add(AddBucketReplicationRequest.ReplicationAction.PUT);
            // Replicate create and update operations on objects in the source bucket to the destination bucket.
            //request.setReplicationActionList(actions);
            ossClient.addBucketReplication(request);
        } catch (OSSException oe) {
            System.out.println("Caught an OSSException, which means your request made it to OSS, "
                    + "but was rejected with an error response for some reason.");
            System.out.println("Error Message:" + oe.getErrorMessage());
            System.out.println("Error Code:" + oe.getErrorCode());
            System.out.println("Request ID:" + oe.getRequestId());
            System.out.println("Host ID:" + oe.getHostId());
        } catch (ClientException ce) {
            System.out.println("Caught an ClientException, which means the client encountered "
                    + "a serious internal problem while trying to communicate with OSS, "
                    + "such as not being able to access the network.");
            System.out.println("Error Message:" + ce.getMessage());
        } finally {
            if (ossClient != null) {
                ossClient.shutdown();
            }
        }
    }
}        

Python

# -*- coding: utf-8 -*-
import oss2
from oss2.credentials import EnvironmentVariableCredentialsProvider
from oss2.models import ReplicationRule
# Obtain the access credential from environment variables. Before running this example code, ensure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are set.
auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
# Specify the endpoint of the region where the source bucket is located. For example, for the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com.
# Specify the name of the source bucket, for example, src-bucket.
bucket = oss2.Bucket(auth, 'https://oss-cn-hangzhou.aliyuncs.com', 'src-bucket')
replica_config = ReplicationRule(
    # Specify the destination bucket to which you want to replicate data. The destination and source buckets must belong to the same account.
    target_bucket_name='dest-bucket',
    # Specify the region where the destination bucket is located. The destination and source buckets must be in different regions.
    target_bucket_location='oss-cn-shanghai',
    # Specify the name of the role that authorizes OSS to perform data replication. This role must have been granted permissions to perform cross-region replication on the source bucket and to receive replicated objects in the destination bucket.
    sync_role_name='roleNameTest',
)

# Specify the prefix of the objects to be replicated. If you specify a prefix, only objects with that prefix are replicated to the destination bucket.
# prefix_list = ['prefix1', 'prefix2']
# Configure the data replication rule.
# replica_config = ReplicationRule(
     # prefix_list=prefix_list,
     # Replicate create and update operations on objects in the source bucket to the destination bucket.
     # action_list=[ReplicationRule.PUT],
     # Specify the destination bucket to which you want to replicate data. The destination and source buckets must belong to the same account.
     # target_bucket_name='dest-bucket',
     # Specify the region where the destination bucket is located. The destination and source buckets must be in different regions.
     # target_bucket_location='yourTargetBucketLocation',
     # By default, historical data is replicated. Set this to `False` to disable historical data replication.
     # is_enable_historical_object_replication=False,
     # Specify the data transfer link used for data replication.
     # target_transfer_type='oss_acc',    
  #)

# Enable data replication.
bucket.put_bucket_replication(replica_config)

Go

package main

import (
	"context"
	"flag"
	"log"

	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
)

// Define global variables.
var (
	region     string // Region in which the bucket is located.
	bucketName string // Name of the bucket.
)

// Specify the init function used to initialize command line parameters.
func init() {
	flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
	flag.StringVar(&bucketName, "bucket", "", "The name of the bucket.")
}

func main() {
	// Parse command line parameters.
	flag.Parse()

	var (
		targetBucket   = "target bucket name" // Name of the destination bucket.
		targetLocation = "oss-cn-beijing"     // Region in which the destination bucket is located.
	)

	// Check whether the name of the bucket is specified.
	if len(bucketName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, bucket name required")
	}

	// Check whether the region is specified.
	if len(region) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, region required")
	}

	// Load the default configurations and specify the credential provider and region.
	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	// Create an OSS client.
	client := oss.NewClient(cfg)

	// Create a request to enable data replication for the bucket.
	request := &oss.PutBucketReplicationRequest{
		Bucket: oss.Ptr(bucketName), // Name of the bucket.
		ReplicationConfiguration: &oss.ReplicationConfiguration{
			Rules: []oss.ReplicationRule{
				{
					RTC: &oss.ReplicationTimeControl{
						Status: oss.Ptr("enabled"), // Enable the RTC feature.
					},
					Destination: &oss.ReplicationDestination{
						Bucket:       oss.Ptr(targetBucket),   // Name of the destination bucket.
						Location:     oss.Ptr(targetLocation), // Region in which the destination bucket is located.
						TransferType: oss.TransferTypeOssAcc,  // Type of transfer.
					},
					HistoricalObjectReplication: oss.HistoricalObjectReplicationEnabled, // Enable the historical data replication feature.
				},
			},
		},
	}

	// Enable data replication.
	result, err := client.PutBucketReplication(context.TODO(), request)
	if err != nil {
		log.Fatalf("failed to put bucket replication %v", err)
	}

	// Display the result.
	log.Printf("put bucket replication result:%#v\n", result)
}

ossutil CLI

For detailed instructions on how to use the command-line tool ossutil to enable cross-region replication, see put-bucket-replication.

REST API

If your application has specific requirements, you can make REST API requests directly. This requires you to manually write code to calculate the signature. For more information, see PutBucketReplication.

Related topics