Same-account Same-Region Replication

更新时间:
复制 MD 格式

Same-account same-region replication refers to automatically and asynchronously (near real-time) replicating the creation, update, and deletion operations of files (Objects) in a source bucket in a specific account and region to a target bucket in the same region of the current account. This topic describes how to configure same-account same-region replication.

Prerequisites

  • Bucket A (source bucket) has been created in a specific account and region, and the account UID, Bucket A name, and region have been recorded.

  • Bucket B (target bucket) has been created in the same account and region, and the Bucket B name has been recorded.

Role Types

When performing same-account same-region replication, you need to specify a role for executing replication operations between the source bucket and target bucket. You can choose any of the following roles to complete the same-account same-region replication task.

Important

You can choose to create roles through RAM users. RAM users must have the following permissions: ram:CreateRoleram:GetRoleram:ListPoliciesForRoleram:AttachPolicyToRole. Considering that granting RAM users ram:CreateRole, ram:GetRole and other role-related permissions carries high risk, we recommend that you create RAM roles and complete role authorization through the Alibaba Cloud account associated with the RAM user. After authorization is complete, RAM users can directly reuse RAM roles created by the Alibaba Cloud account.

(Recommended) Create New Role

When creating replication rules for the same account and region, you can choose to create a new role to complete the replication task. After selecting create new role, the system will automatically create a role in the format oss-replication-{uuid} and grant different permission policies based on whether you choose to replicate KMS encrypted objects.

  • Choose to replicate KMS encrypted objects

    After creating a new role, you need to complete the role authorization operation according to the page guidance. After authorization is completed, the role has precise permission policies for synchronizing from source Bucket to target Bucket as well as AliyunKMSCryptoUserAccess (permission to manage Key Management Service KMS).

  • Choose not to replicate KMS encrypted objects

    After creating a new role, you need to complete the role authorization operation according to the page guidance. After authorization is completed, the role has precise permission policies for synchronizing from source Bucket to target Bucket.

AliyunOSSRole

When creating replication rules for the same account in the same region, you can choose the AliyunOSSRole to complete the replication task. After selecting this role, the backend will grant different permission policies based on whether you choose to replicate KMS encrypted objects.

  • Select Copy KMS encrypted objects

    After selecting the AliyunOSSRole role, the system will automatically grant the following permission policies to the AliyunOSSRole role: AliyunOSSFullAccess (permissions to manage Object Storage Service OSS) and AliyunKMSCryptoUserAccess (permissions to manage Key Management Service KMS).

    Warning

    This role has all operation permissions for all Buckets and KMS under the current account. The permission scope is extensive, please use with caution.

  • Select Do not copy KMS encrypted objects

    After selecting the AliyunOSSRole role, the system will automatically grant AliyunOSSFullAccess (permissions to manage Object Storage Service OSS) to the AliyunOSSRole role.

    Warning

    This role has all operation permissions for all buckets under the current account. The permission scope is extensive, so please use it with caution.

Custom Role

When creating same-account same-region replication rules, you can use custom roles to complete replication tasks. You need to create a custom role through the RAM console and grant relevant permissions to the role.

  1. Create a normal service role.

    During role creation, select Cloud Service for the trusted entity type and select Object Storage Service for the trusted entity name. For detailed steps, see Create a RAM role for a trusted Alibaba Cloud service

  2. Grant permissions to the role.

    You can choose any of the following methods to grant permissions to the role.

    Grant system policies to the RAM role

    Warning

    You can choose to grant the AliyunOSSFullAccess system policy to the RAM role. AliyunOSSFullAccess grants all operation permissions on all buckets in the current account by default. Use this policy with caution.

    If you want to copy KMS-encrypted objects to the destination bucket, you also need to grant the AliyunKMSFullAccess system policy to the role.

    For specific steps, see Manage a RAM role's permissions

    Grant custom policies to the RAM role

    We recommend that you choose to grant the RAM role the minimum permissions required for copying between the source bucket (src-bucket) and destination bucket (dest-bucket).

    Note

    In actual use, please replace the source Bucket and target Bucket names accordingly.

    {
       "Version":"1",
       "Statement":[
          {
             "Effect":"Allow",
             "Action":[
                "oss:ReplicateList",
              	"oss:ReplicateGet"
             ],
             "Resource":[
              	"acs:oss:*:*:src-bucket",
                "acs:oss:*:*:src-bucket/*"
             ]
          },
          {
             "Effect":"Allow",
             "Action":[
              	"oss:ReplicateList",
                "oss:ReplicateGet",
                "oss:ReplicatePut",
                "oss:ReplicateDelete"
             ],
             "Resource":[
              	"acs:oss:*:*:dest-bucket",
    "acs:oss:*:*:dest-bucket/*"
             ]
          }
       ]
    }

    For detailed steps, see Manage a RAM role's permissions

    Note

    If you want to copy KMS-encrypted objects to the target Bucket, you also need to grant the AliyunKMSFullAccess system policy to the role.

Important

When copying data in the same region under the same account, OSS only verifies the RAM role permission policy for the copy operation and does not verify the Bucket Policy configured for the source or target Bucket.

Procedure

Using OSS console

  1. Log on to the OSS console.

  2. Click Buckets, then click src-bucket.

  3. In the left-side navigation pane, select Data Management > SRR

  4. On the SRR tab, click SRR

  5. In the SRR dialog box, configure the parameters as follows.

    Section

    Parameter

    Description

    Configure Destination Bucket

    Source bucket

    The region and name of the source bucket.

    Destination Bucket

    Select Select a bucket in the current account, and then select the destination bucket from the drop-down list. The destination bucket must be in the same region as the source bucket.

    Objects to Replicate

    Specify which objects to replicate.

    • Synchronize all files: Replicates all objects in the bucket to the destination bucket.

    • Replicate objects with a specific prefix: Replicates only objects with the specified prefixes. You can add up to 10 prefixes by default. To increase the limit to 100, contact technical support.

    Object tags

    Note

    To configure this parameter, the following conditions must be met:

    • You have configured object tags.

    • The Replicate delete markers and Replicate Delete Operations options are not selected.

    Select the Set Rules check box to replicate objects with specific tags to the destination bucket. You can add up to 10 tags (key-value pairs). After you add tags, select one of the following filter policies:

    • Match all tags: Replicates an object only if all of its tags are included in the filter rule.

    • Match any tag: Replicates an object if at least one of its tags is included in the filter rule.

    Note

    Currently, tag-based filtering is not available in the China (Shenzhen) Finance and China (Shanghai) Finance regions.

    Replicate KMS-Encrypted Source Objects

    Specifies whether to replicate objects encrypted with KMS to the destination bucket.

    • Replicate: Replicates an object if either the source object or the destination bucket is encrypted using server-side encryption with KMS-managed keys (SSE-KMS).

      Note

      You can call the HeadObject and GetBucketEncryption operations to query the encryption status of the source object and the destination bucket, respectively.

    • Do not replicate: Does not replicate objects encrypted with KMS to the destination bucket.

    CMK ID

    If you choose to replicate KMS-encrypted objects, you must specify the KMS key to encrypt the replicated objects in the destination bucket.

    Before you specify a KMS key, you must create one in the KMS console. The key must be in the same region as the destination bucket. For more information, see Create a key.

    RAM Role

    We recommend that you select New RAM Role. After you select this option from the drop-down list, follow the on-screen instructions to complete the role authorization.

    You can also select the AliyunOSSRole or a custom role. For more information about these three types of roles, see Role types.

    Configure Replication Policy

    Replicate Historical Data

    Specifies whether to replicate objects that existed in the source bucket before you enable the replication rule.

    • Replicate: Replicates existing objects to the destination bucket.

      Important

      When you replicate existing objects, objects from the source bucket may overwrite objects with the same name in the destination bucket. To prevent this type of data loss, we recommend that you enable versioning for both the source and destination buckets.

    • Do not replicate: Replicates only objects that are uploaded or updated after you enable the replication rule.

    Copy Delete Operation

    Specifies whether to replicate delete operations.

    Note

    After a data replication rule is created, OSS does not replicate changes to an object's storage class in the source bucket that result from lifecycle rules or CopyObject operations. The last access time (x-oss-last-access-time) attribute of objects in the source bucket is also not replicated.

    • No (for disaster recovery): Replicates object creation and update operations from the source bucket to the destination bucket.

      Important
      • This policy ensures that only new and updated objects are replicated. Delete operations in the source bucket do not affect the destination bucket. This way, you can prevent data loss in the destination bucket resulting from accidental manual deletions or automatic deletions by lifecycle rules in the source bucket.

      • If versioning is enabled on the source bucket, OSS creates a delete marker in the source bucket when an object is deleted without a version ID specified. OSS then replicates this delete marker to the destination bucket.

    • Yes (for data sharing): Replicates object creation, update, and deletion operations from the source bucket to the destination bucket.

      Important

      This policy ensures that OSS replicates object creations, updates, and deletions to the destination bucket. This ensures data consistency and is suitable for environments where multiple users or applications need to share and access the same dataset. However, with this policy, when you delete an object in the source bucket or a lifecycle rule deletes it, OSS also deletes the corresponding object in the destination bucket and it cannot be recovered.

    If an object is uploaded to the source Bucket by using multipart upload, the upload operation for each part is replicated to the destination Bucket. The final object that is generated after the CompleteMultipartUpload operation is called for all parts is also replicated to the destination Bucket.

    For more information about replication behavior when versioning is enabled, see Same-Region Replication with versioning.

  6. Click OK, and then in the dialog box that appears, click Enable.

    • After the same-region replication rule is created, you cannot edit or delete this rule.

    • The replication task starts 3 to 5 minutes after the same-region replication rule is configured. You can view the replication progress on the SRR tab of the source bucket.

    • Same-region replication between buckets uses asynchronous (near real-time) replication. The time required to replicate data to the destination bucket depends on the size of the data, typically ranging from a few minutes to several hours.

Using Alibaba Cloud SDK

Only Java, Python, and Go SDKs support same-account same-region replication.

Java

import com.aliyun.oss.ClientException;
import com.aliyun.oss.OSS;
import com.aliyun.oss.common.auth.*;
import com.aliyun.oss.OSSClientBuilder;
import com.aliyun.oss.OSSException;
import com.aliyun.oss.model.AddBucketReplicationRequest;
import com.aliyun.oss.ClientBuilderConfiguration;
import com.aliyun.oss.common.comm.SignVersion;

public class Demo {

    public static void main(String[] args) throws Exception {
        // Endpoint uses East China 1 (Hangzhou) as an example. Please fill in according to your actual situation for other Regions.
        String endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
        // Fill in the Region information corresponding to the Endpoint, for example cn-hangzhou.
        String region = "cn-hangzhou";
        // We strongly recommend that you do not save access credentials in your project code. Otherwise, access credentials may be leaked, which poses a threat to the security of all resources in your account. In this code example, access credentials are obtained from environment variables. Before running this code example, configure environment variables first.
        EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider();
        // Fill in the name of the source bucket.
        String bucketName = "src-bucket";
        // Specify the target bucket to which data is replicated. The target bucket and source bucket must belong to the same account.
        String targetBucketName = "dest-bucket";
        // Specify the region where the target bucket is located. The target bucket and source bucket must be in the same region.
        String targetBucketLocation = "oss-cn-hangzhou";

        // Create an OSSClient instance.
        // When the OSSClient instance is no longer used, call the shutdown method to release resources.
        ClientBuilderConfiguration clientBuilderConfiguration = new ClientBuilderConfiguration();
        // Explicitly declare the use of V4 signature algorithm
        clientBuilderConfiguration.setSignatureVersion(SignVersion.V4);
        OSS ossClient = OSSClientBuilder.create()
                .endpoint(endpoint)
                .credentialsProvider(credentialsProvider)
                .clientConfiguration(clientBuilderConfiguration)
                .region(region)
                .build();

        try {
            AddBucketReplicationRequest request = new AddBucketReplicationRequest(bucketName);

            request.setTargetBucketName(targetBucketName);
            request.setTargetBucketLocation(targetBucketLocation);
            // By default, historical data is replicated. This setting is set to false, indicating that replication of historical data is disabled.
            request.setEnableHistoricalObjectReplication(false);
            // Specify the name of the role that authorizes OSS to perform data replication. This role must be granted permissions to perform same-region replication on the source bucket and to receive replicated objects on the target bucket.
            request.setSyncRole("yourRole");
            // Specify whether OSS replicates objects created with SSE-KMS encryption.
            //request.setSseKmsEncryptedObjectsStatus("Enabled");
            // Specify the SSE-KMS key ID. If Status is specified as Enabled, this element must be specified.
            //request.setReplicaKmsKeyID("3542abdd-5821-4fb5-a425-90adca***");
            //List prefixes = new ArrayList();
            //prefixes.add("image/");
            //prefixes.add("video");
            //prefixes.add("a");
            //prefixes.add("A");
            // Specify the prefix of objects to be replicated. After a prefix is specified, only objects that match the prefix are replicated to the target bucket.
            //request.setObjectPrefixList(prefixes);
            //List actions = new ArrayList();
            //actions.add(AddBucketReplicationRequest.ReplicationAction.PUT);
            // Replicate the creation and update operations of objects in the source bucket to the target bucket.
            //request.setReplicationActionList(actions);
            ossClient.addBucketReplication(request);
        } catch (OSSException oe) {
            System.out.println("Caught an OSSException, which means your request made it to OSS, "
                    + "but was rejected with an error response for some reason.");
            System.out.println("Error Message:" + oe.getErrorMessage());
            System.out.println("Error Code:" + oe.getErrorCode());
            System.out.println("Request ID:" + oe.getRequestId());
            System.out.println("Host ID:" + oe.getHostId());
        } catch (ClientException ce) {
            System.out.println("Caught an ClientException, which means the client encountered "
                    + "a serious internal problem while trying to communicate with OSS, "
                    + "such as not being able to access the network.");
            System.out.println("Error Message:" + ce.getMessage());
        } finally {
            if (ossClient != null) {
                ossClient.shutdown();
            }
        }
    }
}        

Python

# -*- coding: utf-8 -*-
import oss2
from oss2.credentials import EnvironmentVariableCredentialsProvider
from oss2.models import ReplicationRule
# Obtain access credentials from environment variables. Before running this code example, make sure that the environment variables OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET have been set.
auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
# Fill in the endpoint that corresponds to the region where the source bucket is located. In this example, the source bucket is located in the China (Hangzhou) region. Set the endpoint to https://oss-cn-hangzhou.aliyuncs.com.
# Fill in the source Bucket name, for example src-bucket.
bucket = oss2.Bucket(auth, 'https://oss-cn-hangzhou.aliyuncs.com', 'src-bucket')
replica_config = ReplicationRule(
    # Specify the target Bucket to replicate data to. The target Bucket and source Bucket must belong to the same account.
    target_bucket_name='dest-bucket',
    # Specify the region where the target Bucket is located. The target Bucket and source Bucket must be in the same region.
    target_bucket_location='oss-cn-hangzhou',
    # Specify the role name authorized for OSS to perform data replication, and this role must have been granted permissions to perform same-region replication on the source Bucket and receive replicated objects on the target Bucket.
    sync_role_name='roleNameTest',
)

# Specify the prefix of Objects to be replicated. After specifying a Prefix, only Objects matching this Prefix will be replicated to the target Bucket.
# prefix_list = ['prefix1', 'prefix2']
# Set data replication rules.
# replica_config = ReplicationRule(
     # prefix_list=prefix_list,
     # Replicate the creation and update operations of Objects in the source Bucket to the target Bucket.
     # action_list=[ReplicationRule.PUT],
     # Specify the target Bucket to replicate data to.
     # target_bucket_name='dest-bucket',
     # Specify the region where the target Bucket is located.
     # target_bucket_location='yourTargetBucketLocation',
     # By default, replicate historical data. Setting this to False means disabling replication of historical data.
     # is_enable_historical_object_replication=False,    
     # Replicate objects created with SSE-KMS encryption.
     # sse_kms_encrypted_objects_status=ReplicationRule.ENABLED
     # Specify the SSE-KMS key ID. If you specify replication of objects created with SSE-KMS encryption, this element must be specified.
     # replica_kms_keyid='9468da86-3509-4f8d-a61e-6eab1eac****',
  #)

# Enable data replication.
bucket.put_bucket_replication(replica_config)

Go

package main

import (
	"encoding/xml"
	"fmt"
	"github.com/aliyun/aliyun-oss-go-sdk/oss"
	"os"
)

func HandleError(err error) {
	fmt.Println("Error:", err)
	os.Exit(-1)
}

// Enable data replication.
func main() {
	// Obtain access credentials from environment variables. Before running this code example, ensure that the environment variables OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET are set.
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Create an OSSClient instance.
	// Fill in yourEndpoint with the Endpoint corresponding to the Bucket. For example, for China (Hangzhou), fill in https://oss-cn-hangzhou.aliyuncs.com. For other regions, fill in according to the actual situation.
	client, err := oss.New("yourEndpoint", "", "", oss.SetCredentialsProvider(&provider))
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Specify the source Bucket name.
	srcbucketName := "yourSrcBucket"
	// Specify the target Bucket to replicate data to.
	destBucketName := "yourDestBucket"
	// Specify the prefixes prefix_1 and prefix_2 for Objects to be replicated. After specifying a Prefix, only Objects matching this Prefix will be replicated to the target Bucket.
	// If you need to replicate all Objects in the source Bucket to the target Bucket, there is no need to set a Prefix.
	prefix1 := "prefix_1"
	prefix2 := "prefix_2"
	// Specify the SSE-KMS key ID. This element must be specified if Status is set to Enabled.
	keyId := "c4d49f85-ee30-426b-a5ed-95e9****"
	// Specify whether OSS replicates objects encrypted with SSE-KMS.
	source := "Enabled"
	prefixSet := oss.ReplicationRulePrefix{Prefix: []*string{&prefix1, &prefix2}}
	reqReplication := oss.PutBucketReplication{
		Rule: []oss.ReplicationRule{
			{
				PrefixSet: &prefixSet,
				//Replicate the creation and update operations of objects in the source bucket to the target bucket.
				Action: "PUT",				
				Destination: &oss.ReplicationRuleDestination{
					Bucket: destBucketName,
					// Specify the region where the target bucket is located. The source bucket and target bucket must be in different regions.
					Location: "oss-cn-hangzhou",					
				},
				// Replicate historical data by default. Set to disabled to prohibit replication of historical data.
				HistoricalObjectReplication: "disabled",
				// Specify the role name authorized by OSS to perform data replication. This role must have been granted permissions to perform same-region replication on the source bucket and receive replicated objects on the target bucket.
				SyncRole:                "yourRole",
				EncryptionConfiguration: &keyId,
				SourceSelectionCriteria: &source,
			},
		},
	}

	xmlBody, err := xml.Marshal(reqReplication)
	if err != nil {
		HandleError(err)
	}
	err = client.PutBucketReplication(srcbucketName, string(xmlBody))

	if err != nil {
		HandleError(err)
	}

	fmt.Println("Put Bucket Replication Success!")
}

Use the ossutil command line tool

For more information about the specific steps to enable same-region replication using ossutil, see put-bucket-replication

Use REST API

If your program has high customization requirements, you can directly initiate REST API requests. Directly initiating REST API requests requires manually writing code to calculate signatures. For more information, see PutBucketReplication