Same-account same-region replication

更新时间:
复制 MD 格式

Same-account same-region replication automatically and asynchronously (in near real time) replicates operations such as the creation, update, and deletion of objects in a source bucket in a region of an account to a destination bucket in the same region of the same account. This topic describes how to configure same-account same-region replication.

Prerequisites

  • A bucket (Bucket A, the source bucket) is created in a region of an account, and the account UID, the name of Bucket A, and the region are recorded.

  • A bucket (Bucket B, the destination bucket) is created in the same account and region, and the name of Bucket B is recorded.

Role types

When you perform same-account same-region replication, you must specify a role that is used to perform replication operations between the source bucket and the destination bucket. You can select any of the following roles to complete the same-account same-region replication task.

Important

You can create the role by using a RAM user. The RAM user must have the following permissions: ram:CreateRole, ram:GetRole, ram:ListPoliciesForRole, and ram:AttachPolicyToRole. Because granting a RAM user role-related permissions such as ram:CreateRole and ram:GetRole poses relatively high risks, we recommend that you create a RAM role and complete role authorization by using the Alibaba Cloud account associated with the RAM user. After the authorization is complete, the RAM user can directly reuse the RAM role created by the Alibaba Cloud account.

(Recommended) Create a role

When you create a same-account same-region replication rule, you can select to create a role to complete the replication task. After you select to create a role, the system automatically creates a role in the oss-replication-{uuid} format and grants different permission policies based on whether you select to replicate KMS-encrypted objects.

  • Replicate KMS-encrypted objects

    After you create the role, you must complete role authorization based on the on-screen instructions. After the authorization is complete, the role has the precise permission policy for synchronizing data from the source bucket to the destination bucket, and the AliyunKMSCryptoUserAccess policy (permissions to manage Key Management Service (KMS)).

  • Do not replicate KMS-encrypted objects

    After you create the role, you must complete role authorization based on the on-screen instructions. After the authorization is complete, the role has the precise permission policy for synchronizing data from the source bucket to the destination bucket.

AliyunOSSRole

When you create a same-account same-region replication rule, you can select the AliyunOSSRole role to complete the replication task. After you select this role, the system grants different permission policies based on whether you select to replicate KMS-encrypted objects.

  • Replicate KMS-encrypted objects

    After you select the AliyunOSSRole role, the system automatically grants the following permission policies to the AliyunOSSRole role: AliyunOSSFullAccess (permissions to manage OSS) and AliyunKMSCryptoUserAccess (permissions to manage KMS).

    Warning

    This role has permissions to perform all operations on all buckets and KMS under the current account. The permission scope is large. Exercise caution.

  • Do not replicate KMS-encrypted objects

    After you select the AliyunOSSRole role, the system automatically grants AliyunOSSFullAccess (permissions to manage OSS) to the AliyunOSSRole role.

    Warning

    This role has permissions to perform all operations on all buckets under the current account. The permission scope is large. Exercise caution.

Custom role

When you create a same-account same-region replication rule, you can use a custom role to complete the replication task. You must create a custom role in the RAM console and grant the required permissions to the role.

  1. Create a normal service role.

    When you create the role, set the trusted entity type to Alibaba Cloud Service and the trusted service to Object Storage Service. For the specific steps, see Create a normal service role.

  2. Grant permissions to the role.

    You can grant permissions to the role in any of the following ways.

    Grant a system policy to the RAM role

    Warning

    You can grant the system policy AliyunOSSFullAccess to the RAM role. By default, AliyunOSSFullAccess has permissions to perform all operations on all buckets under the current account. Exercise caution.

    If you want to replicate KMS-encrypted objects to the destination bucket, you must also grant the AliyunKMSFullAccess system policy to the role.

    For the specific steps, see Manage the permissions of a RAM role.

    Grant a custom policy to the RAM role

    We recommend that you grant the RAM role the minimum permissions required for replication between the source bucket (src-bucket) and the destination bucket (dest-bucket).

    Note

    In actual use, replace the source bucket and destination bucket names accordingly.

    {
       "Version":"1",
       "Statement":[
          {
             "Effect":"Allow",
             "Action":[
                "oss:ReplicateList",
              	"oss:ReplicateGet"
             ],
             "Resource":[
              	"acs:oss:*:*:src-bucket",
                "acs:oss:*:*:src-bucket/*"
             ]
          },
          {
             "Effect":"Allow",
             "Action":[
              	"oss:ReplicateList",
                "oss:ReplicateGet",
                "oss:ReplicatePut",
                "oss:ReplicateDelete"
             ],
             "Resource":[
              	"acs:oss:*:*:dest-bucket",
                "acs:oss:*:*:dest-bucket/*"
             ]
          }
       ]
    }

    For the specific steps, see Manage the permissions of a RAM role.

    Note

    If you want to replicate KMS-encrypted objects to the destination bucket, you must also grant the AliyunKMSFullAccess system policy to the role.

Important

When you replicate data in the same region under the same account, OSS verifies only the permission policy of the RAM role used for replication, and does not verify the bucket policy configured for the source or destination bucket.

Procedure

Use the OSS console

  1. Log on to the OSS console.

  2. Click Buckets, and then click src-bucket.

  3. In the left-side navigation pane, choose Data Management > SRR.

  4. On the SRR tab, click SRR.

  5. In the SRR dialog box, configure the parameters as described below.

    Section

    Parameter

    Description

    Configure Destination Bucket

    Source Bucket

    Displays the region and name of the source bucket.

    Destination Bucket

    Select Select a bucket that belongs to the Alibaba Cloud account, and then select the name of the destination bucket in the same region from the drop-down list.

    Objects to Replicate

    Select the source data to replicate.

    • Synchronize all files: Replicates all objects in the bucket to the destination bucket.

    • Replicate objects with a specific prefix: Replicates objects with the specified prefix in the bucket to the destination bucket. By default, up to 10 prefixes can be added. To increase the number of prefixes, contact 技术支持 to request an adjustment, up to a maximum of 100.

    Object Tags

    Note

    To set this parameter, the following conditions must be met:

    • Object tags are configured.

    • Copy delete marker and Copy delete operations for specified versions are not selected.

    After you select the Set rule check box, you can replicate objects with specified tags to the destination bucket. Up to 10 tags (key-value pairs) can be added. After you add tags, you can select one of the following tag filter policies:

    • Contain all tags: An object is replicated only if all of its tags are in the tag set specified in the filter rule.

    • Contain any tag: An object is replicated if any one of its tags is in the tag set specified in the filter rule.

    Replicate KMS-Encrypted Source Objects

    Select whether to replicate KMS-encrypted objects to the destination bucket.

    • Yes: Replicates objects to the destination bucket when the source object or the destination bucket uses a KMS-managed key for encryption (SSE-KMS with a specified CMK ID).

      Note

      You can use HeadObject and GetBucketEncryption to query the encryption status of the source object and the destination bucket, respectively.

    • No: Does not replicate KMS-encrypted objects to the destination bucket.

    CMK ID

    If you select to replicate KMS-encrypted objects to the destination bucket, you must specify the KMS key used to encrypt the destination objects.

    Before you specify a KMS key, you must create a KMS key in the same region as the destination bucket on the KMS platform. For the specific steps, see Create a key.

    RAM Role

    We recommend that you select New RAM Role. After you select to create a role from the drop-down list, you must complete role authorization based on the on-screen instructions.

    You can also select AliyunOSSRole or a custom role. For more information about these three types of roles, see Role types.

    Configure Replication Policy

    Replicate Historical Data

    Select whether to replicate the historical data that exists in the source bucket before the same-region replication rule takes effect.

    • Replicate: Replicates historical data to the destination bucket.

      Important

      When historical data is replicated, objects replicated from the source bucket may overwrite objects with the same name in the destination bucket. To prevent these files from being lost, we recommend that you enable versioning for both the source bucket and the destination bucket.

    • Do not replicate: Replicates only the objects that are uploaded or updated after the same-region replication rule takes effect.

    Copy Delete Operation

    Select whether to replicate the delete operations of the source bucket to the destination bucket.

    Note

    This option is displayed in the console when versioning is not enabled for the source bucket. If versioning is enabled for the source bucket, this option is replaced by two independent settings: Copy Delete Marker and Copy Delete Operation of Specified Version.

    • Yes (applicable to scenarios where the same dataset is shared and accessed): Replicates the creation, update, and deletion operations of objects in the source bucket to the destination bucket.

      Important

      With this policy, the creation, update, and deletion of objects are all replicated to the destination bucket. When you manually delete an object in the source bucket or an object is automatically deleted by a lifecycle rule, the corresponding object in the destination bucket is also deleted and cannot be recovered.

    • No (applicable to disaster recovery scenarios): Replicates only the creation and update operations of objects in the source bucket to the destination bucket. Deletion operations do not affect the destination bucket.

      Note

      This method effectively prevents data loss in the destination bucket caused by manual deletion or automatic deletion by lifecycle rules in the source bucket.

    Copy Delete Marker

    Select whether to replicate the delete markers of the source bucket to the destination bucket.

    • Replicate: When an object is deleted from the source bucket without a specified version ID, the delete marker created by OSS in the source bucket is synchronized to the destination bucket. This is applicable to scenarios where the same dataset is shared and accessed, ensuring that the data states of the source and destination buckets are consistent.

      Important

      After you configure this policy, when you manually delete an object in the source bucket or an object is automatically deleted by a lifecycle rule, the delete marker is also synchronized to the destination bucket, making the data inaccessible on the destination side as well.

    • Do not replicate (applicable to disaster recovery scenarios): Delete markers created in the source bucket are not replicated to the destination bucket. This effectively prevents data loss in the destination bucket caused by accidental deletion or automatic deletion by lifecycle rules in the source bucket.

    Copy Delete Operation of Specified Version

    Select whether to replicate the permanent deletion operations for specified versions in the source bucket to the destination bucket.

    Note

    After you create a data replication rule, storage class changes in the source bucket caused by lifecycle rules or by modifying the storage class through CopyObject, as well as the last access time (x-oss-last-access-time) attribute of objects in the bucket, are not synchronized to the destination bucket.

    • Replicate: When a specified version of a source file (including the current version and historical versions) is permanently deleted, the corresponding version in the destination bucket is also permanently deleted. This is applicable to scenarios where you need to ensure that the source and destination data are completely consistent.

      Important

      After you configure this policy, object versions permanently deleted from the source bucket cannot be recovered in the destination bucket. Exercise caution.

    • Do not replicate (applicable to disaster recovery scenarios): When a specified version of a source object is permanently deleted, the corresponding version in the destination bucket is not deleted. This prevents permanent deletion operations in the source bucket from affecting the data security of the destination bucket.

    If an object is uploaded to the source bucket by using multipart upload, the upload operation of each part is replicated to the destination bucket. Finally, the object generated after CompleteMultipartUpload is performed on all parts is also replicated to the destination bucket.

    For information about the replication behavior of same-region replication combined with versioning, see Same-region replication combined with versioning.

  6. Click OK, and then in the dialog box that appears, click Enable.

    • After the same-region replication rule is created, the rule cannot be edited or deleted.

    • The replication task starts 3 to 5 minutes after the same-region replication rule is configured. You can view the replication progress on the SRR tab of the source bucket.

    • Because same-region replication between buckets uses asynchronous (near real-time) replication, the time required to replicate data to the destination bucket depends on the size of the data, and usually ranges from a few minutes to a few hours.

Use Alibaba Cloud SDKs

Only the Java, Python, and Go SDKs support same-account same-region replication.

Java

import com.aliyun.oss.ClientException;
import com.aliyun.oss.OSS;
import com.aliyun.oss.common.auth.*;
import com.aliyun.oss.OSSClientBuilder;
import com.aliyun.oss.OSSException;
import com.aliyun.oss.model.AddBucketReplicationRequest;
import com.aliyun.oss.ClientBuilderConfiguration;
import com.aliyun.oss.common.comm.SignVersion;

public class Demo {

    public static void main(String[] args) throws Exception {
        // In this example, the endpoint of the China (Hangzhou) region is used. Specify the endpoint based on your actual region.
        String endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
        // Specify the region that corresponds to the endpoint, for example, cn-hangzhou.
        String region = "cn-hangzhou";
        // We strongly recommend that you do not save access credentials in project code. Otherwise, the access credentials may be leaked and the security of all resources under your account may be compromised. In this example, access credentials are obtained from environment variables. Before you run this sample code, configure the environment variables.
        EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider();
        // Specify the name of the source bucket.
        String bucketName = "src-bucket";
        // Specify the destination bucket to which data is replicated. The destination bucket and the source bucket must belong to the same account.
        String targetBucketName = "dest-bucket";
        // Specify the region of the destination bucket. The destination bucket and the source bucket must be in the same region.
        String targetBucketLocation = "oss-cn-hangzhou";

        // Create an OSSClient instance.
        // When the OSSClient instance is no longer used, call the shutdown method to release resources.
        ClientBuilderConfiguration clientBuilderConfiguration = new ClientBuilderConfiguration();
        // Explicitly declare the use of the V4 signature algorithm.
        clientBuilderConfiguration.setSignatureVersion(SignVersion.V4);
        OSS ossClient = OSSClientBuilder.create()
                .endpoint(endpoint)
                .credentialsProvider(credentialsProvider)
                .clientConfiguration(clientBuilderConfiguration)
                .region(region)
                .build();

        try {
            AddBucketReplicationRequest request = new AddBucketReplicationRequest(bucketName);

            request.setTargetBucketName(targetBucketName);
            request.setTargetBucketLocation(targetBucketLocation);
            // Historical data is replicated by default. Set this to false to disable the replication of historical data.
            request.setEnableHistoricalObjectReplication(false);
            // Specify the name of the role that authorizes OSS to replicate data. The role must be granted the permissions to perform same-region replication on the source bucket and to receive replicated objects in the destination bucket.
            request.setSyncRole("yourRole");
            // Specify whether OSS replicates objects created by using SSE-KMS encryption.
            //request.setSseKmsEncryptedObjectsStatus("Enabled");
            // Specify the SSE-KMS key ID. This element is required if Status is set to Enabled.
            //request.setReplicaKmsKeyID("3542abdd-5821-4fb5-a425-90adca***");
            //List prefixes = new ArrayList();
            //prefixes.add("image/");
            //prefixes.add("video");
            //prefixes.add("a");
            //prefixes.add("A");
            // Specify the prefixes of the objects to replicate. After you specify prefixes, only objects that match the prefixes are replicated to the destination bucket.
            //request.setObjectPrefixList(prefixes);
            //List actions = new ArrayList();
            //actions.add(AddBucketReplicationRequest.ReplicationAction.PUT);
            // Replicate the creation and update operations of objects in the source bucket to the destination bucket.
            //request.setReplicationActionList(actions);
            ossClient.addBucketReplication(request);
        } catch (OSSException oe) {
            System.out.println("Caught an OSSException, which means your request made it to OSS, "
                    + "but was rejected with an error response for some reason.");
            System.out.println("Error Message:" + oe.getErrorMessage());
            System.out.println("Error Code:" + oe.getErrorCode());
            System.out.println("Request ID:" + oe.getRequestId());
            System.out.println("Host ID:" + oe.getHostId());
        } catch (ClientException ce) {
            System.out.println("Caught an ClientException, which means the client encountered "
                    + "a serious internal problem while trying to communicate with OSS, "
                    + "such as not being able to access the network.");
            System.out.println("Error Message:" + ce.getMessage());
        } finally {
            if (ossClient != null) {
                ossClient.shutdown();
            }
        }
    }
}        

Python

# -*- coding: utf-8 -*-
import oss2
from oss2.credentials import EnvironmentVariableCredentialsProvider
from oss2.models import ReplicationRule
# Obtain access credentials from environment variables. Before you run this sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are set.
auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
# Specify the endpoint that corresponds to the region of the source bucket. For example, if the source bucket is in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com.
# Specify the name of the source bucket, for example, src-bucket.
bucket = oss2.Bucket(auth, 'https://oss-cn-hangzhou.aliyuncs.com', 'src-bucket')
replica_config = ReplicationRule(
    # Specify the destination bucket to which data is replicated. The destination bucket and the source bucket must belong to the same account.
    target_bucket_name='dest-bucket',
    # Specify the region of the destination bucket. The destination bucket and the source bucket must be in the same region.
    target_bucket_location='oss-cn-hangzhou',
    # Specify the name of the role that authorizes OSS to replicate data. The role must be granted the permissions to perform same-region replication on the source bucket and to receive replicated objects in the destination bucket.
    sync_role_name='roleNameTest',
)

# Specify the prefixes of the objects to replicate. After you specify prefixes, only objects that match the prefixes are replicated to the destination bucket.
# prefix_list = ['prefix1', 'prefix2']
# Configure the data replication rule.
# replica_config = ReplicationRule(
     # prefix_list=prefix_list,
     # Replicate the creation and update operations of objects in the source bucket to the destination bucket.
     # action_list=[ReplicationRule.PUT],
     # Specify the destination bucket to which data is replicated.
     # target_bucket_name='dest-bucket',
     # Specify the region of the destination bucket.
     # target_bucket_location='yourTargetBucketLocation',
     # Historical data is replicated by default. Set this to False to disable the replication of historical data.
     # is_enable_historical_object_replication=False,
     # Replicate objects created by using SSE-KMS encryption.
     # sse_kms_encrypted_objects_status=ReplicationRule.ENABLED
     # Specify the SSE-KMS key ID. This element is required if you specify to replicate objects created by using SSE-KMS encryption.
     # replica_kms_keyid='9468da86-3509-4f8d-a61e-6eab1eac****',
  #)

# Enable data replication.
bucket.put_bucket_replication(replica_config)

Go

package main

import (
	"encoding/xml"
	"fmt"
	"github.com/aliyun/aliyun-oss-go-sdk/oss"
	"os"
)

func HandleError(err error) {
	fmt.Println("Error:", err)
	os.Exit(-1)
}

// Enable data replication.
func main() {
	// Obtain access credentials from environment variables. Before you run this sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are set.
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Create an OSSClient instance.
	// Set yourEndpoint to the endpoint that corresponds to the bucket. For example, if the bucket is in the China (Hangzhou) region, set it to https://oss-cn-hangzhou.aliyuncs.com. Specify other regions based on your actual situation.
	client, err := oss.New("yourEndpoint", "", "", oss.SetCredentialsProvider(&provider))
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Specify the name of the source bucket.
	srcbucketName := "yourSrcBucket"
	// Specify the destination bucket to which data is replicated.
	destBucketName := "yourDestBucket"
	// Specify the prefixes prefix_1 and prefix_2 of the objects to replicate. After you specify prefixes, only objects that match the prefixes are replicated to the destination bucket.
	// If you want to replicate all objects in the source bucket to the destination bucket, you do not need to specify prefixes.
	prefix1 := "prefix_1"
	prefix2 := "prefix_2"
	// Specify the SSE-KMS key ID. This element is required if Status is set to Enabled.
	keyId := "c4d49f85-ee30-426b-a5ed-95e9****"
	// Specify whether OSS replicates objects created by using SSE-KMS encryption.
	source := "Enabled"
	prefixSet := oss.ReplicationRulePrefix{Prefix: []*string{&prefix1, &prefix2}}
	reqReplication := oss.PutBucketReplication{
		Rule: []oss.ReplicationRule{
			{
				PrefixSet: &prefixSet,
				//Replicate the creation and update operations of objects in the source bucket to the destination bucket.
				Action: "PUT",
				Destination: &oss.ReplicationRuleDestination{
					Bucket: destBucketName,
					// Specify the region of the destination bucket. The source bucket and the destination bucket must be in different regions.
					Location: "oss-cn-hangzhou",
				},
				// Historical data is replicated by default. Set this to disabled to disable the replication of historical data.
				HistoricalObjectReplication: "disabled",
				// Specify the name of the role that authorizes OSS to replicate data. The role must be granted the permissions to perform same-region replication on the source bucket and to receive replicated objects in the destination bucket.
				SyncRole:                "yourRole",
				EncryptionConfiguration: &keyId,
				SourceSelectionCriteria: &source,
			},
		},
	}

	xmlBody, err := xml.Marshal(reqReplication)
	if err != nil {
		HandleError(err)
	}
	err = client.PutBucketReplication(srcbucketName, string(xmlBody))

	if err != nil {
		HandleError(err)
	}

	fmt.Println("Put Bucket Replication Success!")
}

Use the ossutil command-line tool

For the specific steps to enable same-region replication by using ossutil, see put-bucket-replication.

Use the REST API

If your program has high customization requirements, you can directly initiate REST API requests. To directly initiate REST API requests, you must manually write code to calculate the signature. For more information, see PutBucketReplication.