Same-account same-region replication

更新时间:
复制 MD 格式

同账号同区域复制是指将某个账号某个地域下源存储空间(Bucket)的文件(Object)的创建、更新和删除等操作自动、异步(近实时)地复制到当前账号相同地域下的目标Bucket中。本文介绍如何进行同账号同区域复制。

前提条件

  • 已在某账号的某地域创建了Bucket A(源Bucket),并记录账号UID、Bucket A名称及地域。

  • 已在相同账号和地域创建了Bucket B(目标Bucket),并记录Bucket B名称。

角色类型

执行同账号同区域复制时,需要指定用于源Bucket与目标Bucket之间执行复制操作的角色。您可以选择以下任意角色完成同账号同区域复制任务。

Important

您可以选择通过RAM用户创建角色,RAM用户必须拥有以下权限:ram:CreateRoleram:GetRoleram:ListPoliciesForRoleram:AttachPolicyToRole。考虑到授予RAM用户ram:CreateRoleram:GetRole等角色相关的权限风险较大,建议通过RAM用户关联的阿里云账号创建RAM角色并完成角色授权。授权完成后,RAM用户可以直接复用阿里云账号创建的RAM角色。

(推荐)新建角色

创建同账号同区域复制规则时,支持选择新建角色来完成复制任务。选择新建角色后,后台将自动创建格式为oss-replication-{uuid} 的角色,并根据是否选择复制KMS加密对象授予不同权限策略。

  • 选择复制KMS加密对象

    新建角色后,您需要按照页面指引完成角色授权操作。授权完成后,该角色拥有源Bucket同步到目标Bucket精准权限策略以及AliyunKMSCryptoUserAccess(管理密钥管理服务KMS的权限)。

  • 选择不复制KMS加密对象

    新建角色后,您需要按照页面指引完成角色授权操作。授权完成后,该角色拥有源Bucket同步到目标Bucket精准权限策略。

AliyunOSSRole

创建同账号同区域复制规则时,支持选择AliyunOSSRole角色来完成复制任务。选择该角色后,后台会根据是否选择复制KMS加密对象授予不同权限策略。

  • 选择复制KMS加密对象

    选择AliyunOSSRole角色后,后台将自动为AliyunOSSRole角色授予以下权限策略:AliyunOSSFullAccess(管理对象存储OSS的权限)以及AliyunKMSCryptoUserAccess(管理密钥管理服务KMS的权限)。

    Warning

    该角色拥有当前账号下所有Bucket以及KMS的所有操作权限,权限范围较大,请谨慎使用。

  • 选择不复制KMS加密对象

    选择AliyunOSSRole角色后,后台将自动为AliyunOSSRole角色授予AliyunOSSFullAccess(管理对象存储OSS的权限)。

    Warning

    该角色拥有当前账号下所有Bucket所有操作权限,权限范围较大,请谨慎使用。

自定义角色

创建同账号同区域复制规则时,支持使用自定义角色来完成复制任务。您需要通过RAM控制台创建自定义角色,并为角色赋予相关权限。

  1. 创建普通服务角色。

    创建角色过程中,信任主体类型选择云服务,信任主体名称选择对象存储。具体步骤,请参见Create a standard service role

  2. 为角色授权。

    您可以选择以下任意方式为角色授权。

    RAM角色授予系统策略

    Warning

    您可以选择为RAM角色授予系统策略AliyunOSSFullAccessAliyunOSSFullAccess默认拥有当前账号下所有Bucket的所有操作权限,请谨慎使用。

    如果您希望将KMS加密的Object复制到目标Bucket,您还需要为角色授予AliyunKMSFullAccess系统策略。

    具体步骤,请参见Manage permissions for a RAM role

    RAM角色授予自定义策略

    建议您选择为RAM角色授予源Bucket(src-bucket)和目标Bucket(dest-bucket)复制所需的最小权限。

    Note

    实际使用时,请相应替换源Bucket和目标Bucket名称。

    {
       "Version":"1",
       "Statement":[
          {
             "Effect":"Allow",
             "Action":[
                "oss:ReplicateList",
              	"oss:ReplicateGet"
             ],
             "Resource":[
              	"acs:oss:*:*:src-bucket",
                "acs:oss:*:*:src-bucket/*"
             ]
          },
          {
             "Effect":"Allow",
             "Action":[
              	"oss:ReplicateList",
                "oss:ReplicateGet",
                "oss:ReplicatePut",
                "oss:ReplicateDelete"
             ],
             "Resource":[
              	"acs:oss:*:*:dest-bucket",
                "acs:oss:*:*:dest-bucket/*"
             ]
          }
       ]
    }

    具体步骤,请参见Manage permissions for a RAM role

    Note

    如果您希望将KMS加密的Object复制到目标Bucket,您还需要为角色授予AliyunKMSFullAccess系统策略。

Important

在同一账号下相同地域复制数据时,OSS仅校验复制操作的RAM角色权限策略,不校验源或者目标Bucket配置的Bucket Policy。

操作步骤

使用OSS控制台

  1. Log on to the OSS console.

  2. 单击Buckets,然后单击src-bucket。

  3. 在左侧导航栏,选择Data Management > SRR

  4. SRR页签,单击SRR

  5. SRR对话框,按以下说明配置各项参数。

    Section

    Parameter

    Description

    Configure Destination Bucket

    Source Bucket

    The region and name of the source bucket.

    Destination Bucket

    Select Select a bucket in this account, and then select the destination bucket in the same region from the drop-down list.

    Objects to Replicate

    Select the source data that you want to replicate.

    • Synchronize all files: Replicates all objects in the bucket to the destination bucket.

    • Replicate objects with specified prefixes: Replicates only the objects with a specified prefix to the destination bucket. By default, you can add up to 10 prefixes. To add more prefixes, contact technical support. The prefix limit can be increased to a maximum of 100.

    Object Tag

    Note

    You can configure this parameter only if the following conditions are met:

    Select the Set Rule checkbox to replicate objects with specific tags to the destination bucket. You can add up to 10 tags (key-value pairs). After adding tags, you can select one of the following filtering policies:

    • Match All Tags: An object is replicated only if all of its tags are included in the set of tags specified in the filter rule.

    • Match Any Tag: An object is replicated if at least one of its tags is included in the set of tags specified in the filter rule.

    Note

    Currently, the tag-based filtering feature is not available in the finance cloud regions in China (Shenzhen) and China (Shanghai).

    Replicate KMS-Encrypted Source Objects

    Specifies whether to replicate KMS-encrypted objects to the destination bucket.

    • Replicate: Replicates objects to the destination bucket when either the source objects or the destination bucket is encrypted by using SSE-KMS with a specified CMK ID.

      Note

      You can call the HeadObject and GetBucketEncryption operations to query the encryption status of the source object and the destination bucket, respectively.

    • Do not replicate: Does not replicate KMS-encrypted objects to the destination bucket.

    CMK ID

    If you choose to replicate KMS-encrypted objects, you must specify the KMS key to encrypt the destination objects.

    Before you specify a KMS key, you must create a KMS key in the same region as the destination bucket. For more information, see Create a CMK.

    RAM Role

    We recommend that you select New RAM Role. After you select this option from the drop-down list, you must follow the on-screen instructions to grant permissions to the role.

    You can also select AliyunOSSRole or a custom role. For more information about these three role types, see Role types.

    Configure Replication Policy

    Replicate Historical Data

    Specifies whether to replicate objects that existed in the source bucket before the replication rule was enabled.

    • Replicate: Replicates existing data to the destination bucket.

      Important

      When you replicate existing data, objects from the source bucket may overwrite objects with the same name in the destination bucket. To prevent this type of data loss, we recommend that you enable versioning for both the source and destination buckets.

    • Do not replicate: Replicates only the objects that are uploaded or updated after the same-region replication rule takes effect.

    Copy Delete Operation

    Specifies whether to replicate delete operations from the source bucket to the destination bucket.

    Note

    This option appears only if versioning is disabled for the source bucket. If versioning is enabled, it is replaced by Copy Delete Marker and Copy Delete Operation of Specified Version.

    • Yes (for scenarios where you need to share and access the same dataset): Replicates creations, updates, and deletions of objects in the source bucket to the destination bucket.

      Important

      With this policy, the creation, update, and deletion of objects are all replicated to the destination bucket. When an object is deleted from the source bucket, either manually or by a lifecycle rule, the corresponding object in the destination bucket is also deleted and cannot be recovered.

    • No (for disaster recovery scenarios): Replicates only creations and updates of objects in the source bucket to the destination bucket. Delete operations do not affect the destination bucket.

      Note

      This method prevents data loss in the destination bucket that could result from accidental manual deletions or automatic deletions by lifecycle rules in the source bucket.

    Copy Delete Marker

    Specifies whether to replicate delete markers from the source bucket to the destination bucket.

    • Replicate: When an object is deleted from the source bucket without a version ID specified, the delete marker created by OSS in the source bucket is replicated to the destination bucket. This is suitable for scenarios where you need to share and access the same dataset and ensure data consistency between the source and destination buckets.

      Important

      If you configure this policy, when an object is deleted from the source bucket, either manually or by a lifecycle rule, the delete marker is also replicated to the destination bucket, making the data inaccessible in the destination bucket.

    • Do not replicate (for disaster recovery scenarios): Delete markers created in the source bucket are not replicated to the destination bucket. This effectively prevents data loss in the destination bucket caused by accidental deletion or automatic deletion by lifecycle rules in the source bucket.

    Copy Delete Operation of Specified Version

    Specifies whether to replicate the permanent deletion of a specific object version from the source bucket to the destination bucket.

    Note

    After a replication rule is enabled, OSS does not replicate the following to the destination bucket: changes to object storage class resulting from lifecycle rules or CopyObject operations, and updates to the last access time (x-oss-last-access-time) attribute.

    • Replicate: When a specific version of a source object (including the current and previous versions) is permanently deleted, the corresponding version in the destination bucket is also permanently deleted. This is suitable for scenarios that require perfect data consistency between the source and destination buckets.

      Important

      If you configure this policy, object versions that are permanently deleted from the source bucket cannot be recovered in the destination bucket. Use this option with caution.

    • Do not replicate (for disaster recovery scenarios): When a specific version of a source object is permanently deleted, the corresponding version in the destination bucket is not deleted. This prevents permanent delete operations in the source bucket from affecting data security in the destination bucket.

    If an object is uploaded to the source bucket by using multipart upload, the upload operation for each part is replicated to the destination bucket. The final object generated after the CompleteMultipartUpload operation is called is also replicated to the destination bucket.

    For more information about replication behavior when versioning is enabled, see Same-region replication with versioning.

  6. Click OK, and in the dialog box that appears, click Enable.

    • 当同区域复制规则创建完成后,不允许对此规则进行编辑或删除。

    • 复制任务会在同区域复制规则配置完成的3~5分钟后启动。您可以在源BucketSRR页签,查看复制进度。

    • 由于Bucket间的同区域复制采用异步(近实时)复制,数据复制到目标Bucket需要的时间取决于数据的大小,通常几分钟到几小时不等。

使用阿里云SDK

Java、Python以及Go SDK支持同账号同区域复制。

Java

import com.aliyun.oss.ClientException;
import com.aliyun.oss.OSS;
import com.aliyun.oss.common.auth.*;
import com.aliyun.oss.OSSClientBuilder;
import com.aliyun.oss.OSSException;
import com.aliyun.oss.model.AddBucketReplicationRequest;
import com.aliyun.oss.ClientBuilderConfiguration;
import com.aliyun.oss.common.comm.SignVersion;

public class Demo {

    public static void main(String[] args) throws Exception {
        // Endpoint以华东1(杭州)为例,其它Region请按实际情况填写。
        String endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
        // 填写Endpoint对应的Region信息,例如cn-hangzhou。
        String region = "cn-hangzhou";
        // 强烈建议不要把访问凭证保存到工程代码里,否则可能导致访问凭证泄露,威胁您账号下所有资源的安全。本代码示例以从环境变量中获取访问凭证为例。运行本代码示例之前,请先配置环境变量。
        EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider();
        // 填写源Bucket名称。
        String bucketName = "src-bucket";
        // 指定数据要复制到的目标Bucket。目标Bucket与源Bucket必须属于相同账号。
        String targetBucketName = "dest-bucket";
        // 指定目标Bucket所在地域。目标Bucket与源Bucket必须处于相同地域。
        String targetBucketLocation = "oss-cn-hangzhou";

        // 创建OSSClient实例。
        // 当OSSClient实例不再使用时,调用shutdown方法以释放资源。
        ClientBuilderConfiguration clientBuilderConfiguration = new ClientBuilderConfiguration();
        // 显式声明使用 V4 签名算法
        clientBuilderConfiguration.setSignatureVersion(SignVersion.V4);
        OSS ossClient = OSSClientBuilder.create()
                .endpoint(endpoint)
                .credentialsProvider(credentialsProvider)
                .clientConfiguration(clientBuilderConfiguration)
                .region(region)
                .build();

        try {
            AddBucketReplicationRequest request = new AddBucketReplicationRequest(bucketName);

            request.setTargetBucketName(targetBucketName);
            request.setTargetBucketLocation(targetBucketLocation);
            // 默认复制历史数据。此处设置为false,表示禁止复制历史数据。
            request.setEnableHistoricalObjectReplication(false);
            // 指定授权OSS进行数据复制的角色名称,且该角色必须已被授予源Bucket执行同区域复制以及目标Bucket接收复制对象的权限。
            request.setSyncRole("yourRole");
            // 指定OSS是否复制通过SSE-KMS加密创建的对象。
            //request.setSseKmsEncryptedObjectsStatus("Enabled");
            // 指定SSE-KMS密钥ID。如果指定Status为Enabled,则必须指定该元素。
            //request.setReplicaKmsKeyID("3542abdd-5821-4fb5-a425-90adca***");
            //List prefixes = new ArrayList();
            //prefixes.add("image/");
            //prefixes.add("video");
            //prefixes.add("a");
            //prefixes.add("A");
            // 指定待复制Object的前缀Prefix。指定Prefix后,只有匹配该Prefix的Object才会复制到目标Bucket。
            //request.setObjectPrefixList(prefixes);
            //List actions = new ArrayList();
            //actions.add(AddBucketReplicationRequest.ReplicationAction.PUT);
            // 将源Bucket内Object的新增、更新操作复制到目标Bucket。
            //request.setReplicationActionList(actions);
            ossClient.addBucketReplication(request);
        } catch (OSSException oe) {
            System.out.println("Caught an OSSException, which means your request made it to OSS, "
                    + "but was rejected with an error response for some reason.");
            System.out.println("Error Message:" + oe.getErrorMessage());
            System.out.println("Error Code:" + oe.getErrorCode());
            System.out.println("Request ID:" + oe.getRequestId());
            System.out.println("Host ID:" + oe.getHostId());
        } catch (ClientException ce) {
            System.out.println("Caught an ClientException, which means the client encountered "
                    + "a serious internal problem while trying to communicate with OSS, "
                    + "such as not being able to access the network.");
            System.out.println("Error Message:" + ce.getMessage());
        } finally {
            if (ossClient != null) {
                ossClient.shutdown();
            }
        }
    }
}        

Python

# -*- coding: utf-8 -*-
import oss2
from oss2.credentials import EnvironmentVariableCredentialsProvider
from oss2.models import ReplicationRule
# 从环境变量中获取访问凭证。运行本代码示例之前,请确保已设置环境变量OSS_ACCESS_KEY_ID和OSS_ACCESS_KEY_SECRET。
auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
# 填写源Bucket所在地域对应的Endpoint。以华东1(杭州)为例,Endpoint填写为https://oss-cn-hangzhou.aliyuncs.com。
# 填写源Bucket名称,例如src-bucket。
bucket = oss2.Bucket(auth, 'https://oss-cn-hangzhou.aliyuncs.com', 'src-bucket')
replica_config = ReplicationRule(
    # 指定数据要复制到的目标Bucket。目标Bucket与源Bucket必须属于相同账号。
    target_bucket_name='dest-bucket',
    # 指定目标Bucket所在地域。目标Bucket与源Bucket必须处于相同地域。
    target_bucket_location='oss-cn-hangzhou',
    # 指定授权OSS进行数据复制的角色名称,且该角色必须已被授予源Bucket执行同区域复制以及目标Bucket接收复制对象的权限。
    sync_role_name='roleNameTest',
)

# 指定待复制Object的前缀Prefix。指定Prefix后,只有匹配该Prefix的Object才会复制到目标Bucket。
# prefix_list = ['prefix1', 'prefix2']
# 设置数据复制规则。
# replica_config = ReplicationRule(
     # prefix_list=prefix_list,
     # 将源Bucket内Object的新增、更新操作复制到目标Bucket。
     # action_list=[ReplicationRule.PUT],
     # 指定数据要复制到的目标Bucket。
     # target_bucket_name='dest-bucket',
     # 指定目标Bucket所在地域。
     # target_bucket_location='yourTargetBucketLocation',
     # 默认复制历史数据。此处设置为False,表示禁止复制历史数据。
     # is_enable_historical_object_replication=False,    
     # 复制通过SSE-KMS加密创建的对象。
     # sse_kms_encrypted_objects_status=ReplicationRule.ENABLED
     # 指定SSE-KMS密钥ID。如果指定复制通过SSE-KMS加密创建的对象,则必须指定该元素。
     # replica_kms_keyid='9468da86-3509-4f8d-a61e-6eab1eac****',
  #)

# 开启数据复制。
bucket.put_bucket_replication(replica_config)

Go

package main

import (
	"encoding/xml"
	"fmt"
	"github.com/aliyun/aliyun-oss-go-sdk/oss"
	"os"
)

func HandleError(err error) {
	fmt.Println("Error:", err)
	os.Exit(-1)
}

// 开启数据复制。
func main() {
	// 从环境变量中获取访问凭证。运行本代码示例之前,请确保已设置环境变量OSS_ACCESS_KEY_ID和OSS_ACCESS_KEY_SECRET。
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// 创建OSSClient实例。
	// yourEndpoint填写Bucket对应的Endpoint,以华东1(杭州)为例,填写为https://oss-cn-hangzhou.aliyuncs.com。其它Region请按实际情况填写。
	client, err := oss.New("yourEndpoint", "", "", oss.SetCredentialsProvider(&provider))
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// 指定源Bucket名称。
	srcbucketName := "yourSrcBucket"
	// 指定数据要复制到的目标Bucket。
	destBucketName := "yourDestBucket"
	// 指定待复制Object的前缀prefix_1和prefix_2。指定Prefix后,只有匹配该Prefix的Object才会复制到目标Bucket。
	// 如果您需要将源Bucket中的所有Object复制到目标Bucket,则无需设置Prefix。
	prefix1 := "prefix_1"
	prefix2 := "prefix_2"
	// 指定SSE-KMS密钥ID。如果指定Status为Enabled,则必须指定该元素。
	keyId := "c4d49f85-ee30-426b-a5ed-95e9****"
	// 指定OSS是否复制通过SSE-KMS加密创建的对象。
	source := "Enabled"
	prefixSet := oss.ReplicationRulePrefix{Prefix: []*string{&prefix1, &prefix2}}	
	reqReplication := oss.PutBucketReplication{
		Rule: []oss.ReplicationRule{
			{
				PrefixSet: &prefixSet,
				//将源Bucket内Object的新增、更新操作复制到目标Bucket。
				Action: "PUT",				
				Destination: &oss.ReplicationRuleDestination{
					Bucket: destBucketName,
					// 指定目标Bucket所在地域。源Bucket与目标Bucket必须处于不同的地域。
					Location: "oss-cn-hangzhou",					
				},
				// 默认复制历史数据。此处设置为disabled,表示禁止复制历史数据。
				HistoricalObjectReplication: "disabled",
				// 指定授权OSS进行数据复制的角色名称,且该角色必须已被授予源Bucket执行同区域复制以及目标Bucket接收复制对象的权限。
				SyncRole:                "yourRole",
				EncryptionConfiguration: &keyId,
				SourceSelectionCriteria: &source,
			},
		},
	}

	xmlBody, err := xml.Marshal(reqReplication)
	if err != nil {
		HandleError(err)
	}
	err = client.PutBucketReplication(srcbucketName, string(xmlBody))

	if err != nil {
		HandleError(err)
	}

	fmt.Println("Put Bucket Replication Success!")
}

使用命令行工具ossutil

关于使用ossutil开启同区域复制的具体步骤,请参见put-bucket-replication

使用REST API

如果您的程序自定义要求较高,您可以直接发起REST API请求。直接发起REST API请求需要手动编写代码计算签名。更多信息,请参见PutBucketReplication