Use OSS SDK for Go to configure pre-upload client-side encryption-Object Storage Service(OSS)-阿里云帮助中心

Disclaimer

When you use client-side encryption, you must ensure the integrity and validity of the CMK. If the CMK is incorrectly used or lost due to improper maintenance, you are responsible for all losses and consequences caused by decryption failures.
When you copy or migrate encrypted data, you are responsible for the integrity and validity of object metadata. If the encrypted metadata is incorrect or lost due to improper maintenance, you are responsible for all losses and consequences caused by data decryption failures.

Scenarios

Highly sensitive data: You want to encrypt sensitive data, such as Personal Identifiable Information (PII), transaction records, and health data, before the data leaves your local environment. This ensures effective protection of your data even when the data is intercepted.
Regulatory compliance: Regulations, such as Health Insurance Portability and Accountability Act (HIPAA) and General Data Protection Regulation (GDPR), require encryption of data stored on third-party platforms. Client-side encryption meets these requirements because CMKs are managed by users and not passed over networks or to cloud service providers.
Strong permission control: Your enterprise or developer may want to have full control over the encryption process, including the selection of encryption algorithms and the management and rotation of keys. In this case, you can use client-side encryption to ensure that only authorized users can decrypt and access data.
Secure data migration across regions: Client-side encryption helps keep data encrypted throughout migration. This enhances data security during transmission over the Internet.

Notes

In this topic, the public endpoint of the China (Hangzhou) region is used. If you want to access OSS from other Alibaba Cloud services in the same region as OSS, use an internal endpoint. For more information about OSS regions and endpoints, see Regions and Endpoints.
In this topic, access credentials are obtained from environment variables. For more information about how to configure access credentials, see Configure access credentials.
This topic demonstrates creating an OSSClient instance with an OSS endpoint. For alternative configurations, such as using a custom domain or authenticating with credentials from Security Token Service (STS), see Configure a client (Go SDK V1).

Background information

In client-side encryption, a random data key is generated for each object to perform symmetric encryption on the object. The client uses a CMK to encrypt the random data key. The encrypted data key is uploaded as part of the object metadata and stored in the OSS server. When an encrypted object is downloaded, the client uses the CMK to decrypt the random data key, and then uses the decrypted data key to decrypt the object. To ensure data security, the CMK is used only on the client and is not transmitted over the network or stored on the server.

Important

Client-side encryption supports multipart upload for objects larger than 5 GB. You must specify the total object size and each part size. All parts except the last must be the same size and a multiple of 16 bytes.
After you upload objects encrypted on the client, object metadata related to client-side encryption is protected, and cannot be modified by calling the CopyObject operation.

Encryption methods

A master key is currently used in the following two ways:

KMS-managed CMKs

When you use a CMK managed in Key Management Service (KMS) for client-side encryption, you must provide OSS SDK for Python with the CMK ID.
RSA-based CMKs managed by yourself

When you use a CMK managed by yourself for client-side encryption, you must send the public key and the private key of your CMK to OSS SDK for Python as parameters.

You can use the preceding encryption methods to prevent data leaks and protect your data on the client. Even if your data is leaked, the data cannot be decrypted by others.

Object metadata related to client-side encryption

Parameter	Description	Required
x-oss-meta-client-side-encryption-key	The encrypted data key. The encrypted data key is a string encrypted by using a CMK and encoded in Base64.	Yes
x-oss-meta-client-side-encryption-start	The initial value that is randomly generated for data encryption. The initial value is a string encrypted by using a CMK and encoded in Base64.	Yes
x-oss-meta-client-side-encryption-cek-alg	The algorithm that is used to encrypt data.	Yes
x-oss-meta-client-side-encryption-wrap-alg	The algorithm that is used to encrypt the data key.	Yes
x-oss-meta-client-side-encryption-matdesc	The description of the CMK in JSON format. Warning We recommend that you configure a description for each CMK and store the mapping relationship between the CMK and its description. A CMK without a matching description cannot be replaced.	No
x-oss-meta-client-side-encryption-unencrypted-content-length	The length of data before encryption. If content-length is not specified, this parameter is not generated.	No
x-oss-meta-client-side-encryption-unencrypted-content-md5	The MD5 hash of the data before encryption. If Content-MD5 is not specified, this parameter is not generated.	No
x-oss-meta-client-side-encryption-data-size	The total size of the object for which you want to perform multipart upload. If you want to encrypt the object on the client side, you must specify this parameter when you initialize a multipart upload task for the object.	Yes (for multipart upload)
x-oss-meta-client-side-encryption-part-size	The size of each part in a multipart upload task for the object. If you want to encrypt the object on the client side, you must specify this parameter when you initialize a multipart upload task for the object. Note The size of each part must be an integer multiple of 16 bytes.	Yes (for multipart upload)

Sample code

Use RSA-based CMKs to encrypt objects in simple upload or decrypt objects in simple download

The following code provides an example on how to use an RSA-based CMK to encrypt objects in simple upload or decrypt objects in simple download:

package main

import (
	"bytes"
	"io"
	"log"

	"github.com/aliyun/aliyun-oss-go-sdk/oss"
	osscrypto "github.com/aliyun/aliyun-oss-go-sdk/oss/crypto"
)

func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		log.Fatalf("Error creating credentials provider: %v", err)
	}

	// Create an OSSClient instance. 
	// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint. 
	// Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to cn-hangzhou. Specify the actual region.
	clientOptions := []oss.ClientOption{oss.SetCredentialsProvider(&provider)}
	clientOptions = append(clientOptions, oss.Region("yourRegion"))
	// Specify the version of the signature algorithm.
	clientOptions = append(clientOptions, oss.AuthVersion(oss.AuthV4))
	client, err := oss.New("yourEndpoint", "", "", clientOptions...)
	if err != nil {
		log.Fatalf("Error creating OSS client: %v", err)
	}

	// Specify a description for the CMK. The description cannot be modified after being specified. You can specify only one description for a CMK. 
	// If all objects share a CMK, you can leave the description of the CMK empty. In this case, the CMK cannot be replaced. 
	// If the description of the CMK is left empty, the client cannot determine which CMK to use for decryption. 
	// We recommend that you configure a description in the form of a JSON string for each CMK and store the mapping relationship between the CMK and the description on the client. The server does not store the mapping relationship. 

	// Obtain the mapping relationship based on the CMK description that is a JSON string. 
	materialDesc := map[string]string{
		"desc": "your master encrypt key material describe information",
	}

	// Create a CMK object based on the description of the CMK. 
	// Set yourRsaPublicKey to the public key of the CMK that you manage, and set yourRsaPrivateKey to the private key of the CMK that you manage. 
	masterRsaCipher, err := osscrypto.CreateMasterRsa(materialDesc, "yourRsaPublicKey", "yourRsaPrivateKey")
	if err != nil {
		log.Fatalf("Error creating master RSA cipher: %v", err)
	}

	// Create an encryption operation based on the CMK object and perform encryption in AES CTR mode. 
	contentProvider := osscrypto.CreateAesCtrCipher(masterRsaCipher)

	// Obtain an existing bucket to be used for client-side encryption. 
	// The bucket for which client-side encryption is configured is used in the same manner in which a common bucket is used.
	cryptoBucket, err := osscrypto.GetCryptoBucket(client, "yourBucketName", contentProvider)
	if err != nil {
		log.Fatalf("Error getting crypto bucket: %v", err)
	}

	// Encryption is automatically performed when you call the PutObject operation. 
	err = cryptoBucket.PutObject("yourObjectName", bytes.NewReader([]byte("yourObjectValueByteArrary")))
	if err != nil {
		log.Fatalf("Error putting object: %v", err)
	}

	// Decryption is automatically performed when you call the GetObject operation. 
	body, err := cryptoBucket.GetObject("yourObjectName")
	if err != nil {
		log.Fatalf("Error getting object: %v", err)
	}
	defer body.Close()

	data, err := io.ReadAll(body)
	if err != nil {
		log.Fatalf("Error reading object data: %v", err)
	}
	log.Printf("Data: %s", string(data))
}

Use RSA-based CMKs to encrypt objects in multipart upload

The following code provides an example on how to use an RSA-based CMK to encrypt objects in multipart upload:

package main

import (
	"log"
	"os"

	"github.com/aliyun/aliyun-oss-go-sdk/oss"
	osscrypto "github.com/aliyun/aliyun-oss-go-sdk/oss/crypto"
)

func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		log.Fatalf("Error creating credentials provider: %v", err)
	}

	// Create an OSSClient instance. 
	// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint. 
	// Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to cn-hangzhou. Specify the actual region.
	clientOptions := []oss.ClientOption{oss.SetCredentialsProvider(&provider)}
	clientOptions = append(clientOptions, oss.Region("yourRegion"))
	// Specify the version of the signature algorithm.
	clientOptions = append(clientOptions, oss.AuthVersion(oss.AuthV4))
	client, err := oss.New("yourEndpoint", "", "", clientOptions...)
	if err != nil {
		log.Fatalf("Error creating OSS client: %v", err)
	}

	// Specify a description for the CMK. The description cannot be modified after being specified. You can specify only one description for a CMK. 
	// If all objects share a CMK, you can leave the description of the CMK empty. In this case, the CMK cannot be replaced. 
	// If the description of the CMK is left empty, the client cannot determine which CMK to use for decryption. 
	// We recommend that you configure a description in the form of a JSON string for each CMK and store the mapping relationship between the CMK and the description on the client. The server does not store the mapping relationship. 

	// Obtain the mapping relationship based on the CMK description that is a JSON string
	materialDesc := map[string]string{
		"desc": "your master encrypt key material describe information",
	}

	// Create a CMK object based on the description of the CMK. 
	// Set yourRsaPublicKey to the public key of the CMK that you manage, and set yourRsaPrivateKey to the private key of the CMK that you manage.
	masterRsaCipher, err := osscrypto.CreateMasterRsa(materialDesc, "yourRsaPublicKey", "yourRsaPrivateKey")
	if err != nil {
		log.Fatalf("Error creating master RSA cipher: %v", err)
	}

	// Create an encryption operation based on the CMK object and perform encryption in AES CTR mode. 
	contentProvider := osscrypto.CreateAesCtrCipher(masterRsaCipher)

	// Obtain an existing bucket to be used for client-side encryption. 
	// The bucket for which client-side encryption is configured is used in the same manner in which a common bucket is used. 
	cryptoBucket, err := osscrypto.GetCryptoBucket(client, "yourBucketName", contentProvider)
	if err != nil {
		log.Fatalf("Error getting crypto bucket: %v", err)
	}

	fileName := "yourLocalFilePath"
	fileInfo, err := os.Stat(fileName)
	if err != nil {
		log.Fatalf("Error getting file info: %v", err)
	}
	fileSize := fileInfo.Size()

	// Encrypt the context. 
	var cryptoContext osscrypto.PartCryptoContext
	cryptoContext.DataSize = fileSize

	// Specify the number of parts that you want to encrypt. The actual number is based on the calculation result. 
	expectPartCount := int64(10)

	// Specify the size of a part encrypted in AES CTR mode, which is an integer multiple of 16 bytes. 
	cryptoContext.PartSize = (fileSize / expectPartCount / 16) * 16

	// Initiate Multipart Upload.
	imur, err := cryptoBucket.InitiateMultipartUpload("yourObjectName", &cryptoContext)
	if err != nil {
		log.Fatalf("Error initiating multipart upload: %v", err)
	}

	// Split the file.
	chunks, err := oss.SplitFileByPartSize(fileName, cryptoContext.PartSize)
	if err != nil {
		log.Fatalf("Error splitting file: %v", err)
	}

	var partsUpload []oss.UploadPart
	for _, chunk := range chunks {
		part, err := cryptoBucket.UploadPartFromFile(imur, fileName, chunk.Offset, chunk.Size, int(chunk.Number), cryptoContext)
		if err != nil {
			log.Fatalf("Error uploading part: %v", err)
		}
		partsUpload = append(partsUpload, part)
	}

	// Complete the task.
	_, err = cryptoBucket.CompleteMultipartUpload(imur, partsUpload)
	if err != nil {
		log.Fatalf("Error completing multipart upload: %v", err)
	}

	log.Println("Multipart upload completed successfully")
}

Decrypt objects that are encrypted by using different RSA-based CMKs

The following code provides an example on how to decrypt objects that are encrypted by using different RSA-based CMKs:

package main

import (
	"bytes"
	"io"
	"log"

	"github.com/aliyun/aliyun-oss-go-sdk/oss"
	osscrypto "github.com/aliyun/aliyun-oss-go-sdk/oss/crypto"
)

// Query the CMK based on the description of the CMK. To decrypt objects that are encrypted by using different CMKs, you must perform this operation. 
type MockRsaManager struct{}

func (mg *MockRsaManager) GetMasterKey(matDesc map[string]string) ([]string, error) {
	keyList := []string{"yourRsaPublicKey", "yourRsaPrivateKey"}
	return keyList, nil
}

// Decrypt objects encrypted by using different CMKs. 
func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		log.Fatalf("Error creating credentials provider: %v", err)
	}

	// Create an OSSClient instance. 
	// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint. 
	// Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to cn-hangzhou. Specify the actual region.
	clientOptions := []oss.ClientOption{oss.SetCredentialsProvider(&provider)}
	clientOptions = append(clientOptions, oss.Region("yourRegion"))
	// Specify the version of the signature algorithm.
	clientOptions = append(clientOptions, oss.AuthVersion(oss.AuthV4))
	client, err := oss.New("yourEndpoint", "", "", clientOptions...)
	if err != nil {
		log.Fatalf("Error creating OSS client: %v", err)
	}

	// Specify a description for the CMK. The description cannot be modified after being specified. You can specify only one description for a CMK.
	// If all objects share a CMK, you can leave the description of the CMK empty. In this case, the CMK cannot be replaced. 
	// If the description of the CMK is left empty, the client cannot determine which CMK to use for decryption. 
	// We recommend that you configure a description in the form of a JSON string for each CMK and store the mapping relationship between the CMK and the description on the client. The server does not store the mapping relationship. 

	// Obtain the mapping relationship based on the CMK description that is a JSON string. 
	materialDesc := map[string]string{
		"desc": "your master encrypt key material describe information",
	}

	// Create a CMK object based on the description of the CMK.
	// Set yourRsaPublicKey to the public key of the CMK that you manage, and set yourRsaPrivateKey to the private key of the CMK that you manage. 
	masterRsaCipher, err := osscrypto.CreateMasterRsa(materialDesc, "yourRsaPublicKey", "yourRsaPrivateKey")
	if err != nil {
		log.Fatalf("Error creating master RSA cipher: %v", err)
	}

	// Create an encryption operation based on the CMK object and perform encryption in AES CTR mode. 
	contentProvider := osscrypto.CreateAesCtrCipher(masterRsaCipher)

	// To decrypt objects that are encrypted by using different CMKs, you must perform this operation. 
	var mockRsaManager MockRsaManager
	var options []osscrypto.CryptoBucketOption
	options = append(options, osscrypto.SetMasterCipherManager(&mockRsaManager))

	// Obtain an existing bucket to be used for client-side encryption. 
	// The bucket for which client-side encryption is configured is used in the same manner in which a common bucket is used. 
	cryptoBucket, err := osscrypto.GetCryptoBucket(client, "yourBucketName", contentProvider, options...)
	if err != nil {
		log.Fatalf("Error getting crypto bucket: %v", err)
	}

	// Encryption is automatically performed when you call the PutObject operation. 
	err = cryptoBucket.PutObject("yourObjectName", bytes.NewReader([]byte("yourObjectValueByteArrary")))
	if err != nil {
		log.Fatalf("Error putting object: %v", err)
	}

	// Decryption is automatically performed when you call the GetObject operation. 
	body, err := cryptoBucket.GetObject("otherObjectNameEncryptedWithOtherRsa")
	if err != nil {
		log.Fatalf("Error getting object: %v", err)
	}
	defer body.Close()

	data, err := io.ReadAll(body)
	if err != nil {
		log.Fatalf("Error reading object data: %v", err)
	}
	log.Printf("Data: %s", string(data))
}

Use CMKs in KMS to encrypt objects in simple upload or decrypt objects in simple download

The following code provides an example on how to use a CMK in KMS to encrypt objects in simple upload or decrypt objects in simple download:

package main

import (
	"bytes"
	"io"
	"log"

	"github.com/aliyun/alibaba-cloud-sdk-go/services/kms"
	"github.com/aliyun/aliyun-oss-go-sdk/oss"
	crypto "github.com/aliyun/aliyun-oss-go-sdk/oss/crypto"
)

func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		log.Fatalf("Error creating credentials provider: %v", err)
	}

	// Create an OSSClient instance. 
	// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint.
	// Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to cn-hangzhou. Specify the actual region.
	clientOptions := []oss.ClientOption{oss.SetCredentialsProvider(&provider)}
	clientOptions = append(clientOptions, oss.Region("yourRegion"))
	// Specify the version of the signature algorithm.
	clientOptions = append(clientOptions, oss.AuthVersion(oss.AuthV4))
	client, err := oss.New("yourEndpoint", "", "", clientOptions...)
	if err != nil {
		log.Fatalf("Error creating OSS client: %v", err)
	}

  // Create a KMSClient instance. 
	kmsClient, err := kms.NewClientWithAccessKey("yourKmsRegion", "yourKmsAccessKeyId", "yourKmsAccessKeySecret")
	if err != nil {
		log.Fatalf("Error creating KMS client: %v", err)
	}

  // Specify a description for the CMK. The description cannot be modified after being specified. You can specify only one description for a CMK. 
  // If all objects share a CMK, you can leave the description of the CMK empty. In this case, the CMK cannot be replaced. 
  // If the description of the CMK is left empty, the client cannot determine which CMK to use for decryption. 
  // We recommend that you configure a description in the form of a JSON string for each CMK and store the mapping relationship between the CMK and the description on the client. The server does not store the mapping relationship. 

    // Obtain the mapping relationship based on the CMK description that is a JSON string.
	materialDesc := map[string]string{
		"desc": "your kms encrypt key material describe information",
	}

  // Create a CMK object based on the description of the CMK. 
  // Set yourKmsId to the ID of the CMK in KMS. 
	masterkmsCipher, err := crypto.CreateMasterAliKms(materialDesc, "yourKmsId", kmsClient)
	if err != nil {
		log.Fatalf("Error creating master KMS cipher: %v", err)
	}

  // Create an encryption operation based on the CMK object and perform encryption in AES CTR mode. 
	contentProvider := crypto.CreateAesCtrCipher(masterkmsCipher)

  // Obtain an existing bucket to be used for client-side encryption. 
  // The bucket for which client-side encryption is configured is used in the same manner in which a common bucket is used. 
	cryptoBucket, err := crypto.GetCryptoBucket(client, "yourBucketName", contentProvider)
	if err != nil {
		log.Fatalf("Error getting crypto bucket: %v", err)
	}

  // Encryption is automatically performed when you call the PutObject operation. 
	err = cryptoBucket.PutObject("yourObjectName", bytes.NewReader([]byte("yourObjectValueByteArrary")))
	if err != nil {
		log.Fatalf("Error putting object: %v", err)
	}

  // Decryption is automatically performed when you call the GetObject operation. 
	body, err := cryptoBucket.GetObject("yourObjectName")
	if err != nil {
		log.Fatalf("Error getting object: %v", err)
	}
	defer body.Close()

	data, err := io.ReadAll(body)
	if err != nil {
		log.Fatalf("Error reading object data: %v", err)
	}
	log.Printf("Data: %s", string(data))
}

References

For more information about the API operation that you can call to configure server-side encryption, see SetBucketEncryption.
For more information about the API operation that you can call to query server-side encryption configurations, see GetBucketEncryption.