Multipart upload (Go SDK V2)
The multipart upload feature in Object Storage Service (OSS) lets you split a large object into multiple parts. You can upload these parts separately. After all parts are uploaded, you can call the CompleteMultipartUpload operation to combine the parts into a complete object.
Notes
-
The sample code in this topic uses the
cn-hangzhouregion ID for the China (Hangzhou) region and uses a public endpoint by default. If you access OSS from other Alibaba Cloud services in the same region, use an internal endpoint. For more information about OSS regions and endpoints, see Regions and endpoints. -
The examples in this topic read access credentials from environment variables. For more information about how to configure access credentials, see Configure access credentials.
-
To perform a multipart upload, you must have the
oss:PutObjectpermission. For more information, see Grant custom permissions to a RAM user.
Multipart upload process
A multipart upload consists of the following three steps:
-
Initialize a multipart upload event.
Call the Client.InitiateMultipartUpload method to obtain a globally unique upload ID from OSS.
-
Upload parts.
Call the Client.UploadPart method to upload the parts.
Note-
For a given upload ID, the part number identifies the position of the part within the object. If you upload a new part with an existing part number, OSS overwrites the existing part.
-
OSS includes the MD5 hash of the received part in the ETag header of the response.
-
OSS calculates the MD5 hash of the uploaded data and compares it with the hash calculated by the software development kit (SDK). If the hashes do not match, OSS returns an `InvalidDigest` error code.
-
-
Complete the multipart upload.
After all parts are uploaded, call the Client.CompleteMultipartUpload method to combine all parts into a complete object.
Sample code
The following code shows how to split a large local file into multiple parts, upload the parts to a bucket concurrently, and then combine the parts into a complete object.
package main
import (
"bufio"
"bytes"
"context"
"flag"
"io"
"log"
"os"
"sync"
"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
)
// Define global variables.
var (
region string // The region where the bucket is located.
bucketName string // The name of the source bucket.
objectName string // The name of the source object.
)
// The init function initializes command-line parameters.
func init() {
flag.StringVar(®ion, "region", "", "The region in which the bucket is located.")
flag.StringVar(&bucketName, "bucket", "", "The name of the source bucket.")
flag.StringVar(&objectName, "object", "", "The name of the source object.")
}
func main() {
// Parse command-line parameters.
flag.Parse()
// Define the upload ID.
var uploadId string
// Check if the source bucket name is empty.
if len(bucketName) == 0 {
flag.PrintDefaults()
log.Fatalf("invalid parameters, source bucket name required")
}
// Check if the region is empty.
if len(region) == 0 {
flag.PrintDefaults()
log.Fatalf("invalid parameters, region required")
}
// Check if the source object name is empty.
if len(objectName) == 0 {
flag.PrintDefaults()
log.Fatalf("invalid parameters, source object name required")
}
// Load default configurations and set the credential provider and region.
cfg := oss.LoadDefaultConfig().
WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
WithRegion(region)
// Create an OSS client.
client := oss.NewClient(cfg)
// Initialize the multipart upload request.
initRequest := &oss.InitiateMultipartUploadRequest{
Bucket: oss.Ptr(bucketName),
Key: oss.Ptr(objectName),
}
initResult, err := client.InitiateMultipartUpload(context.TODO(), initRequest)
if err != nil {
log.Fatalf("failed to initiate multipart upload %v", err)
}
// Print the result of the multipart upload initialization.
log.Printf("initiate multipart upload result:%#v\n", *initResult.UploadId)
uploadId = *initResult.UploadId
// Initialize the wait group and mutex.
var wg sync.WaitGroup
var parts []oss.UploadPart
count := 3
var mu sync.Mutex
// Read the local file into memory. Replace yourLocalFile with the actual local file name and path.
file, err := os.Open("yourLocalFile")
if err != nil {
log.Fatalf("failed to open local file %v", err)
}
defer file.Close()
bufReader := bufio.NewReader(file)
content, err := io.ReadAll(bufReader)
if err != nil {
log.Fatalf("failed to read local file %v", err)
}
log.Printf("file size: %d\n", len(content))
// Calculate the size of each part.
chunkSize := len(content) / count
if chunkSize == 0 {
chunkSize = 1
}
// Start multiple goroutines to upload parts concurrently.
for i := 0; i < count; i++ {
start := i * chunkSize
end := start + chunkSize
if i == count-1 {
end = len(content)
}
wg.Add(1)
go func(partNumber int, start, end int) {
defer wg.Done()
// Create an upload part request.
partRequest := &oss.UploadPartRequest{
Bucket: oss.Ptr(bucketName), // The name of the destination bucket.
Key: oss.Ptr(objectName), // The name of the destination object.
PartNumber: int32(partNumber), // The part number.
UploadId: oss.Ptr(uploadId), // The upload ID.
Body: bytes.NewReader(content[start:end]), // The content of the part.
}
// Send the upload part request.
partResult, err := client.UploadPart(context.TODO(), partRequest)
if err != nil {
log.Fatalf("failed to upload part %d: %v", partNumber, err)
}
// Record the result of the part upload.
part := oss.UploadPart{
PartNumber: partRequest.PartNumber,
ETag: partResult.ETag,
}
// Use a mutex to protect shared data.
mu.Lock()
parts = append(parts, part)
mu.Unlock()
}(i+1, start, end)
}
// Wait for all goroutines to complete.
wg.Wait()
// Complete the multipart upload request.
request := &oss.CompleteMultipartUploadRequest{
Bucket: oss.Ptr(bucketName),
Key: oss.Ptr(objectName),
UploadId: oss.Ptr(uploadId),
CompleteMultipartUpload: &oss.CompleteMultipartUpload{
Parts: parts,
},
}
result, err := client.CompleteMultipartUpload(context.TODO(), request)
if err != nil {
log.Fatalf("failed to complete multipart upload %v", err)
}
// Print the result of the completed multipart upload.
log.Printf("complete multipart upload result:%#v\n", result)
}
Common scenarios
References
-
For the complete sample code for multipart upload, see the GitHub example.
-
A complete multipart upload involves three API operations:
-
For information about the API operation to initialize a multipart upload event, see InitiateMultipartUpload.
-
For information about the API operation to upload a part, see UploadPart.
-
For information about the API operation to complete a multipart upload, see CompleteMultipartUpload.
-
-
For information about the API operation to cancel a multipart upload event, see AbortMultipartUpload.
-
For information about the API operation to list uploaded parts, see NewListPartsPaginator.
-
For information about the API operation to list all in-progress multipart upload events, which are events that are initiated but not yet completed or canceled, see NewListMultipartUploadsPaginator.