How to upload large files over 5 GB to OSS-Object Storage Service(OSS)-阿里云帮助中心

Object Storage Service (OSS) provides multiple methods for uploading files. Uploading large files that are over 5 GB can be slow or may fail. To upload large files, you can use methods such as multipart upload, resumable upload, ossbrowser, and ossutil.

Upload methods

The size of a single file can range from 0 B to 48.8 TB. To upload large files or many files at once, use one of the following methods to avoid these common issues:

The file is larger than 5 GB and cannot be uploaded using the OSS console, simple upload, form upload, or append upload.
The file is smaller than 5 GB, but you have a poor network connection. A poor connection can cause connection timeouts or disconnections, which leads to upload failures.

Method 1: Multipart upload or resumable upload

The following scenarios are applicable:

The file is larger than 5 GB.
Streaming upload: You can start uploading a file before its total size is known. This is useful for scenarios such as cloud video recording in the video surveillance industry.
Poor network environment: You can use resumable upload for large files. For example, if an upload from a mobile device fails, you can resume the upload from the point of failure instead of restarting the upload from the beginning.
Accelerated upload: To upload a large local file to OSS, you can upload multiple parts in parallel to accelerate the upload.

For more information, see Multipart upload and Resumable upload.

Method 2: Use ossbrowser

Keep the following points in mind when you use ossbrowser:

ossbrowser supports logging in with a Security Token Service (STS) token.
If the AccessKey pair used to log in to ossbrowser has RAM configuration permissions, you can use simplified policy authorization.
When you upload many files, you can increase the number of upload tasks to improve the upload speed.
By default, ossbrowser uses multipart upload and resumable upload. The maximum file size for an upload is 48.8 TB.

For more information, see ossbrowser 2.0.

Method 3: Use ossutil

Keep the following points in mind when you use ossutil:

To upload large files, you can use the cp command in the ossutil command line interface.
You can use the --bigfile-threshold parameter to set the file size threshold for concurrent multipart uploads.
When you upload one or more large files with ossutil, you can use the -jobs and -parallel options to control concurrency and optimize upload performance.
To upload a large batch of files to OSS, you can also use the ossimport tool.
ossutil is recommended if the number of files to upload is not large.

For more information, see Upload files.

FAQ

The following are frequently asked questions about uploading large files:

Simple upload (PutObject) may not support large file uploads for the following reasons.
- If a file is larger than 5 GB, you can use ossutil to perform a multipart upload. This process splits the file into multiple smaller parts and uploads them separately.
- The file is smaller than 5 GB, but you are experiencing network issues. For example, small files upload successfully, but larger files time out. This issue may be caused by a large Maximum Transmission Unit (MTU) setting on the server. On an ECS instance, you can run the netstat -i command to retrieve the current MTU value. The following figure shows that the default MTU value for both the internal and public network interface controllers (NICs) is 1500.
  - Decrease the native MTU value. On a Linux server, you can run the ip link set dev eth0 mtu 1470 command to change the MTU. In this example, the MTU is changed to 1470 for an ECS instance that accesses OSS over the internal network through the eth0 network interface.
  - Use the multipart upload API or the ossutil tool.
Can a single large file be uploaded concurrently?
- Yes. When you use ossutil, you can use the --bigfile-threshold, -jobs, and -parallel options to manage concurrent uploads.
- When you use the multipart upload API provided by an OSS API or a software development kit (SDK), you can write code to call the upload interface concurrently. Some SDKs provide built-in concurrency control for multipart uploads. For example, the Java SDK uses the taskNum parameter, and the Python SDK uses the num_threads parameter.