Use ossutil to perform resumable file transfer
In unstable network conditions, connection interruptions may cause transfer failures. Use the resumable transfer feature in ossutil 2.0 to continue uploads or downloads from the point of interruption, improving transfer efficiency. Resumable transfers are disabled by default in ossutil 2.0. To enable this feature, specify the --checkpoint-dir parameter when executing the cp command. In unstable network conditions, connection interruptions may cause file transfers to fail. You can use the resumable transfer feature of ossutil 2.0 to continue uploading or downloading from the point of interruption, thereby improving transfer efficiency. In ossutil 2.0, resumable transfer is not enabled by default. To use this feature, specify the --checkpoint-dir parameter when executing the cp command to enable resumable transfer.
Prerequisites
Resumable transfer mechanism
Resumable transfer divides files into multiple smaller segments for transmission, supporting continuation from the point of interruption when a transfer is disrupted, thereby ensuring data integrity and transfer efficiency. Resumable transfer is a method that divides files into smaller parts for transmission. It supports resuming transfers from the last interruption point when transmission is interrupted, ensuring data integrity and transfer efficiency.
-
Starting the transfer: Specify the
--checkpoint-dirparameter when transferring file to enable resumable transfer. -
Information storage: When resumable transfer is enabled, ossutil stores key progress information including file name, size,
upload idand uploaded parts in.ucpfiles undercheckpoint-dir. -
Interruption recovery: If the transfer is interrupted due to network issues or other reasons, restart the transfer. ossutil reads the
.ucpfile incheckpoint-dir, using previously saved progress information to determine which parts were successfully transferred and which need to continue transfer. -
Continuing transfer: ossutil continues to transfer the untransferred parts based on the information of the transferred parts.
Parameter description
|
Parameter |
Description |
|
--checkpoint-dir |
The directory that stores the checkpoint file. |
Examples
Upload a single object
The following example demonstrates how to upload the local file upload.rar to a specified bucket storage space using the ossutil cp command. After the upload operation initiates, a .ucp file is generated in the your-checkpoint-dir directory to store the information of upload progress.
ossutil cp upload.rar oss://bucket --checkpoint-dir D:\\your-checkpoint-dir\If the file upload is interrupted due to network disruption, disk read/write exceptions, or other reasons, you might see the following error message:
Error: upload failed, upload id: 1F71C936CF7B423493BC7A664DB3FD82, cause: operation error UploadPart: Put "https://bucket.oss-cn-hangzhou.aliyuncs.com/upload.rar?partNumber=15&uploadId=1F71C936CF7B423493BC7A664DB3FD82": dial tcp: lookup bucket.oss-cn-hangzhou.aliyuncs.com: no such hostIn this case, you can continue uploading the file by executing the same upload command, using the information saved in the .ucp file: In this scenario, you can resume the upload by reissuing the original upload command, leveraging the checkpoint information stored in the .ucp file:
ossutil cp upload.rar oss://bucket --checkpoint-dir D:\\your-checkpoint-dir\After the transfer continues, ossutil reads the upload progress information stored in the checkpoint-dir directory and starts uploading from the parts that have not been uploaded: Once resumption begins, ossutil reads the upload progress information stored in the checkpoint-dir directory and starts uploading from the untransferred parts:
Total 1 objects,6.833 GiB, Upload... done:(0 objects), 3.000%, avg 4.740 MiB/s Batch upload objects
The following example shows how to use the ossutil cp command to batch upload files from the local_directory to the specified bucket. During this process:
-
The
-uparameter is used to upload only source files that are newer than the target files. The-uparameter specifies uploading only source files newer than the destination. -
The
-rparameter allows recursive copying of the entire directory and all files in its subdirectories. The-rparameter enables recursive copying of entire directory and its contents.
During the upload process, progress information will be stored in a .ucp file generated in the your-checkpoint-dir directory.
ossutil cp -u -r local_directory oss://bucket --checkpoint-dir D:\\your-checkpoint-dir\If the file upload is interrupted due to network disruption, disk read/write exceptions, or other reasons, you might see the following error message:
Error occurs. See more information in file: /ossutil-2.0.6-beta.01091200-mac-arm64/ossutil_output/ossutil_report_20250122_115545_96481.report
FinishWithError: Total 2 files, 7836503730 B, Upload done:(0 objects) failed:(2 objects, 7836503730 B)In this case, you can continue uploading the file by executing the same upload command, using the information saved in the .ucp file: In this scenario, you can resume the upload by reissuing the original upload command, leveraging the checkpoint information stored in the .ucp file:
ossutil cp -u -r local_directory oss://bucket --checkpoint-dir D:\\your-checkpoint-dir\After the transfer continues, ossutil reads the upload progress information stored in the checkpoint-dir directory and starts uploading from the parts that have not been uploaded: Once resumption begins, ossutil reads the upload progress information stored in the checkpoint-dir directory and starts uploading from the untransferred parts:
Total 2 files,7.298 GiB, Upload... done:(0 objects), 1.645%, avg 17.521 MiB/sDownload a single object
The following example shows how to use the cp command of ossutil to download the download.rar object from the remote bucket to the current local directory and name it download.rar. After the file download operation starts, a .dcp file will be generated in the your-checkpoint-dir/.ossutil_checkpoint directory to store the download progress information. The following example demonstrates how to download the download.rar object from a remote bucket to your local current directory using the ossutil cp command and save it as download.rar. Upon download initiation, a .dcp file will be generated in the your-checkpoint-dir/.ossutil_checkpoint directory to store the information of download progress.
ossutil cp oss://bucket/download.rar ./download.rar --checkpoint-dir D:\\your-checkpoint-dir\Batch download objects
The following example demonstrates how to use the ossutil cp command to download all files from the folder directory in the remote bucket to the current local local_directory directory. After the file download operation starts, a .dcp file will be generated in the your-checkpoint-dir/.ossutil_checkpoint directory to store the download progress information. The following example demonstrates how to download all files from the folder directory in a remote bucket to the local_directory using the ossutil cp command. Upon download initiation, a .dcp file will be generated in the your-checkpoint-dir/.ossutil_checkpoint directory to store the information of download progress.
-
The
-uparameter is used to download only source files that are newer than the target files.The-uparameter specifies downloading only source files newer than the destination. -
The
-rparameter allows recursive copying of the entire directory and all files in its subdirectories.The-rparameter enables recursive copying of entire directory and its contents.
ossutil cp -u -r oss://bucket/folder ./local_director --checkpoint-dir D:\\your-checkpoint-dir\