sync (Synchronize files between OSS buckets)

Precautions

• Synchronizing files between OSS buckets requires the oss:GetObject, oss:ListObjects, oss:PutObject, and oss:DeleteObject permissions. Grant custom permissions to a RAM user.

• Binary name

  For ossutil 1.6.16 and later, you can use ossutil directly as the binary name in the command line. You do not need to update the binary name based on the operating system. If you use a version of ossutil earlier than 1.6.16, update the binary name based on your operating system. For more information, see ossutil command reference.

• Number of files

  When you run the sync command without the --delete option, there is no limit on the number of files that you can synchronize at a time. If you include the --delete option, you can synchronize up to 1 million files at a time. If the number of files to synchronize exceeds 1 million, the over max sync numbers 1000000. error is returned.

• Differences between the sync and cp commands

  • sync recursively traverses all files and subdirectories in the specified folder. cp requires the -r option for recursive operations.

  • sync supports the --delete option to remove destination files not present in the source, keeping only the current synchronization result. cp does not support --delete.

  • sync does not support --version-id for synchronizing previous object versions in versioning-enabled buckets. cp supports --version-id.

  Beyond these differences, sync and cp work similarly. cp (Copy files).

Limits

You cannot synchronize Cold Archive or Deep Cold Archive objects from a source bucket to a destination bucket.

Command syntax

ossutil sync cloud_url cloud_url
[-f --force]
[-u --update]
[--delete]
[--enable-symlink-dir]
[--disable-all-symlink]
[--disable-ignore-error]
[--only-current-dir]
[--output-dir <value>]
[--bigfile-threshold <value>]
[--part-size <value>]
[--checkpoint-dir <value>]
[--encoding-type <value>]
[--include <value>]
[--exclude <value>]
[--meta <value>]
[--acl <value>]
[--maxupspeed <value>]
[--disable-crc64]
[--payer <value>]
[-j, --job <value>]
[--parallel <value>]
[--retry-times <value>]
[--tagging <value>]

Parameters and options:

Examples

examplebucket1          examplebucket2
├── exampledir/            └── destdir/
│      ├── a.txt               ├── c.txt
│      └── b.txt               └── e.txt
└── srcdir/     
        └── d.txt       

Example scenarios:

• Synchronize the exampledir folder to the srcdir folder within the examplebucket1 bucket.

  ossutil sync oss://examplebucket1/exampledir/  oss://examplebucket1/srcdir/

  Result: all files from exampledir are synchronized to srcdir.

  examplebucket1                    examplebucket1         
  ├── exampledir/                  ├── exampledir/
  │      ├── a.txt                 │      ├── a.txt 
  │      └── b.txt                 │      └── b.txt 
  └── srcdir/                      └── srcdir/    
          └── d.txt                        ├── a.txt
                                           ├── b.txt
                                           └── d.txt     

• Synchronize the exampledir folder from examplebucket1 to the destdir folder in examplebucket2.

  ossutil sync oss://examplebucket1/exampledir/ oss://examplebucket2/destdir/

  Result: all files from exampledir in examplebucket1 are synchronized to destdir in examplebucket2.

  examplebucket1          examplebucket2
  ├── exampledir/            └── destdir/
  │      ├── a.txt               ├── a.txt  
  │      └── b.txt               ├── b.txt
  └── srcdir/                     ├── c.txt 
          └── d.txt               └── e.txt

• Synchronize all files from examplebucket1 to examplebucket2 and use the --delete option to delete files that exist in examplebucket2 but not in examplebucket1.

  ossutil sync oss://examplebucket1 oss://examplebucket2 --delete

  Result: files from examplebucket1 are synchronized to examplebucket2. Existing files c.txt and e.txt in examplebucket2 are deleted.

  examplebucket1          examplebucket2
  ├── exampledir/            ├── exampledir/
  │      ├── a.txt           │      ├── a.txt  
  │      └── b.txt           │      └── b.txt
  └── srcdir/                └── srcdir/  
          └── d.txt                  └── d.txt                  

• On success, output similar to the following is returned:

  Succeed: Total num: 2, size: 750,081. OK num: 2(upload 2 files).
  
  average speed 1641000(byte/s)

Common options

To access a bucket in a different region, use -e to specify the endpoint. To access a bucket owned by a different Alibaba Cloud account, use -i for the AccessKey ID and -k for the AccessKey secret.

ossutil sync oss://examplebucket/srcfolder/  oss://testbucket/examplefolder/ -e oss-cn-shanghai.aliyuncs.com -i yourAccessKeyID  -k yourAccessKeySecret

Common options.