Backup rules

Full backup

A full backup is a complete copy of your repository, including all refs and objects. This process can be time-consuming. After a successful backup, you can download the full backup file from OSS and easily restore it as a Git repository. For more details on the restoration process, see the sections below.

By default, Codeup performs a full backup for a repository's first backup, as no previous backups exist in OSS. For example, if a repository with the organization repository path alibaba/codeup.git is successfully backed up, the backup files are stored in OSS with the following path structure:

Storage path: All backups for an organization, excluding archives, are stored in a directory named repositories within an OSS bucket. The storage path format for a repository's backup files is /repositories/<organization repository path>/<repository UUID>, where:

• The first-level directory, /repositories, is the root path for all organization repository backups.

• The second-level directory, <organization repository path>, is the path of the organization repository, such as alibaba/codeup in this example.

• The third-level directory, <repository UUID>, is the unique identifier for the repository.

Backup-related files:

• File naming format: A single repository backup generates and uploads multiple files. These files share a common prefix but have different extensions. For example, a backup might generate four files with the prefix alibaba_codeup-full-20210223094414 and varying file extensions.

• .bundle file: This is the core backup file. You can use it to restore the backup as a Git repository.

• .list file: This file contains a summary of the repository's ref information at the time of the backup. By comparing different .list files, you can easily track changes to refs (updates, additions, or deletions) between backups.

• .checksum file: This file contains the checksum value of the repository data at the time of the backup. You can ignore this file.

• .lsremote file: This file records the ls-remote command output for the bundle file after the backup is created. You can ignore this file.

Incremental backup

Incremental backups are slightly more complex than full backups but typically use less backup space and complete faster. If Codeup finds a previous backup in the /repositories/<organization repository path>/<repository UUID> directory, it performs an incremental backup instead of a full one.

Scenario: Incremental backup with repository updates

Assume a full backup file already exists for the repository. Before the next backup is triggered, new code is pushed to a branch in the repository. When a new backup is initiated, Codeup performs an incremental backup because the repository has been updated.

• Storage path: The storage path for an incremental backup is the same as for a full backup. The path format remains /repositories/<organization repository path>/<repository UUID>. The new backup files are:

  # Four incremental backup files are generated. The sequence number is 1, as reflected in the filenames:
  alibaba_codeup-inc-1-20210224074738.bundle
  alibaba_codeup-inc-1-20210224074738.checksum
  alibaba_codeup-inc-1-20210224074738.list
  alibaba_codeup-inc-1-20210224074738.lsremote

• File naming format: Incremental backup files are similar to full backup files and have the same file types. The main difference is the naming convention. Full backup files are identified by full, while incremental backup files are identified by inc-<order>, where <order> is the sequence number of the increment.

Scenario: Incremental backup with no repository updates

If an incremental backup is triggered for a repository with no updates, such as no new commits or only ref deletion operations (for example, deleting a branch), no .bundle or .lsremote file is generated.

# Three incremental backup files are generated. The sequence number is 2 because this is the second incremental backup:
alibaba_codeup-inc-2-20210224082031.bundle.log
alibaba_codeup-inc-2-20210224082031.checksum
alibaba_codeup-inc-2-20210224082031.list

• *.bundle.log file: Since the repository has no updates, this incremental backup does not generate a .bundle or .lsremote file. Instead, it creates a .bundle.log file, which explains why a .bundle file was not created:

  # alibaba_codeup-inc-2-20210224082031.bundle.log file
  Content:
  fatal: Refusing to create empty bundle.

Although no bundle file is created, the backup is still considered successful. This is because ref deletions may have occurred, which can be tracked by comparing the new .list file with the one from the previous backup.

Restore from full backup

1. Download the full backup file from OSS to your local machine:

   ➜  ls |grep bundle
   alibaba_codeup-full-20210224084658.bundle

2. After downloading the bundle file, use it to restore the repository:

   ➜  git clone alibaba_codeup-full-20210224084658.bundle
   Cloning into 'alibaba_codeup-full-20210224084658'...
   Receiving objects: 100% (6/6), done.
   ➜  cd alibaba_codeup-full-20210224084658
   ➜  ls -ltra
   total 8
   drwx------@  7 tenglong.tl  staff  224  2 24 16:54 ..
   drwxr-xr-x   4 tenglong.tl  staff  128  2 24 16:54 .
   -rw-r--r--   1 tenglong.tl  staff    6  2 24 16:54 test.file
   drwxr-xr-x  13 tenglong.tl  staff  416  2 24 16:54 .git

Restore from incremental backup

Restoring from an incremental backup is similar to restoring from a full backup but slightly more complex. It involves a fetch operation instead of a clone. Assume you have backup files created at four different times (T0, T1, T2, and T3):

• T0: The corresponding bundle file is alibaba_codeup-full-20210224084658.bundle.

• T1: The corresponding bundle file is alibaba_codeup-inc-1-20210224084659.bundle.

• T2: The repository has no incremental updates compared to T1, so no .bundle file is generated. Instead, a .bundle.log file is created.

• T3: The corresponding bundle file is alibaba_codeup-inc-3-20210224084700.bundle.

1. First, restore the repository from the T0 backup file. Download the full backup file alibaba_codeup-full-20210224084658.bundle to your local machine and run the git clone command:

   ➜  git clone alibaba_codeup-full-20210224084658.bundle recovery.git

2. recovery.git is now a copy of the repository at time T0.

3. To update the recovery.git repository to its state at time T1, download the incremental backup file from T1, alibaba_codeup-inc-1-20210224084659.bundle, to your local machine and run the following commands:

   ➜  cd recovery.git
   ➜  git remote add inc-1 ~/Downloads/alibaba_codeup-inc-1-20210224084659.bundle 
   ➜  git fetch inc-1
   Receiving objects: 100% (6/6), done.
   From /Users/tenglong.tl/Downloads/alibaba_codeup-inc-1-20210224084659.bundle
    * [new branch]      topic_2    -> inc-1/topic_2
    * [new branch]      master     -> inc-1/master
    * [new tag]         v2.0       -> v2.0

4. The incremental changes from T1 have been successfully fetched into the recovery.git repository. To switch to one of the updated branches, run the following command:

   ➜  git checkout -b topic_2 inc-1/topic_2
   Branch 'topic_2' set up to track remote branch 'topic_2' from 'inc-1'.
   Switched to a new branch 'topic_2'

5. To apply the changes from T3 on top of the T0 repository, download the incremental backup file from T3, alibaba_codeup-inc-3-20210224084700.bundle, and run the following commands:

   ➜  git remote add inc-3 ~/Downloads/alibaba_codeup-inc-3-20210224084700.bundle 
   ➜  git fetch inc-3
   Receiving objects: 100% (3/3), done.
   From /Users/tenglong.tl/Downloads/alibaba_codeup-inc-3-20210224084700.bundle
    * [new branch]      master     -> inc-3/master

6. Similarly, the incremental changes from T3 have now been fetched into the recovery.git repository. The master branch was updated. To check out the master branch as it was at time T3, run the following commands:

   ➜  git checkout master
   ➜  git pull inc-3 master
   From /Users/tenglong.tl/Downloads/alibaba_codeup-inc-3-20210224084700.bundle
    * branch            master     -> FETCH_HEAD
   Updating 324711e..4bdbff9
   Fast-forward
    lol.txt | 2 ++
    1 file changed, 2 insertions(+)
    create mode 100644 lol.txt

   The master branch is now restored to its state at T3.