Traditionally, the UCC's backup system has left much to be desired, such as "existing" or "running on reliable hardware", let alone living up to the [[https://www.lawtechnologytoday.org/2014/12/backups-rule-three/|Rule of Three]].

Backups at present run on [[Mollitz]], which is housed in another location (off-site backups!). Contact <trs80@ucc.asn.au> if you need to know where it is living.

Backups are run using [[http://rdiff-backup.nongnu.org/|rdiff-backup]], a disk-based incremental backup system. These are managed by [[https://gitlab.ucc.asn.au/UCC/rdiff-manager|rdiff-manager]], a Python wrapper written by DavidAdam.

== Adding new machines ==

On the target system (the machine you want to back up):
 * Make sure the UCC backup key is installed (e.g. with authroot)
 * Install `rdiff-backup` packages.

On the backup server (`mollitz`):
 * Copy `/backups/conf/example-copy-me` to `/backups/conf/<HOSTNAME>.conf`
 * Edit `/backups/conf/<HOSTNAME>.conf` as required - the syntax is documented in [[http://manpages.ubuntu.com/manpages/bionic/en/man1/rdiff-backup.1.html|rdiff-backup(1)]] under FILE SELECTION
 * Add the SSH host key using `su -c 'ssh-keyscan HOSTNAME >> ~/.ssh/known_hosts' backups`

Now wait until the nightly backup run. The output is:

  * sent by email to hostmasters
  * successful backups leave a log in `/backups/<HOSTNAME>/rdiff-backup-data/backup.log` and `/backups/<HOSTNAME>/rdiff-backup-data/session_statistics.<TIMESTAMP>.data`
  * partially successful backups leave a log in `/backups/<HOSTNAME>/rdiff-backup-data/error_log.<TIMESTAMP>.data.gz`
    * /!\ In some cases, e.g. if a particular file constantly changes during each and every backup run, a successful backup or update may never be possible
    * TODO: this could be mitigated by backing up a stable snapshot, instead of the live filesystem

== Checking backup status ==

To list all the backups available for a particular host, or to see when it was last successful, on the backup server (Mollitz) run:
 * `rdiff-backup --list-increments /backups/<HOSTNAME>`

To list how much data is taken for each incremental backup (which is much slower), on the backup server (Mollitz) run:
 * `rdiff-backup --list-increment-sizes /backups/<HOSTNAME>`

== Restoring a backup ==

To restore files from backup, on the backup server (Mollitz):
 * Run `rdiff-backup --list-increments /backups/<HOSTNAME>` and choose a backup to restore from
 * Copy the timestamp from the increment list - `2022-02-22T02:00:03+08:00`
 * Decide where you are going to restore the files - locally (i.e. to Mollitz), where you can inspect them, or back to the remote host

=== Restoring files locally ===
 * Run `mkdir /backups/tmp/<HOSTNAME>`
 * Run `rdiff-backup --restore <TIMESTAMP> /backups/<HOSTNAME>/path/to/file-or-directory/you/want/to/restore /backups/tmp/<HOSTNAME>`
For example, using `rdiff-backup -r  2022-02-22T02:00:03+08:00 /backups/merlo/etc /backups/tmp/merlo` will restore the contents of merlo's `/etc/` directory as of 22nd February to `/backups/tmp/merlo`.

=== Restoring files remotely ===
/!\ Be careful - this is easy to mess up, particularly if you are trying to restore to the original path. Note the double-colons in the restore path!
 * Run `rdiff-backup --restore <TIMESTAMP> /backups/<HOSTNAME>/path/to/file-or-directory/you/want/to/restore root@<HOSTNAME>::/path/to/file-or-directory/you/want/to/restore`
For example, `rdiff-backup -r  2022-02-22T02:00:03+08:00 /backups/merlo/etc merlo::/restored/etc` will restore the contents of merlo's `/etc/` directory as of 22nd February to `/restored/etc` on Merlo.

== Improvements to rdiff-manager ==

`rdiff-manager` is pretty simple but there is plenty of room for improvement. Check the TODO file in the distribution for ideas.

----
CategorySystemAdministration