Traditionally, the UCC's backup system has left much to be desired, such as "existing" or "running on reliable hardware", let alone living up to the [[https://web.archive.org/web/20150919143600/https://www.lawtechnologytoday.org/2014/12/backups-rule-three/|Rule of Three]]. Backups at present run on [[Mollitz]], which is housed in another location (off-site backups!). Contact if you need to know where it is living. Backups are run using [[http://rdiff-backup.nongnu.org/|rdiff-backup]], a disk-based incremental backup system. These are managed by [[https://gitlab.ucc.asn.au/UCC/rdiff-manager|rdiff-manager]], a Python wrapper written by DavidAdam. == Adding new machines == On the target system (the machine you want to back up): * Make sure the UCC backup key is installed (e.g. with authroot) * Install `rdiff-backup` packages. On the backup server (`mollitz`): * Copy `/backups/conf/example-copy-me` to `/backups/conf/.conf` * Edit `/backups/conf/.conf` as required - the syntax is documented in [[http://manpages.ubuntu.com/manpages/bionic/en/man1/rdiff-backup.1.html|rdiff-backup(1)]] under FILE SELECTION * Add the SSH host key using `su -c 'ssh-keyscan HOSTNAME >> ~/.ssh/known_hosts' backups` Now wait until the nightly backup run. The output is: * sent by email to hostmasters * successful backups leave a log in `/backups//rdiff-backup-data/backup.log` and `/backups//rdiff-backup-data/session_statistics..data` * partially successful backups leave a log in `/backups//rdiff-backup-data/error_log..data.gz` * /!\ In some cases, e.g. if a particular file constantly changes during each and every backup run, a successful backup or update may never be possible * TODO: this could be mitigated by backing up a stable snapshot, instead of the live filesystem == Checking backup status == To list all the backups available for a particular host, or to see when it was last successful, on the backup server (Mollitz) run: * `rdiff-backup --list-increments /backups/` To list how much data is taken for each incremental backup (which is much slower), on the backup server (Mollitz) run: * `rdiff-backup --list-increment-sizes /backups/` == Restoring a backup == To restore files from backup, on the backup server (Mollitz): * Run `rdiff-backup --list-increments /backups/` and choose a backup to restore from * Copy the timestamp from the increment list - `2022-02-22T02:00:03+08:00` * Decide where you are going to restore the files - locally (i.e. to Mollitz), where you can inspect them, or back to the remote host === Restoring files locally === * Run `mkdir /backups/tmp/` * Run `rdiff-backup --restore /backups//path/to/file-or-directory/you/want/to/restore /backups/tmp/` For example, using `rdiff-backup -r 2022-02-22T02:00:03+08:00 /backups/merlo/etc /backups/tmp/merlo` will restore the contents of merlo's `/etc/` directory as of 22nd February to `/backups/tmp/merlo`. === Restoring files remotely === /!\ Be careful - this is easy to mess up, particularly if you are trying to restore to the original path. Note the double-colons in the restore path! * Run `rdiff-backup --restore /backups//path/to/file-or-directory/you/want/to/restore root@::/path/to/file-or-directory/you/want/to/restore` For example, `rdiff-backup -r 2022-02-22T02:00:03+08:00 /backups/merlo/etc merlo::/restored/etc` will restore the contents of merlo's `/etc/` directory as of 22nd February to `/restored/etc` on Merlo. == Improvements to rdiff-manager == `rdiff-manager` is pretty simple but there is plenty of room for improvement. Check the TODO file in the distribution for ideas. ---- CategorySystemAdministration == Latest update on the backup system == [ROY] 20240930, 20241111 This section is about a temporary solution for `rdiff-backup`'s version discrepancy issue which causes some machines can't be backup correctly. === molmol space dataset === It's a cronjob on `molmol` calling `/root/zfs-send-script.sh` everyday at 02:00, which take incremental snapshot on Space dataset first, then `zfs send` to `dell-ph1` over ssh. Similar cronjob applies to `dell-ph1`->`dell-ph2`, 05:00 every Saturday. Sample script could be found under wheel/docs. === good'ol rdiff-backup style backup === Also a cronjob, on `dell-ph1`, calling all .sh scripts under `/etc/rsync-conf/` everyday 03:00; then take `zfs snapshot` (via `/etc/rsync-conf/zfs-snapshot`, on the whole `dpool` zfs pool) everyday 04:00 to create diffirential entries. * The include/exclude files are the .conf files under `/etc/rsync-conf/` * To add a new host check `/etc/rsync-conf/setuphost`, and add .conf & .sh file for this host * rsync script is adopted untouched from [[https://www.ucc.gu.uwa.edu.au/~dagobah/things/secure-backups.html]], stored on target host === data retention === Snapshots older than 6mo will be destroyed on all 3 machines, by cronjobs. === TODO === * ZFS-send all dataset: `dell-ph1`=>`dell-ph2` === PVE VMs === All machines are backup to `wobbegong` via `PBS`, scheduled every Saturday 00:00.