Back up your server

Your server's disk is one copy of everything. A mistyped rm, an upgrade gone wrong, a break-in or a cancelled server can each take it away. A backup only counts if it lives on another machine and you've restored from it at least once.

The panel's snapshots are good for undoing a change, but they sit on the same machine as the server and are deleted with it, so they aren't backups.

We'll copy backups to a second machine you control, called backup.example.com here. It can be another server, a box at home, or a storage service that accepts SSH.

What to include#

Usually:

• /etc, your configuration;
• /home and /root;
• /var/www, or wherever your sites and apps live;
• /srv, /opt, and Docker volumes under /var/lib/docker/volumes if you use them;
• database dumps. Not the raw files of a running database.

You can leave out /usr, /var/cache and /tmp. A reinstall brings the system back; it's your data you can't replace.

Dump the databases first#

Copying a live database's files can give you a copy that won't open. Write a dump before each backup instead. For MySQL or MariaDB:

install -d -m 700 /var/backups/db
mysqldump --all-databases --single-transaction --routines | gzip > /var/backups/db/mysql.sql.gz

For PostgreSQL:

sudo -u postgres pg_dumpall | gzip > /var/backups/db/postgres.sql.gz

Then make sure /var/backups/db is in the backup.

The simple way: rsync#

rsync copies only what changed since last time. Give the server its own SSH key for backups and put the public half on the backup machine:

ssh-keygen -t ed25519 -N '' -f /root/.ssh/backup -C backup@web-1
ssh-copy-id -i /root/.ssh/backup.pub [email protected]

Then:

rsync -a --delete -e 'ssh -i /root/.ssh/backup' /etc /home /var/www /var/backups/db [email protected]:web-1/

That gives you a mirror. Its weakness is that it mirrors mistakes too: if a file gets deleted or encrypted on the server, the next run faithfully copies the damage across. If that worries you, and it should a bit, use Borg.

The better way: BorgBackup#

Borg keeps compressed, encrypted, deduplicated snapshots, so you can go back to last Tuesday. Install it on both machines with apt install -y borgbackup. Then, on the server:

export BORG_RSH='ssh -i /root/.ssh/backup'
borg init --encryption=repokey-blake2 [email protected]:web-1.borg
borg key export [email protected]:web-1.borg /root/borg-key.txt

borg init asks you for a passphrase. Put that passphrase and the exported key file somewhere that isn't this server, like your password manager. Lose both and the backup is unreadable, by you included.

Make a backup and trim the old ones:

borg create --stats --compression zstd [email protected]:web-1.borg::'{hostname}-{now}' /etc /home /root /var/www /var/backups/db
borg prune --keep-daily 7 --keep-weekly 4 --keep-monthly 6 [email protected]:web-1.borg
borg compact [email protected]:web-1.borg

That keeps a week of dailies, a month of weeklies and six monthlies.

Run it every night#

Put your commands in /usr/local/sbin/backup, starting with #!/bin/sh and set -e. For Borg, add the export BORG_RSH=... line from above, and export BORG_PASSCOMMAND='cat /root/.borg-passphrase' with that file set to mode 600. Make the script executable with chmod 700, then add this with crontab -e as root:

30 3 * * * /usr/local/sbin/backup >> /var/log/backup.log 2>&1

Look at the log the next morning. Then look again now and then. Backups have a way of failing quietly.

Prove you can restore#

Restore into a scratch folder, never over live files:

mkdir /tmp/restore-test && cd /tmp/restore-test
borg list [email protected]:web-1.borg
borg extract [email protected]:web-1.borg::ARCHIVE-NAME etc/hostname
cat etc/hostname

Use an archive name from borg list in place of ARCHIVE-NAME. With rsync, copy a file back the other way. And load a database dump into a test database at least once.