Tutorial: simple backup rotation scripts

I received several requests to post my backup scripts. Before I start, I think some background info is needed (i.e. what their purpose is): I've written them as part of the "Enterprise Class Backup" system we offer for webhosting accounts. Following rotation is used:

1) Full account backups:
a) on-server (dedicated hard drive used only for storing backups)
- daily backups, 7 days rotation
- weekly backups, 4 weeks rotation
- the standard daily/weekly/monthly cPanel backups are also available on this drive (1 day/1 week/1 month rotation)
b) off-server (another server in the same datacenter)
- weekly backups, 12 weeks rotation
c) off-site (stored on DVDs in a Swiss bank vault)
- monthly backups, 24 months rotation

2) MySQL database backups:
a) on-server (dedicated hard drive used only for storing backups)
- hourly backups, 24 hours rotation
b) off-server (another server in the same datacenter)
- daily backups, 7 days rotation
c) off-site
- weekly backups, stored on the monthly DVDs

Step by step guide

First of all, I'll describe the environment that's needed to run these scripts 1:1. This is a cPanel server running GNU/Linux, and while cPanel is not neccessary (you could use any other backups instead of the daily cPanel account backups, just modify the scripts), I can't say on what non-GNU/Linux OSes it would work without changes. Since we're talking about simple shell commands, they might work on other *Nix OSes like *BSD or Solaris, but I haven't tested it. You're welcome to post your findings.

a) on-server backups

You'll need a certain directory structure so the scripts work without modifications. The backup hard drive is mounted as /backup and has the following directories:

/backup/cpbackup : here are the cPanel backups. The three subdirs will be created automatically by cPanel/WHM.

/backup/rotation: this is where my scripts write their output. You need to create following subdirs: hourly, daily, weekly, monthly. You don't need to create any directories under these subdirs, the scripts will take care of that.

Let's start with the hourly rotation for MySQL backups. These aren't compressed because their primary purpose is if a client suddenly calls you in tears saying they just destroyed/corrupted/whatever their big and important DB 5 minutes ago, you can just copy back their files from the backup location to the /var/lib/mysql subdir and they'll be happy. This is my script in /etc/cron.hourly (file permissions 7xx):

code:

#!/bin/sh
rm -rf /backup/rotation/hourly/$(date +"%H")
mkdir /backup/rotation/hourly/$(date +"%H")
cp -R /var/lib/mysql/* /backup/rotation/hourly/$(date +"%H")

The daily rotation runs a couple of hours after the cPanel nightly backup and copies the latest account backups into the directory that corresponds to the current day and keeps them for 7 days before overwriting them:

code:

#!/bin/sh
rm -rf /backup/rotation/daily/$(date +"%u")
mkdir /backup/rotation/daily/$(date +"%u")
cp -R /backup/cpbackup/daily/* /backup/rotation/daily/$(date +"%u")

It's in /etc/cron.daily of course and needs to be owner executable (7xx).

The weekly rotation script does the same. Note that this one doesn't have an automatic delete mechanism (well a 52-week one), because I prefer to monitor available disc space and as long as there are sufficient reserves I keep the files, longer than advertised (after all, advertising and SLA is in my eyes the minimum, more is always OK). The script is located in /etc/cron.weekly and needs to be chmoded to 7xx.

code:

#!/bin/sh
rm -rf /backup/rotation/weekly/$(date +"%V")
mkdir /backup/rotation/weekly/$(date +"%V")
cp -R /backup/cpbackup/daily/* /backup/rotation/weekly/$(date +"%V")

The monthly script adds some new twists for still another layer of security: there have been reports again and again that cPanel/WHM corrupted a certain account's backup and all backup files were useless. In this case, the following (mind you, storage space wasting) addition makes sure that in this worst case you still have the majority of data for the customer. While it needs you to setup the account from scratch and copy lots of things into it and still do some of the steps manually (like setting up the MySQL DBs and users in cPanel), at least all website data, MySQL DBs, e-mails stored on the server, webstat outputs etc is still available. The script goes into /etc/cron.monthly and has a 7xx permission.

code:

#!/bin/sh
rm -rf /backup/rotation/monthly/$(date +"%m")
mkdir /backup/rotation/monthly/$(date +"%m")
cp -R /backup/cpbackup/daily /backup/rotation/monthly/$(date +"%m")
cp -R /var/lib/mysql /backup/rotation/monthly/$(date +"%m")
cp -R /home /backup/rotation/monthly/$(date +"%m")

b) off-server backups

These are stored on an external backup server to which I have only FTP access (thus no fancy stuff). The first is the MySQL daily backup, located in /etc/cron.daily (7xx as always):

code:

#!/bin/sh
cd /backup/rotation
tar -zcf mysql_$(date +"%F").tar.gz /var/lib/mysql/*
ftp -in <open 127.0.0.1
user yourusername yourpassword
bin
hash
prompt
put mysql_$(date +"%F").tar.gz
bye
rm -f mysql_$(date +"%F").tar.gz

You need to modify three variables: IP address (or hostname) where you connect to, username and password for the FTP access. I have to admit that the last line doesn't work (not yet sure why), the files are put but then they aren't deleted from the local place. Since these are relatively small and I've got ample storage space, I didn't look into this further (yet).

Next is the weekly FTP backup. This is a rather crude job, taking lots of CPU horsepower if you've got lots of accounts. It creates a giant .tar.gz of everything in the daily cPanel backup dir and pushes this on to the FTP backup server. Since by now we're speaking about disaster recovery backups (which you should not need in a normal case), I prefer having them in one big file instead of many small ones (if you prefer hundreds of small files every week in your FTP space, you can always use the mput command):

code:

#!/bin/sh
cd /backup/rotation
tar -zcf cpbackup_$(date +"%F").tar.gz /backup/cpbackup/daily/*
ftp -in <open 127.0.0.1
user yourusername yourpassword
bin
hash
prompt
put cpbackup_$(date +"%F").tar.gz
bye
rm -f cpbackup_$(date +"%F").tar.gz

Same comments apply as to the daily job.

There is no monthly FTP job because in its place is the monthly off-site backup job.

c) off-site backups

The off-site part is done manually, I first copy the contents of /backup/cpbackup/daily to a directory with FTP access, then use an FTP client to download the files to a local machine and shovel them on a DVD. For the weekly MySQL part, I prefer to use the most recent state, so I create a .tar.gz directly into an FTP-accessible directory and download it.

Notes

* You shouldn't even try to set up this system on an overloaded or low-end server.
* You cannot sell even remotely all the space on your primary hard drive (unless your backup drive is several times larger).
* I am aware that doing full backups every time is the most space-wasting kind of backup, but I prefer it over more sophisticated methods because with a differential or incremental backup system once you lose the starting point, the rest is so much dead bytes. With full backups, any single one stands on its own.
* Offering such a sophisticated solution on a per client basis (like a paid addon) needs only small modifications: all you need is to specify the account backup files and the subdirs in /var/lib/mysql that are to be part of the process. You could e.g. solve it by creating a dir where you first copy over the needed files from the original places, and then run the backup scripts on that directory.