After having my fair share of backup dilemmas and stress Tim Bray's post at least gives hope that I am not alone. It is strange that there really are not that many decent backup products considering how simple the task really is.
The most effective backup system I have used is the one I wrote myself for Linux servers. It is simple, flexible, non-proprietary and most importantly it seems to work without hassle. It comprises of a bash script and some configuration files.
The bash script compresses gzipped tar archives onto a removable hard drive. In most cases I use two removeable drives for redundancy. Typically all user data is stored in the /home directory with a few special directories: media (music/movies), temp and data shared amoungst the users. The /home/data directory holds most of the office data and as such the backup script breaks archiving of this data into smaller pieces in order to facilitate easier restoration.
The backup script stores multiple backups on disk to ensure that a corrupted backup does not lead to significant loss of data. The backup usually runs every night with the removable disks swapped during the day. On completion (or failure) of a backup an automated email is sent out to interested parties.
Below is the bash script (typically /usr/local/sbin/backup_script):
#!/bin/bash
# configure variables for script
servername=`awk '/^servername/{print $2}' $1`
serveremail=`awk '/^serveremail/{print $2}' $1`
destdir=`awk '/^backupdir/{print $2}' $1`
movedir=`awk '/^movedir/{print $2}' $1`
lockfile=`awk '/^lockfile/{print $2}' $1`
endofweek=`awk '/^endofweek/{print $2}' $1`
eowflag=`awk '/^eowflag/{print $2}' $1`
device=`awk '/^device/{print $2}' $1`
volume=`awk '/^volume/{print $2}' $1`
email=`awk '/^email/{print $2}' $1`
dbhost=`awk '/^dbhost/{print $2}' $1`
dbuser=`awk '/^dbuser/{print $2}' $1`
dbpassword=`awk '/^dbpassword/{print $2}' $1`
processexists=`awk '/^processexists/{print $2}' $1`
mountfailure=`awk '/^mountfailure/{print $2}' $1`
if [ -e $lockfile ]
then
echo "Backup task already operating, please try again later"
mail -s "Backup of $servername Failed: Process already running" -r $serveremail $email < $processexists
exit
else
# See if device is plugged in and available
umount $device
mount $device $mount
if [[ $? -eq 0 ]] ;
then
echo "Media available and ready to go..."
else
echo "Backup task could not continue, media unmountable"
mail -s "Backup of $servername Failed: Media Unavailable" -r $serveremail $email < $mountfailure
exit
fi
# Perform backup
/bin/touch $lockfile
echo "Performing data backup"
# Change directory to a safe place just in case anything bad happens
cd /tmp
if [ endofweek != "true" ]
then
# Delete end of week flag if it exists
/bin/rm $eowflag
/bin/rm -fr $volume$movedir/*
/bin/mv $volume$destdir/* $volume$movedir/
echo "Beginning daily backup: " `date` > $volume$destdir/backup.log
else
# Does the end of week flag exist? If it does do not move files
if [ -e $eowflag ]
then
# Flag exists - weekly backup already made, just delete existing files
/bin/rm -fr $volume$destdir/*
echo "Beginning daily backup: " `date` > $volume$destdir/backup.log
else
# Move files
/bin/rm -fr $volume$movedir/*
/bin/mv $volume$destdir/* $volume$movedir/
/bin/touch $eowflag
echo "Beginning weekly backup: " `date` > $volume$destdir/backup.log
fi
fi
# note time
echo "Backup started: " `date` > $volume$destdir/backup.log
echo "-------------------------------------------------------" >> $volume$destdir/backup.log
# backup project data
cd $volume$destdir
mkdir data
# Change to data directory for file listing
cd /home/data
for datadir in *
do
echo `date` ": Backing up data folder $datadir..." >> $volume$destdir/backup.log
tar czf "$volume$destdir/data/$datadir.tar.gz" "$datadir"
done
# Jump back into backup directory
cd $volume$destdir
# backup home directories
for homedir in `ls /home`
do
if [ $homedir != ".cpan" ]
then
if [ $homedir != "backup" ]
then
if [ $homedir != "temp" ]
then
if [ $homedir != "media" ]
then
if [ $homedir != "data" ]
then
echo `date` ": Backing up home directory of $homedir..." >> $volume$destdir/backup.log
tar zcf $volume$destdir/home_$homedir.tar.gz /home/$homedir
fi
fi
fi
fi
fi
done
# Backup Apache Files
echo `date` ": Backing up Web Server Files" >> $volume$destdir/backup.log
tar zcf $volume$destdir/server.tar.gz /srv
# backup databases
for dbname in `echo 'show databases;' | /usr/bin/mysql -h $dbhost -u$dbuser -p$dbpassword`
do
if [ $dbname != "Database" ];
then
echo `date` ": Backing up database $dbname..." >> $destdir/backup.log
/usr/bin/mysqldump --opt -h $dbhost -u$dbuser -p$dbpassword $dbname > $destdir/$dbname.sql
fi
done
# Now compress backed up databases into archive
tar cf - $destdir/*.sql | gzip > $destdir/db_backup.tar.gz
# Remove SQL
rm $destdir/*.sql
# Backup LDAP Data
echo `date` ": Backing up LDAP data" >> $volume$destdir/backup.log
ldapsearch -x > $volume$destdir/ldap.ldif
# Backup Configuration Files
echo `date` ": Backing up configuration files" >> $volume$destdir/backup.log
tar zcf $volume$destdir/server_configs.tar.gz /etc
# Backup completed
echo "-------------------------------------------------------" >> $volume$destdir/backup.log
echo "Backup completed: " `date` >> $volume$destdir/backup.log
echo "Backup size: " `du -smh $volume$destdir` >> $volume$destdir/backup.log
echo "-------------------------------------------------------" >> $volume$destdir/backup.log
echo df $device -h
echo "Backup process completed"
rm $lockfile
mail -s "Backup of $servername Completed" -r $serveremail $email < $volume$destdir/backup.log
fi
umount $device
exit
The backup script pulls data from a few configuration files, the main file is normally named backup-daily.conf and stored in /etc/backup
servername SomeServer
serveremail email@someserver.com
backupdir /backupA
movedir /backupB
lockfile /var/run/backup_script.pid
endofweek false
eowflag /var/run/backup_eow
device /dev/sda1
volume /media/backup_daily
dbhost localhost
dbuser user
dbpassword password
email notifyme@someplace.com
processexists /etc/backup/processexists
mountfailure /etc/backup/mountfailure
Here is a definition of the configuration options:
servername - name of server
serveremail - From email address for server emails
backupdir - directory on backup drive to backup to
movedir - directory on backup drive to move previous backup to
lockfile - name of lock file to create whilst backup is taking place
endofweek - true/false whether this is an end of week backup
eowflag - name of flag to set if this is an end of week backup
device - removable media device address
volume - location removable media is mounted to
email - addresses to send backup notification email to
processexists - message to email if the process is still running and a new process cannot begin
mountfailure - message to send if there was a problem mounting the backup drive
The backup can then be run using the following command:
/usr/local/sbin/backup_script /etc/backup/backup-daily.conf
Multiple backup configurations can be created and executed just by substituing /etc/backup/backup-daily.conf with the location of the desired configuration file.