Rsync

Notes in progress.

See also Backup procedure

For backups

Initiation (assuming use of the standard backup or replication scripts):

In the root of your mounted storage volume, create a directory for the backup target's data:

mkdir mytarget

Then make an initial copy run

# rsync --old-compress -rlptD --numeric-ids --hard-links --inplace --acls --xattrs --rsync-path "/usr/bin/sudo /usr/bin/rsync" --link-dest /path/to/dest/latest --old-compress <backupuser>@<serverbeingbackedup>:/path/to/foldertobackup/current/ /path/to/dest/inprogress &

If your initial copy is while both target and backup storage are on the same network, you may get greater tranfer speeds without the "compress" statement, and if both are relatively current installs, you can just use --compress rather than --old-compress, which is only needed if the live server has an old version of rsync.

If you are moving an existing multi-day backup set, you can use

# rsync -aHP sourceserver:/path/* /pathto/mytarget

which aims to preserve time/date stamps, permissions and hardlinks.

NB with a large backup set you may run into load issues, specifically high wait, which is related to updatedb.mlocate - this is a daily cronjob building an index of filenames. It gets into trouble trying to index all the files (including hardlinks) that are being copied, significantly lowering the copy speed. The process can be safely killed and then the backup volume excluded from the index by adding the path to

/etc/updatedb.conf

in the line

PRUNEPATHS 

Prune script

This is a script that deletes old entries of rsync backup.

#!/bin/bash

dir=/srv/rsync-target/

threemonthago=`date -d '-3 month' +"%y-%m-%d"`
year=`date -d $threemonthago +%y`
month=`date -d $threemonthago +%m`
find $dir -name ${year}-${month}-0* -delete

fivemonthago=`date -d '-5 month' +"%y-%m-%d"`
year=`date -d $fivemonthago +%y`
month=`date -d $fivemonthago +%m`
find $dir -name ${year}-${month}-2* -delete

oneyearago=`date -d '-1 year' +"%y-%m-%d"`
year=`date -d $oneyearago +%y`
month=`date -d $oneyearago +%m`
if [ $month -eq 02 ]; then
	find $dir -name ${year}-${month}-1* -not -path ${dir}*${year}-${month}-19* -delete
else
	find $dir -name ${year}-${month}-1* -delete
	find $dir -name ${year}-${month}-31* -delete
fi

This crontab command can be used for setting the script to run first sunday of every month:

00 01 * * 0 [ $(date +\%d) -le 07 ] && /path/to/script

Tidying up

Script for tidying up after an outage causes previous backup to fail (start but not complete, leaving the inprogress directory)

NB only use this script if the previous backup failed to complete, but mostly finished, else the next run will copy lots of new data. If the backup failed early in its normal runtime (which you can get an idea of from the log entries, or by using ls -l to see which directories have the right timestamp (as opposed to the timestamp of the failed backup run). If it failed fairly early, it's best to just delete the "inprogress" directory, and if relevant resolve whatever issue caused the backup to fail.

if [ $# -ne 1 ] ; then
       echo "Error: you need to provide the full path to a directory"
       exit 1
fi

#!/bin/bash
DIRDATE=`/bin/date --date="1 day ago" +"%F"`

FIRSTCHAR=${1:0:1}

echo $FIRSTCHAR

if [ ${FIRSTCHAR} = "/" ] ; then
        ROOTDIR=$1
else
        ROOTDIR=/$1
fi

if [ -d $ROOTDIR/inprogress ] ; then

DESTDIR=${ROOTDIR}/${DIRDATE}

if [ -d $DESTDIR ] ; then
        DIRDATE=`/bin/date +"%F"`
        DESTDIR=${ROOTDIR}/${DIRDATE}
fi

/bin/mv ${ROOTDIR}/inprogress ${DESTDIR}
/bin/rm ${ROOTDIR}/latest
/bin/ln -s ${DESTDIR} ${ROOTDIR}/latest


else
        echo No inprogress directory found at $ROOTDIR
fi

Deleting old Backups

The Following script deletes old backups in the form YYYY-MM-DD with the following rules:

• one week old, if not monday
• one month old, if not first monday of the month
• three months old

#!/bin/bash

if [ $# -ne 1 ] ; then
     echo "Error: you need to provide the dir as a parameter."
     echo "Syntax: ./script.sh /path/to/dir/"
     exit 1
fi

DIR=$1

#Deletes backup if it is one week old, but not monday
oneweekold=`date -d '-7 day' +"%F"`
if [[ $(date -d '-7 day' +%u) -ne 1 ]] && [ -d "$DIR$oneweekold" ] ; then
    rm -r $DIR$oneweekold
    if [ $? -ne 0 ] ; then
        echo `date +"%F %T"` " : Failed to delete one week old backup: $DIR$oneweekold" | mail -s "Tidy up script" supportedsitealerts@newro.co
    fi
fi

#Deletes backup if it is one month old, but not first monday from month
onemonthold=`date -d '-1 month' +"%F"`
day=`date -d $onemonthold +%-d`
if ! [[ $(date -d '-1 month' +%u) -eq 1 && $day -le 7  ]] && [ -d "$DIR$onemonthold" ] ; then
    rm -r $DIR$onemonthold
    if [ $? -ne 0 ] ; then
        echo `date +"%F %T"` " : Failed to delete one month old backup: $DIR$onemonthold" | mail -s "Tidy up script" supportedsitealerts@newro.co
    fi
fi

#Deletes backup if it is three months old
threemonthsold=`date -d '-3 month' +"%F"`
if [ -d "$DIR$threemonthsold" ] ; then
    rm -r $DIR$threemonthsold
    if [ $? -ne 0 ] ; then
        echo `date +"%F %T"` " : Failed to delete three months old backup: $DIR$threemonthsold" | mail -s "Tidy up script" supportedsitealerts@newro.co
    fi
fi

It also takes an argument with the directory in which to look and sends an email if delete failed