How to avoid two backups running at the same time

When your backup script is running for too long it sometimes causes the second backup script starting at the time when previous backup is still running. This increasing pressure on the database, makes server slower, could start chain of backup processes and in some cases may break backup integrity.

Simplest solution is to avoid this undesired situation by adding locking to your backup script and prevent script to start second time when it’s already running.

Here is working sample. You will need to replace “sleep 10″ string with actual backup script call:

#!/bin/bash

LOCK_NAME="/tmp/my.lock"
if [[ -e $LOCK_NAME ]] ; then
        echo "re-entry, exiting"
        exit 1
fi

### Placing lock file
touch $LOCK_NAME
echo -n "Started..."

### Performing required work
sleep 10

### Removing lock
rm -f $LOCK_NAME

echo "Done."

It works perfectly most of the times. Problem is that you could still theoretically run two scripts at the same time so both will pass lock file checks and will be running together. To avoid that you would need to place unique lock file just before check and make sure no other processes did the same.

Here is improved version:

#!/bin/bash

UNIQSTR=$$
LOCK_PREFIX="/tmp/my.lock."
LOCK_NAME="$LOCK_PREFIX$UNIQSTR"

### Placing lock file
touch $LOCK_NAME
if [[ -e $LOCK_NAME && `ls -la $LOCK_PREFIX* | wc -l` == 1 ]] ; then
        echo -n "Started..."
        ### Performing required work
        sleep 10
        ### Removing lock
        rm -f $LOCK_NAME
        echo "Done."
else

### another process is running, removing lock
        echo "re-entry, exiting"
        rm -f $LOCK_NAME
        exit 1
fi

Now even if you managed to run two scripts at the same time only one script could actually start backup. In very rare situation both scripts will refuse to start (because of two lock files existing at the same time) but you could catch this issue by simply monitoring script exit code. Anyway – as soon you receive backup exit code different than zero it’s time to review your backup structure and make sure it works as desired.

Please note – when you terminate this script manually you will also need to remove lock file as well so script will pass check on startup. You could also use this script for any periodic tasks you have like Sphinx indexing, merging or index consistency checking.

For your convenience this script is available for download directly or using wget:

wget http://astellar.com/downloads/backup-wrapper.sh

You could also find more about MySQL backup solutions here.

Keep your data safe and have a nice day!