High availability for MySQL on Amazon EC2 – Part 4

This post is the fourth of a series that started here.

From the previous of this series, we now have resources configured but instead of starting MySQL, Pacemaker invokes a script to start (or restart) the EC2 instance running MySQL. This blog post describes the instance restart script. Remember, I am more a DBA than a script writer so it might not be written in the most optimal way.

First, let's recap what's the script has to perform (the full script is given below).

Kill the MySQL EC2 instance if running
Make sure the MySQL EC2 instance is stopped
Prepare the user-data script for the new MySQL EC2 instance
Launch the new MySQL instance
Make sure it is running
Reconfigure local heartbeat
Broadcast the new MySQL instance IP to the application servers

Kill the MySQL EC2 instance

In order to kill the existing MySQL EC2 instance, we first have to identify it. This is done by:

PLAIN TEXT

CODE:

OLD_INSTANCE_ID=`ec2-describe-instances -K $PK -C $CERT | /usr/local/bin/filtre_instances.pl | grep $AMI_HA_MYSQL | egrep "running|pending" | tail -n 1 | cut -d'|' -f3`

by filtering on the AMI type of the instance. Since an instance can be listed at the "stopped" state, it is mandatory to filter for states "running" or "pending". Then the instance is terminated with:

PLAIN TEXT

CODE:

ec2-terminate-instances -K $PK -C $CERT $OLD_INSTANCE_ID> /dev/null

Make sure the MySQL EC2 instance is stopped

Terminating an EC2 instance is not instantaneous, we can confirm an instance is really stopped by monitoring its status and wait until it is actually "terminated". The code below is how the script performs this task.

PLAIN TEXT

CODE:

#wait until the old instance is terminated it takes a few seconds to stop
done="false"
while [ $done == "false" ]
do
status=`ec2-describe-instances -K $PK -C $CERT $OLD_INSTANCE_ID | /usr/local/bin/filtre_instances.pl | grep -c terminated`
if [ "$status" -eq "1" ]; then
done="true"
else
ec2-terminate-instances -K $PK -C $CERT $OLD_INSTANCE_ID> /dev/null
sleep 5
fi
done

Prepare the user-data script for the new MySQL EC2 instance

The new MySQL instance will be running heartbeat. Since we cannot use neither Ethernet broadcast or multicast, we need to configure the new instance so that it communicates through unicast with its partner node in the cluster, the node on which the restart script is run. This configuration is achieved by providing a user-data script (see the hamysql.user-data below) which completes the heartbeat configuration of the new instance. The hamysql.user-data script only performs a search and replace operation on the /etc/ha.d/ha.cf file and then restart the heartbeat service. In order for this to work properly, we just have to put the IP of the current instance in the script like here:

PLAIN TEXT

CODE:

OUR_IP=`/sbin/ifconfig eth0 | grep 'inet addr' | cut -d':' -f2 | cut -d' ' -f1`
#Now, modify the user-data script, we need to put our IP address in
if [ "$OUR_IP" == "" ]
then
echo "Error getting Our IP"
else
perl -pi -e "s/ucast eth0 (\d+)(\.\d+){3}/ucast eth0 $OUR_IP/g" $USER_DATA_SCRIPT
fi

Launch the new MySQL instance

Once things are ready, a new MySQL instance can be launched with:

PLAIN TEXT

CODE:

#Now we are ready to start a new one
INSTANCE_INFO=`ec2-run-instances -K $PK -C $CERT $AMI_HA_MYSQL -n 1 -g $HA_SECURITY_GROUP -f $USER_DATA_SCRIPT -t $INSTANCE_TYPE -z $INSTANCE_ZONE -k $INSTANCE_KEY | /usr/local/bin/filtre_instances.pl`
#wait until the new instance is running it take a few seconds to start
NEW_INSTANCE_ID=`echo $INSTANCE_INFO | cut -d'|' -f3`

Out of this operation, we retrieve the new instance "instance_id".

Make sure it is running

Since we know the "instance_id" of the new instance, checking if it is running is easy:

PLAIN TEXT

CODE:

done="false"
while [ $done == "false" ]
do
INSTANCE_INFO=`ec2-describe-instances -K $PK -C $CERT $NEW_INSTANCE_ID | /usr/local/bin/filtre_instances.pl`
status=`echo $INSTANCE_INFO | grep -c running`
if [ "$status" -eq "1" ]; then
done="true"
else
sleep 5
fi
done

Reconfigure local heartbeat

Now, Heartbeat, on the monitoring host, must be informed of the IP address of its new partner. In order to achieve this, a search and replace operation in the local ha.cf file followed of restart of Heartbeat is sufficient.

PLAIN TEXT

CODE:

#Set the IP in /etc/ha.d/ha.cf and ask heartbeat to reload its config
MYSQL_IP=`ec2-describe-instances -K $PK -C $CERT $NEW_INSTANCE_ID | /usr/local/bin/filtre_instances.pl | cut -d'|' -f2`
perl -pi -e "s/ucast eth0 (\d+)(\.\d+){3}/ucast eth0 $MYSQL_IP/g" /etc/ha.d/ha.cf
/etc/init.d/heartbeat reload

Broadcast the new MySQL instance IP to the application servers

The final phase is to inform the application servers that the IP of the MySQL has changed. The best way to list those application servers is through a security group and, provided the appropriate ssh keys have been exchanged, this code will push the IP update.

PLAIN TEXT

CODE:

TMPFILE=`mktemp`
ec2-describe-instances -K $PK -C $CERT | /usr/local/bin/filtre_instances.pl | grep $CLIENT_SECURITY_GROUP> $TMPFILE
while read line
do
IP=`echo $line | cut -d'|' -f2`
ssh -i /usr/local/bin/update_mysql ubuntu@$IP sudo ./updated_xinetd.sh $MYSQL_IP
done <$TMPFILE
rm $TMPFILE

The full script:

PLAIN TEXT

CODE:

#!/bin/bash
HA_SECURITY_GROUP=testyves
CLIENT_SECURITY_GROUP=hamysql-client
CLIENT_SCRIPT=/usr/local/bin/update_client.sh
AMI_HA_MYSQL=ami-84a74fed
EBS_DATA_VOL=vol-aefawf
USER_DATA_SCRIPT=/usr/local/bin/hamysql.user-data
PK=/usr/local/bin/pk-FNMBRRABFRKVICBDZ4IOOSF7YROYZRZW.pem
CERT=/usr/local/bin/cert-FNMBRRABFRKVICBDZ4IOOSF7YROYZRZW.pem
INSTANCE_TYPE=m1.small
INSTANCE_ZONE=us-east-1c
INSTANCE_KEY=yves-key
OUR_IP=`/sbin/ifconfig eth0 | grep 'inet addr' | cut -d':' -f2 | cut -d' ' -f1`
#Now, modify the user-data script, we need to put our IP address in
if [ "$OUR_IP" == "" ]
then
echo "Error getting Our IP"
else
perl -pi -e "s/ucast eth0 (\d+)(\.\d+){3}/ucast eth0 $OUR_IP/g" $USER_DATA_SCRIPT
fi
while [ 1 ]; do
#First thing to do, terminate the other instance ID
OLD_INSTANCE_ID=`ec2-describe-instances -K $PK -C $CERT | /usr/local/bin/filtre_instances.pl | grep $AMI_HA_MYSQL | egrep "running|pending" | tail -n 1 | cut -d'|' -f3`
if [ "$OLD_INSTANCE_ID" == "" ]
then
#no running instance
:
else
ec2-terminate-instances -K $PK -C $CERT $OLD_INSTANCE_ID> /dev/null
#wait until the old instance is terminated it takes a few seconds to stop
done="false"
while [ $done == "false" ]
do
status=`ec2-describe-instances -K $PK -C $CERT $OLD_INSTANCE_ID | /usr/local/bin/filtre_instances.pl | grep -c terminated`
if [ "$status" -eq "1" ]; then
done="true"
else
ec2-terminate-instances -K $PK -C $CERT $OLD_INSTANCE_ID> /dev/null
sleep 5
fi
done
fi
#Now we are ready to start a new one
INSTANCE_INFO=`ec2-run-instances -K $PK -C $CERT $AMI_HA_MYSQL -n 1 -g $HA_SECURITY_GROUP -f $USER_DATA_SCRIPT -t $INSTANCE_TYPE -z $INSTANCE_ZONE -k $INSTANCE_KEY | /usr/local/bin/filtre_instances.pl`
#wait until the new instance is running it take a few seconds to start
NEW_INSTANCE_ID=`echo $INSTANCE_INFO | cut -d'|' -f3`
if [ "$NEW_INSTANCE_ID" == "" ]
then
echo "Error creating the new instance"
else
done="false"
while [ $done == "false" ]
do
INSTANCE_INFO=`ec2-describe-instances -K $PK -C $CERT $NEW_INSTANCE_ID | /usr/local/bin/filtre_instances.pl`
status=`echo $INSTANCE_INFO | grep -c running`
if [ "$status" -eq "1" ]; then
done="true"
else
sleep 5
fi
done
#Set the IP in /etc/ha.d/ha.cf and ask heartbeat to reload its config
MYSQL_IP=`ec2-describe-instances -K $PK -C $CERT $NEW_INSTANCE_ID | /usr/local/bin/filtre_instances.pl | cut -d'|' -f2`
perl -pi -e "s/ucast eth0 (\d+)(\.\d+){3}/ucast eth0 $MYSQL_IP/g" /etc/ha.d/ha.cf
TMPFILE=`mktemp`
ec2-describe-instances -K $PK -C $CERT | /usr/local/bin/filtre_instances.pl | grep $CLIENT_SECURITY_GROUP> $TMPFILE
while read line
do
IP=`echo $line | cut -d'|' -f2`
ssh -i /usr/local/bin/update_mysql ubuntu@$IP sudo ./updated_xinetd.sh $MYSQL_IP
done <$TMPFILE
rm $TMPFILE
/etc/init.d/heartbeat reload
fi
sleep 300 # 5 min before attempting again. Normally heartbeat should kill the script before
done

The hamysql.user-data script:

The script sets the IP of the monitor host in the heartbeat ha.cf configuration file and then, finishes up some missing configuration settings of the AMI.

PLAIN TEXT

CODE:

#!/bin/bash
sudo hostname hamysql
sudo perl -pi -e "s/ucast eth0 (\d+)(\.\d+){3}/ucast eth0 10.220.230.18/g" /etc/ha.d/ha.cf
# to eventually be added to the ebs image
sudo perl -pi -e 's/bind-address/#bind-address/g' /etc/mysql/my.cnf
sudo service mysql restart
sleep 5
/usr/bin/mysql -u root -proot -e "grant all on *.* to root@'%' identified by 'root'"
sudo /etc/init.d/heartbeat start