It has been 4 weeks since I last posted the goings-on for Mozilla DBs. April is always a crazy month because of the annual MySQL conference (Some great pics here). This year it was the Percona Live: MySQL Conference and Expo. And of course as soon as I get caught up from the conference, I have to submit more sessions to MySQL Connect (call for papers closes Sunday May 6th) and Percona Live: NYC (anyone know when the call for papers for this will close?).
At the conference, I gave a lightning talk and a tutorial. I have posted the slides for those interested. Unfortunately, Percona asked me not to post the recordings of tutorials. I had cleared recording with them, but apparently during the last tutorial session one of the 2 video cameras was turned off and I was informed after the fact that I was not supposed to be recording them. Which is odd, since
O’Reilly had no problems with me recording tutorials in 2008 (memcached tutorial), 2009 (part 1 and part 2 of a metadata tutorial) and 2010 (part 1>/A> and part 2 of a tutorial about config options). I have no videos from 2011 since I did not attend the conference.
At any rate, this is just to let you know that due to Percona’s policy, there will not be any tutorial recordings available (should they decide to change their policy, the tapes have not been recorded over yet). The session recordings are forthcoming, I will blog here when they are ready for viewing.
We have also been busy with data center moves! Due to MySQL’s flexibile architecure, we were able to move db services with very little interruption to the end user. This is a big deal because we have to change monitoring checks and network flows as well as MySQL ACLs (database authorization – Mozilla is very good about only allowing specific hosts or groups of hosts to connect to MySQL!)
In the last 4 weeks, we have done many moves:
- We not only moved the database cluster that buildbot was on, but we also built a cluster specifically for buildbot, with no other services on it. Before, it shared a database with other services such as graphs, cruncher and autoland.
- Upgraded a production slave of Bugzilla to MySQL 5.1 and put it in the mix. It has been running for a few weeks without any problems, so I will be upgrading the other slaves as soon as I have time.
- Helped debug why https://reps.mozilla.org/people/ was slow. It was not a database issue. (I have to say, after doing 5 years of consulting, where problems are usually the database, it’s nice to work in a shop where the problem usually is NOT the database!)
- Started to turn an old mail server, which had lots of space, into a backup server in our Phoenix data center.
- Added some metadata for http://graphs.mozilla.org.
- Moved our PHPMyAdmin server. It moved, and nobody noticed, so either I did a good job, or nobody’s really using it!
- Added new database grants so our metrics team could access new customBugzilla fields.
- Moved our metrics databases.
- Moved the web development databases.
- Move our internal inventory database. This is tricky, because we relied heavily on our inventory database during our move, so we had to be extra careful that we did not cause any problems with the move…while we were moving this server! Of course that was not a huge issue, as we had a database in a third data center that took all the traffic and became the active database cluster as we moved the original cluster (from San Jose to Santa Clara).
- There was plenty of PostgreSQL work to do as well. I learned how to refresh our stage database from production and also got a lot more practice in reprocessing crash reports for our Crash Stats database.
- Did a test-run of moving the database behind a Mozilla wiki, and mentored the Web Operations team on how to do it. They did it successfully, which meant I did not have to be up in the wee hours of the morning!
- Got a sanitized copy of the Bugzilla database to some developers, a few times.
- Created the Mozilla Labs database cluster.
- Created the Bedrock database cluster – on Percona Server 5.5! I am very excited to be using a 5.5 version on new projects. One of the goals I am making slow progress on is upgrading servers from MySQL 5.0.
- Created the new developer database cluster.
- Created the Personas database cluster.
- Decommissioned old http://support.mozilla.com database servers.
- Decommissioned old http://addons.mozilla.com database servers.
- Decommissioned old Bugzilla staging database servers.
- Created a new Bugzilla staging database cluster.
- Debugged a Hive problem – For reference, the problem was getting this error:
ERROR metadata.Hive: javax.jdo.JDOException: Couldnt obtain a new sequence (unique id) : Binary logging not possible. Message: Transaction level 'READ-COMMITTED' in InnoDB is not safe for binlog mode 'STATEMENT'
The problem is that Hive automatically sets the transaction isolation level to be READ_COMMITTED by default. The solution is
to change Hive’s configuration settings to set the transaction isolation level to REPEATABLE_READ, to match MySQL’s default.
It has been a crazy few weeks, and will only get crazier as I will be traveling more and more!
PlanetMySQL Voting: Vote UP / Vote DOWN