The theme this week was “data integrity and security”:
- Upgraded the kernel in our Puppet Dashboard DB master/slave cluster after one of the machines crashed.
- Verified that none of our databases were running a version of MySQL that was affected by the serious MySQL authentication bypass vulnerability discovered, patched/fixed and reported by MariaDB. (On a personal note, the way they handled finding, fixing and reporting the bug makes me happy that they are such good community stewards)
- We removed duplicate entries from our Bugzilla instance that were put in when assumptions were made about nonexistent unique keys.
- Continued the database consistency check for bugzilla.
- Backed up 2 more database clusters
- Dealt with our development instance of developer.mozilla.org having some latin1 MyISAM tables – converted them to utf8 InnoDB.
- Added permissions for the metrics team to read new fields in Bugzilla.
- Did a database consistency check for addons after a duplicate key error was triggered on several slaves, and re-sync’d the databases that had issues.
- Repaired a crashed table that looked like it worked, other than not showing recent inserts.
- Gave the Bugzilla team a place to test development releases of their code (as opposed to production releases)
- Updated the Firefox Flicks staging database with a fresh copy of the database from development.
- Decommissioned 2 database servers put into production for Addons due to 2 other machines that were crashing. The other 2 machines are healed, and the temporary servers were running out of disk space (they were smaller because they were temporary). We are back to having an n+2 configuration (without the 2 servers, we were exactly at n servers, meaning we had exactly as many servers as we needed. We like to run n+2, for just such an occasion)
PlanetMySQL Voting: Vote UP / Vote DOWN