I have had a few short weeks due to vacation time, so I have not posted one of these in a while. In the past few weeks I have done a lot of puppet configuration, including adding a MariaDB package install/configuration to our setup. I hope to consolidate our puppet configs to have only one MySQL manifest, right now there are three in use. Another theme to our work has been cleaning up legacy users and databases. Here is what the database team has accomplished in the past few weeks:
- Helped upgrade the firmware on a staging server (only to find out it had already been upgraded…isn’t that always the way?)
- Updated the downtime window for our db backup servers because we are adding more servers to it, and thus the backups are taking longer.
- Updated several Bugzilla bugs that had malformed UTF-8, aka mojibake. This was causing the entire bug not to render.
- Made sure we had off-site backups for crash-stats.mozilla.com
- Added permissions for our metrics staff to access code snippets for the Firefox about:home page.
- Purged a lot of historical data from TinderBox PushLog, and reduced the database size from
- Removed legacy databases from our Addons database.
- Inserted some new test values into our graphs database.
- Assisted a drive replacement for many of our SSDs that were faulty – the problem is, their fault was they showed a false positive, so it looked like the disk was failing when it was not. After several months of work with the vendor, this issue is finally resolved.
- Made sure legacy users and databases on our buildbot servers were removed.
- Helped upgrade the kernel and firmware on the databases for our Puppet Dashboard (Mozilla login required, sorry….).
- Built a new TokuDB cluster for our new Datazilla cluster.
- Setup backups on our backup server for that TokuDB cluster. This involved installing TokuDB side-by-side with MySQL on one of our backup servers.
- Helped debug an issue where tokens were not being stored properly in a database.
- Added auto_increment_increment and auto_increment_offset to our list of configurable /etc/my.cnf variables inside puppet, so that we can easily and automatically support master/master clusters.
- Added 4 machines to puppet from an existing dev/staging database cluster. This entailed figuring out what modifications had been done to the machine, including my.cnf, upgrading from MySQL 5.1 to Percona 5.1, and adding basedir, max_connections and wait_timeout to our list of configurable variables in puppet.
- Added expire_logs_days and max_binlog_size as configurable variables in puppet for Puppet Dashboard (Mozilla login required, sorry….).
- Talked to a reporter from Network World about how Mozilla uses ScaleBase to scale our databases.
PlanetMySQL Voting: Vote UP / Vote DOWN
