Cool stuff from TiVo’s MySQL patch

While not as well known as the Google or Facebook patches the TiVo mysql patch includes some changes that make monitoring and query optimization a bit easier than mainline MySQL. The modified tarball, available at http://www.tivo.com/mysql/ contains a modified mysqld_safe, mysqldump, and improvements to the row statistics show commands that came from google.

mysqld_safe modifications

–fallback-ledir
The –ledir option tells mysqld_safe which directory contains the mysqld file that it should use. TiVo has added the –fallback-ledir option which will switch the ledir should mysqld crash. This makes it possible to run a new mysqld binary then fallback to the stock one should the new one crash. This feature has never been called to duty in production but it makes me sleep better.

–crash-script
This is a new option to mysqld_safe which will execute a script when it detects mysqld has crashed. This can be useful for gathering diagnostic info, sending monitoring alerts or even triggering a failover. Remember if you choose to use this option to trigger a failover make sure your failover mechanism has flap prevention so the master won’t ping/pong back and forth between servers.

mysqldump modifications

–slave-data
I think this option has been implemented by several people. We have also added it because it makes life easier for making backups. What this does is mark the replication position of the slave thread in the mysqldump output. It’s also useful for creating a dump file from a slave and using it to restore a peer slave.

–log-pos-comment
This makes dump files a little bit friendlier for restoring backups on one of an HA pair of machines. It writes out a pair of comments with replication positions from both the slave thread and the master position in a human readable format. This is handy for debugging restore scripts as well as those rare situations when a backup needs to be restored by hand.

–no-bin-log
This tells mysqldump to write a SET SQL_LOG_BIN=0; to the top of the dump file in the same way other variables are set. This makes dump files import only on the machine and not be written to the replication stream. It makes dump files a bit safer for importing on an HA pair. TiVo has multiple tools that work with sqldump files so this was a good mechanism to make the dump files replication safe.

The rest of the modifications are in mysqld itself. TiVo makes extensive use of the row stats options originally created at Google. The command syntax has been modified from Google’s original version to make it easier for developers to use the commands to debug queries as well as allowing the application servers themselves to access the data per session or per query.

TiVo has complicated MySQL queries, really complicated. Sometimes it’s difficult to figure out which part of the query is doing the most work or what changed the performance between two different versions of a query. One way to help solve that problem is to keep track of the rows being accessed in a query both for that query and for the session.

This description applies for both SHOW TABLE_STATISTICS and SHOW INDEX_STATISTICS commands.

Row stats in the TiVo patch are tracked at 3 different levels, per query, per session, and globally. The original Google patch only tracked row stats globally. Tracking them per session and per query doesn’t add much overhead and provides valuable information about query performance. It’s true that not every row access is the same but this has provided an easier measure for work done by a query than wall clock time or even rows examined from the slow query log.

After a query has executed the row statistics for that query can be accessed with:

mysql> select * from t;
+------+
| t    |
+------+
|    1 |
|    2 |
|    3 |
+------+
3 rows in set (0.00 sec)

mysql> show table_statistics;
+--------+-----------+--------------+-------------------------+
| Table  | Rows_read | Rows_changed | Rows_changed_x_#indexes |
+--------+-----------+--------------+-------------------------+
| test.t |         3 |            0 |                       0 |
+--------+-----------+--------------+-------------------------+
1 row in set (0.00 sec)

If the query is executed again show table_statistics will still only return results for the previous query but show session table_statistics will return cumulative counts for the duration of the session. The rows changed in the session query is from the insert statement which I used to create the test table.

mysql> select * from t;
+------+
| t    |
+------+
|    1 |
|    2 |
|    3 |
+------+
3 rows in set (0.00 sec)

mysql> show table_statistics;
+--------+-----------+--------------+-------------------------+
| Table  | Rows_read | Rows_changed | Rows_changed_x_#indexes |
+--------+-----------+--------------+-------------------------+
| test.t |         3 |            0 |                       0 |
+--------+-----------+--------------+-------------------------+
1 row in set (0.00 sec)

mysql> show session table_statistics;
+--------+-----------+--------------+-------------------------+
| Table  | Rows_read | Rows_changed | Rows_changed_x_#indexes |
+--------+-----------+--------------+-------------------------+
| test.t |         6 |            3 |                       3 |
+--------+-----------+--------------+-------------------------+
1 row in set (0.00 sec)

The final way to access the row statistics is the global count for the server. All of these commands also support using like syntax to narrow down the list of tables. In this case I’m showing the global stats which include the mysql tables.

mysql> show global table_statistics like 'mysql%';
+----------------------+-----------+--------------+-------------------------+
| Table                | Rows_read | Rows_changed | Rows_changed_x_#indexes |
+----------------------+-----------+--------------+-------------------------+
| mysql.proc           |       293 |            4 |                       4 |
| mysql.user           |  10237928 |            2 |                       2 |
| mysql.help_category  |        36 |            0 |                       0 |
| mysql.help_topic     |       463 |            0 |                       0 |
| mysql.help_relation  |       733 |            0 |                       0 |
| mysql.help_keyword   |       381 |            0 |                       0 |
| mysql.db             |  10964886 |            0 |                       0 |
+----------------------+-----------+--------------+-------------------------+
8 rows in set (0.00 sec)

Since these statistics are readily available within MySQL the TiVo patch modifies the slow query log header to include table and index statistics.

# Time: 100725 14:39:33
# User@Host: tivo[tivo] @ localhost []
# Query_time: 3  Lock_time: 0  Rows_sent: 3  Rows_examined: 3
# Row_Stats: test.t:rows_read=3,rows_changed=0,rows_changed_x_indexes=0;
# Index_Stats:
select *, sleep(1) from t;

The TiVo patch also includes changes to InnoDB from the Google patch and from Percona.