MySQL development is on a roll so now is the time for me to request a fix for another obscure performance problem in InnoDB. The insert buffer is a feature that can make InnoDB much faster than you expect. It significantly reduces random IO required for secondary index maintenance. For servers that I care about I estimate (using data from SHOW INNODB STATUS) the reduction is between 2:1 and 5:1 and secondary index maintenance requires 20% to 50% of the random disk IO it would have used without the insert/change buffer. With MySQL 5.5 the insert buffer is now the change buffer as it does more.

Write-optimized engines

Write-optimized engines have become popular recently. HBase, BigTable and Cassandra use variations of a Log Structured Merge Tree (LSM) and the big selling point for them is that they make writes cheap. However, sometimes the sales pitch leaves out the warning that they can make reads more expensive. 

Allow me to exaggerate but the insert/change buffer makes InnoDB write-optimized. Alas, I don't know what that means. The performance comparison between InnoDB and an LSM engine is not simple and the best evaluation includes real benchmarks, performance modeling on paper and expert advice. There are several factors to consider that have a significant impact on performance:

1. What is the ratio between reads and writes for your workload? If the workload on InnoDB is 80% from reads then that is less likely to benefit from a write-optimized engine than one with a load that is 20% from reads.
2. Is the write workload dominated by read-modify-write (update foo set x=x+1) or pure writes (update foo set x=2)? A read-modify-write workload is likely to have a high percentage of disk reads and will get less of a benefit from a write-optimized engine.
3. What is the distribution of block accesses for your workload? When it is uniform, it is very likely that a row read requires a disk read and a row write requires a disk write. But these assumptions are not true for some workloads and the benefit of a large cache can make performance modeling an interesting and hard problem. If you use InnoDB then you can estimate this by measuring the disk or page IO rate as a function of the buffer pool size.

The problem

There is one case in which the insert buffer stops working. The insert buffer is full when it uses ~50% of the memory from the buffer pool. At that point it is not used to reduce random reads for secondary index maintenance. The real problem is that it can stay full. The main background thread (srv_master_thread) uses at most 5% of innodb_io_capacity to request reads of secondary index leaf blocks. When the reads complete a callback applies pending changes from the insert buffer. For a modern server for which innodb_io_capacity is set to 1000, this will do at most 50 reads per second. Not only does the insert buffer stop helping performance when it stays full, it can hurt performance as a full insert buffer takes 50% of the buffer pool that could be used for caching index blocks.

The problem was fixed in the Google patch and is fixed in Facebook patch. It is also fixed in Percona -- search for "ibuf" in the innodb_io docs. I filed bug 59214 for this. The impact is visible in performance on the insert benchmark. There are significant drops in both the benchmark throughput (rows inserted/second) and the insert buffer insert rate. The drops occur at the same time. The drop does not occur on a server that has been modified to use more IO for insert buffer merges. Performance never recovers after the drop as the insert buffer remains full.

The results are from the Insert Benchmark using my version in launchpad. I inserted 1B rows into an InnoDB table, waited for the server to become idle, restarted the server and then measured performance while inserting 100M rows. I use mstat to collect data. However, the insert buffer statistics are not reported in SHOW STATUS for official MySQL so you need the Facebook patch to get the ibuf insert rate. 

I also used these my.cnf settings:

innodb_buffer_pool_size=8000M
innodb_log_file_size=1900M
innodb_flush_log_at_trx_commit=2
innodb_doublewrite=1
innodb_flush_method=O_DIRECT
innodb_thread_concurrency=0
innodb_max_dirty_pages_pct=80
innodb_file_per_table
innodb_file_format=barracuda
innodb_deadlock_detect=0
innodb_io_capacity=1000

Note that if you have the Facebook patch, then these settings fix the problem:

innodb_ibuf_max_iops_when_below_limit=10
innodb_ibuf_max_pct_of_buffer_pool=10
innodb_ibuf_max_pct_of_io_capacity=50

Image may be NSFW.
Clik here to view.