InnoDB performance: 5.5 versus 5.1 for concurrent writes and an IO bound server

I ran sysbench for a simple read-write workload on a 16-core x86 server using MySQL 5.1 and was disappointed by the results. I then tried 5.5 and was thrilled by the results. I tested MySQL 5.1.50 with the InnoDB plugin and the Facebook patch. That was compared with MySQL 5.5.6-rc with InnoDB.

The benchmark used 8 sysbench tables with 72M rows each. The primary key indexes for all tables were 120GB and a 4GB InnoDB buffer pool was used. 8 instances of sysbench were run where each instance used between 1 and 128 threads so that between 8 and 1024 concurrent connections were used. Each transaction in the test was one SELECT and one UPDATE statement. Each used the primary key to find the row and each accessed one row.

The tests are IO bound and the server can do ~1500 random disk reads per second. Both 5.1 and 5.5 are able to saturate the IO capacity of the box, although 5.1 is only able to do that until it saturates on mutex contention.

The results are listed in transactions per second for 8 to 1024 concurrent clients and can also be viewed as a graph. From the results it is obvious that there is a serious bottleneck in 5.1 and I reported bug 57021 for this. The problem is mutex contention. Fortunately I am unable to reproduce this problem on an 8-core server.

    8        16        32      64      128     256     512    1024        # concurrent clients
355     465     730     754     856     870     658     570          556rc-base
343     525     667     731     698     670     704     697          556rc-bp8
426     445     543     603     614     611     614     624          556rc-bp4
313     541     517     574     664     646     484     286          556rc-io1000
302     497     494     487     499     551     403     234          556rc-purge
483     536     689     836     133       33        32       77          51fb-io1000
484     535     686     838     138       37        32       70          51fb-base

I tested the following servers:

• 556rc-base - MySQL 5.5.6-rc, InnoDB and my.cnf listed below
• 556rc-bp8 - 556rc-base and innodb_buffer_pool_instances=8
• 556rc-bp4 - 556rc-base and innodb_buffer_pool_instances=4
• 556rc-io1000 - 556rc-base and innodb_io_capacity=1000
• 556rc-purge - 556rc-base and innodb_purge_threads=1
• 51fb-io1000 - 51fb-base and innodb_io_capacity=1000
• 51fb-base - MySQL 5.1.50, Facebook patch, InnoDB plugin, innodb_deadlock_detect=0

The default my.cnf is listed below. The binlog is disabled for this test because I want to determine whether bottlenecks exist beyond the lack of group commit.

innodb_buffer_pool_size=4G
innodb_log_file_size=1900M
innodb_flush_log_at_trx_commit=2
innodb_doublewrite=1
innodb_flush_method=O_DIRECT
innodb_thread_concurrency=0
innodb_max_dirty_pages_pct=80
innodb_file_format=barracuda
innodb_file_per_table
max_connections=2000
table_cache=2000

I use a custom version of sysbench available in launchpad and this command line with nt varying from 1 to 128 and tn varying from sbtest1 to sbtest8. Eight instances of sysbench are run concurrently.

sysbench --test=oltp --oltp-table-size=72000000 --max-time=300 \
    --max-requests=0 --mysql-table-engine=innodb \
    --db-ps-mode=disable --mysql-engine-trx=yes \
    --oltp-table-name=sbtest${i} --oltp-skip-trx \
    --oltp-test-mode=simple --oltp-point-select-all-cols \
    --oltp-simple-update --oltp-dist-type=uniform \
    --oltp-range-size=1 --num-threads=$nt --seed-rng=1 run