I did more performance tests with MySQL 5.1.47 unmodified, 5.5.8 unmodified and 5.1.52 with the Facebook patch. Some of the results are interesting so I will document them here for future reference. I run two synthetic benchmarks. These are useful but far from sufficient. Fortunately, Dimitri and Percona do a great job of running and documenting the results for other tests.
I use my fork of sysbench 0.4 for the OLTP test in two configurations: read-only and read-write. The read-only test transaction is fetch one row by primary key. The read-write test transaction is fetch one row by primary key and then update one row by primary key. The tests are simple but are still able to find problems the MySQL community is in the process of fixing.
All of the results below are the average transactions per second rate for tests at different levels of concurrency. Eight instances of the sysbench process were started where each instance used a separate table and started 1, 2, 4, 8, 16, 32 and then 128 threads. Between all instances there were 8, 16, 32, 64, 128, 256 and 512 threads running concurrently. The read-only tests were for 10 minutes at each level of concurrency. The read-write tests were run for 20 minutes at each level of concurrency. The mysqld server was given time to recover between each 20 minute run of the read write test where recovery meant that the test script waited for the InnoDB history list length to go back to 0. All clients were run on a separate host from the MySQL server. All tests use InnoDB. The InnoDB buffer pool was warm prior to the test, all data is cached in it and the binlog was disabled.
The tests were CPU bound as all data can fit in the InnoDB buffer pool. There were eight tables with 4M rows each and about 8G of data. The InnoDB buffer pool cached this data and was 16G.
MySQL 5.5.8 isn't strictly faster than 5.1.47
For many of the results below, MySQL 5.1.47 has higher peak throughput than 5.5.8. I don't know why but will save the investigation for another post.
Must fix bug 54790
Bug 54790 needs to be fixed in MySQL 5.5.8. Networking code in mysqld makes too many calls to fcntl and that creates too much contention on a lock_kernel in the Linux kernel. The results below are for the read-only test and unmodified MySQL (5.1.47, 5.5.8) saturate at 32 or 64 threads and then degrade. The results for 5.1.52 with the Facebook patch do not because of a temporary fix for the problem. Alas, the fix has since been removed from the patch as it breaks support for SSL.
8 16 32 64 128 256 512 threads
28929 59223 107826 85398 65939 66527 63166 mysql5147
28544 57557 94298 100288 62858 60728 58161 mysql558
29834 58566 105443 107148 106174 105270 101856 mysql5152
innodb_thread_concurrency is important for read-write workloads
Peak throughput for read-write workloads depends on the value of innodb_thread_concurrency. The peak degrades much more at high-concurrency when the value is 0.
These are results for innodb_thread_concurrency=0.
14652 26764 36279 31343 31419 5011 13506 mysql5147
13530 24000 26431 27639 24400 23833 18479 mysql558
13806 22711 29122 28808 27243 23269 12131 mysql5152
These are results for innodb_thread_concurrency=32.
14055 26414 36242 31981 34298 32337 24585 mysql5147
13843 24070 26991 27640 25920 25808 24976 mysql558
14094 22726 29043 29063 28812 27279 25294 mysql5152
innodb_thread_concurrency is less important for read-only workloads
Peak throughput for the read-only test does not depend on the value of innodb_thread_concurrency.
These are results for the read-only test with innodb_thread_concurrency=0.
29483 57553 109542 100886 68436 67490 65313 mysql5147
27922 57574 94804 99232 61112 60138 58986 mysql558
30419 56679 106163 108478 106852 106423 102285 mysql5152
These are results for the read-only test with innodb_thread_concurrency=32.
28929 59223 107826 85398 65939 66527 63166 mysql5147
28544 57557 94298 100288 62858 60728 58161 mysql558
29834 58566 105443 107148 106174 105270 101856 mysql5152
The InnoDB doublewrite buffer hurts performance
Peak throughput is frequently 10% to 20% higher with the doublewrite buffer disabled. I don't advocate running with it disabled. I think we can improve performance with the doublewrite buffer enabled.
This is peak throughput with the doublewrite buffer enabled for the read-write test.
14055 26414 36242 31981 34298 32337 24585 mysql5147
13843 24070 26991 27640 25920 25808 24976 mysql558
14094 22726 29043 29063 28812 27279 25294 mysql5152
This is peak throughput with the doublewrite buffer disabled for the read-write test.
14250 25485 36492 36413 35652 33558 29713 mysql5147
13487 23413 19716 33445 32822 32159 30281 mysql558
13953 24087 32624 32254 31705 30105 27240 mysql5152
Purge lag is much higher with 5.5.8 than 5.1.52
The values below are the purge lag at the end of each iteration of the test. The lag is much higher with 5.5.8. Two numbers are reported for 5.5.8: 5.5.8-0 has the result for innodb_purge_threads-0 and 5.5.8-1 has the result for innodb_purge_threads=1. The innodb_purge_threads value does not make a difference at high concurrency. I consider this to be a performance regression versus 5.1 and created bug 59291 for it. Prior to changing my benchmark scripts to pause between runs and wait for the purge lag to reduce to zero, performance for 5.5.8 was much worse, the ibdata1 file grew to 100GB or more and I filed bug 57489 before realizing that purge lag was the problem.
8 16 32 64 128 256 threads
15592 118953 91617 8017707 15197122 15214778 5.5.8-0
359 284 280 6376105 12897579 11738230 5.5.8-1
71260 80561 310605 17609 86113 52303 5.1.52
I repeated the 5.5.8 tests with innodb_purge_threads=1
Multiple buffer pool instances did not help this test
I know it will help other tests. It did not improve peak TPS for either the read-only or the read-write test. This result suggests that mutex contention on these tests occurs elsewhere.
Peak throughput for MySQL 5.5.8 on the read-write test with 1 and 4 buffer pool instances.
13843 24070 26991 27640 25920 25808 24976 1-pool
13017 21027 28659 28795 23573 24629 23548 4-pools
InnoDB compression
These are results using MySQL 5.1.47 with InnoDB compression (key_block_size=8) and without it. The InnoDB buffer pool was large enough to store compressed and uncompressed copies of all pages. That is unlikely to be true for real workloads.
Results for the read-only test
20169 37480 80691 98057 66482 65056 63402 compressed
28929 59223 107826 85398 65939 66527 63166 uncompressed
Results for the read-write test
9446 15092 21620 18657 16964 16022 15716 compressed
14055 26414 36242 31981 34298 32337 24585 uncompressed
my.cnf
Settings for 5.5.8
innodb_buffer_pool_size=16G
innodb_log_file_size=1900M
innodb_flush_log_at_trx_commit=2
innodb_doublewrite=1
innodb_flush_method=O_DIRECT
innodb_thread_concurrency=0
innodb_max_dirty_pages_pct=80
innodb_file_format=barracuda
innodb_file_per_table
max_connections=2000
table_cache=2000
innodb_thread_concurrency=(32 or 0)
Settings for 5.1.52
innodb_buffer_pool_size=16G
innodb_log_file_size=1900M
innodb_flush_log_at_trx_commit=2
innodb_doublewrite=1
innodb_flush_method=O_DIRECT
innodb_thread_concurrency=0
innodb_max_dirty_pages_pct=80
innodb_file_format=barracuda
innodb_file_per_table
max_connections=2000
table_cache=2000
innodb_thread_concurrency=(32 or 0)
plugin-load=innodb=ha_innodb_plugin.so;innodb_trx=ha_innodb_plugin.so;innodb_locks=ha_innodb_plugin.so;innodb_lock_waits=ha_innodb_plugin.so;innodb_cmp=ha_innodb_plugin.so;innodb_cmp_reset=ha_innodb_plugin.so;innodb_cmpmem=ha_innodb_plugin.so;innodb_cmpmem_reset=ha_innodb_plugin.so
innodb_buffer_pool_size=16G
innodb_log_file_size=1900M
innodb_flush_log_at_trx_commit=2
innodb_doublewrite=1
innodb_flush_method=O_DIRECT
innodb_thread_concurrency=0
innodb_max_dirty_pages_pct=80
innodb_file_format=barracuda
innodb_file_per_table
max_connections=2000
table_cache=2000
innodb_thread_concurrency=(32 or 0)
Command lines
Command line for the read-write test:
sysbench --batch --batch-delay=60 --test=oltp \
--oltp-table-size=4000000 --max-time=1200 --maxrequests=0 \
--mysql-table-engine=innodb --db-ps-mode=disable \
--mysql-engine-trx=yes --oltp-table-name=sbtest1 \
--oltp-skip-trx --oltp-test-mode=simple \
--oltp-point-select-all-cols --oltp-simple-update \
--oltp-dist-type=uniform --oltp-range-size=1
--num-threads=${nt} --seed-rng=1 run
Command line for the read-only test:
sysbench --batch --batch-delay=60 --test=oltp \
--oltp-table-size=4000000 --max-time=600 --max-requests=0 \
--mysql-table-engine=innodb --db-ps-mode=disable \
--mysql-engine-trx=yes --oltp-table-name=sbtest1 \
--oltp-read-only --oltp-skip-trx --oltp-test-mode=simple \
--oltp-point-select-all-cols --oltp-dist-type=uniform \
--oltp-range-size=1 --num-threads=${nt} --seed-rng=1 run
Compile flags
5.1.52 with the facebook patch
./configure --without-debug --enable-thread-safe-client \
--prefix=$b --exec-prefix=$b \
--with-plugins=partition,csv,blackhole,myisam,heap,innodb_plugin \
--without-plugin-innobase \
--with-unix-socket-path=$b/var/mysql.sock \
--with-fast-mutexes --with-extra-charsets=all \
C_EXTRA_FLAGS="-g -O2 -fno-omit-frame-pointer -fno-strict-aliasing -DNO_ALARM -DSIGNAL_WITH_VIO_CLOSE -Wall"
5.1.47
./configure --prefix=$b --enable-thread-safe-client \
--with-plugins=partition,csv,blackhole,myisam,heap,innodb_plugin \
--without-plugin-innobase --with-fast-mutexes \
--with-extra-charsets=all --without-debug \
C_EXTRA_FLAGS="-g -O2 -fno-omit-frame-pointer -fno-strict-aliasing -DNO_ALARM -DSIGNAL_WITH_VIO_CLOSE -Wall"
The flags for 5.5.8. I do not use aio for InnoDB background IO. This used -O3 where the other binaries use -O2 optimization.
~/bin/cmake . \
-DBUILD_CONFIG=mysql_release_noaio \
-DMYSQL_UNIX_ADDR=/data/558orig/data/mysql.sock \
-DWITH_EMBEDDED_SERVER=0 \
-DWITH_PERFSCHEMA_STORAGE_ENGINE=1
PlanetMySQL Voting: Vote UP / Vote DOWN