InnoDB compression for OLTP

The worst case for InnoDB compression might be a cached database and write-heavy workload. How bad is it in that case and what can be done to make it better? I describe how to increase throughput on tpcc-mysql by more than 10X when using compressed InnoDB tables. Peak throughput for 64 concurrent clients was 54331 TpmC without compression. Peak throughput with compression and prior to tuning was 2511 TpmC and after tuning increased to 27417 TpmC. All tests used MySQL 5.1.52 with the Facebook patch, 8-core servers (one for mysqld, one for clients) and a cached database.

The rest of this note describes the sequence of tests that were run where each test made one change to improve performance. Using abbreviations that are described below, the following configurations were tested.

• lev6, comp
• pad, lev6, comp
• pad, lev1, comp
• part, pad, lev6, comp
• part, pad, lev1, comp
• nolog, part, pad, lev1, comp
• nocomp

The abbreviations above mean:

• comp - use InnoDB compression with key_block_size=8
• lev6, lev1 - use innodb_compression_level={1, 6}. This is a new option in the Facebook patch and it sets the level of compression used by libz. The default level is 6 and switching to level 1 reduces the CPU overhead and latency from compression. Faster compression reduces mutex contention.
• pad - add a padding column to most of the tables to reduce the frequence of compression failure. A compression failure occurs when a page does not compress to 8kb assuming key_block_size=8 and a page split is done to handle the failure. There is limited concurrency on page writes per-index because the writer must lock the per-index mutex (Percona has written about this and so has Domas) and compression failures and page splits increase the duration for which it is locked. When tables have a padding column they use more space and are easier to compress. It is difficult to determine which tables have compression failures using official MySQL because the stats are not reported per-table. We added compression stats to IS.table_statistics in the Facebook patch to make it easy to determine which tables have frequent compression failures.
• part - use partitioning. This reduces contention on the per-index mutex because there will be one index (and one per-index mutex) per partition. It is much more useful for compressed tables because they suffer more from contention on the per-index mutex. Make sure you have a fix for bug 59013.
• nolog - use innodb_log_compress_pages=0 to reduce logging from InnoDB compression. This is only in the Facebook patch for MySQL. It reduces contention on log_sys->mutex. In many cases InnoDB logs page images for compressed tables. That is not done for uncompressed tables. When there is more to log there is more contention on log_sys->mutex. Nizam added an option to the Facebook patch to disable some cases where page image logging isn't required (assuming mysqld doesn't use a different version of libz during crash recovery).
• nocomp - don't use InnoDB compression

The number of InnoDB log writes during the tests can be used to understand the benefit of innodb_log_compressed_pages.

• 78M - no compression
• 115M - compression with innodb_log_compressed_pages=0
• 661M - compression with innodb_log_compressed_pages=1

Results

This lists throughput (as TpmC) for all of the described configurations.

• 54331 - no compression
• 27417 - nolog, partition, pad, level=1, compress
• 26150 - partition, pad, level=1, compress
• 16923 - pad, level=1, compress
• 11916 - pad, level=6, compress
• 3262 - level=1, compress
• 2511 - level=6, compress

This displays throughput (as TpmC) for all of the described configurations.

tpcc-mysql

The tpcc-mysql test was setup with 20 warehouses and tests were run with a 300 second warmup and 1800 second measurement period. Each test was run for 4, 16, 64, 256 and 1024 clients but throughput is only reported for 64 clients. The load was done prior to the 4 client test and then tpcc_start was run for 4 to 1024 clients. Some of the 1024 client tests failed because there were too many prepared statements. There was a delay after each run of tpcc_start to wait for purge lag to reach 0.

Padding options

A pad column was added to many tables to prevent compression failures. The cost of this is that the tables are much larger when uncompressed in the buffer pool and somewhat larger when compressed. This SQL was run to add pad columns:

    alter table history add column pad char(30) default "";
    alter table stock add column pad char(150) default ''";
    alter table item add column pad char(60) default ''";
    alter table customer add column pad char(255) default ''";
    alter table orders add column pad char(20) default ''";
    alter table new_orders add column pad char(20) default ''";
    alter table order_line add column pad char(40) default ''";

Compression options

When compression was used, all tables except warehouse and district were compressed:

    alter table customer engine=innodb key_block_size=8;
    alter table history engine=innodb key_block_size=8;
    alter table new_orders engine=innodb key_block_size=8;
    alter table orders engine=innodb key_block_size=8;
    alter table order_line engine=innodb key_block_size=8;
    alter table item engine=innodb key_block_size=8;
    alter table stock engine=innodb key_block_size=8;

Partitioning options

These are the per-table partitioning clauses:

• customer - PARTITION BY KEY(c_w_id, c_d_id) PARTITIONS 32
• new_orders - PARTITION BY KEY(no_w_id, no_d_id) PARTITIONS 32
• orders - PARTITION BY KEY(o_w_id, o_d_id) PARTITIONS 32
• order_line - PARTITION BY KEY(ol_w_id, ol_d_id, ol_o_id) PARTITIONS 32
• item - PARTITION BY KEY(i_id) PARTITIONS 32
• stock - PARTITION BY KEY(s_w_id, s_i_id) PARTITIONS 32

Common my.cnf settings

The common settings are:

[mysqld]
innodb_buffer_pool_size=16G
innodb_log_file_size=1900M
innodb_doublewrite=1
innodb_flush_method=O_DIRECT
innodb_max_dirty_pages_pct=80
innodb_file_format=barracuda
innodb_file_per_table
max_connections=2000
table_cache=2000
key_buffer_size=200M
innodb_io_capacity=2000
innodb_thread_concurrency=128
innodb_flush_log_at_trx_commit=2
skip_log_bin
innodb_buffer_pool_size=4G
innodb_use_purge_thread=4
# This is only in the Facebook patch
innodb_ibuf_max_pct_of_buffer_pool=20