Performance problem with Innodb and DROP TABLE

I’ve been working with an application which does a lot of CREATE and DROP table for Innodb tables and we’ve discovered DROP TABLE can take a lot of time and when it happens a lot of other threads stall in “Opening Tables” State. Also contrary to my initial suspect benchmarking create/drop table was CPU bound rather than IO bound.

I’ve got an oprofile run and saw the following:

15753796 56.0725  no-vmlinux               no-vmlinux               /no-vmlinux
11834143 42.1213  mysqld                   mysqld                   buf_LRU_invalidate_tablespace
168823    0.6009  mysql                    mysql                    completion_hash_update(hashtable*, char*, unsigned int, char*)
53667     0.1910  oprofiled                oprofiled                /usr/bin/oprofiled
42116     0.1499  mysqld                   mysqld                   buf_calc_page_new_checksum
32107     0.1143  mysqld                   mysqld                   srv_release_threads
14624     0.0521  mysqld                   mysqld                   srv_table_get_nth_slot

So we can see there is disproportionate amount of time spent in buf_LRU_invalidate_tablespace function which
did not sound right. The great thing with oprofile output you can usually Google couple of top functions and find bug reports or blog posts about this topic. It lead me to this bug at this time.

The problem is basically if you’re running innodb_file_per_table=1 the tablespace is dropped when you’re running DROP TABLE, and Innodb has to go through LRU list and discard the pages which belong to this tablespace. This can take a lot of time with large buffer pool. Worst of all this is done while table_cache lock is being held so no other queries can start.

This was both a shock and “I’ve seen something like this before” – there were problems of scanning large lists which were responsible for poor performance of SHOW STATUS and crash recovery performance over last couple of years and here we go again.

It would be great to get an Innodb a good review and see where are other linked links which are hiding somewhere. Whenever you have the list you really need to think how large it can get. We’re already seeing half a TB innodb_buffer_pool size and I expect it to close 1TB at the high end in the next few year. This will get the list of pages in buffer pool to be getting close to 100 millions of entries.

So as result of this bug – if you’re running innodb_file_per_table you will have a microstalls when tables are dropped and dropping a lot of tables in a batch can cause serve performance problems. The stalls can take more than a second for very large buffer pool.

If you’re using Innodb tables as transient tables where you create and drop them frequently you may avoid such design or use innodb_file_per_table=0 for such tables (thankfully you can change this variable online in Innodb Plugin, MySQL 5.5 and Percona Server).

Yes. if you’re not using innodb_file_per_table you’re not affected because in this case tablespace does not need to be dropped so pages from Buffer pool LRU do not need to be discarded. The raw performance of dropping tables I measured on the test server was 10 times better with single tablespace, though this will vary a lot depending on buffer pool size. MyISAM tables creating/dropping was several times faster still.