Revisiting the InnoDB memory overhead

InnoDB requires an extra 1462 bytes per buffer pool page for MySQL 5.0 with an x86-64 server. For the default page size of 16kb this uses about 8% of the memory on a server and memory is precious for IO-bound workloads. The overhead is 15% for 8kb page. If you were clever enough to make InnoDB support 4kb pages then it is 26%.

A side-effect of the rw-lock rewrite for the InnoDB plugin in MySQL 5.1 reduces the overhead to 1144 bytes per page or 6.4% of the memory on a server because 1144 / (1144 + 16384) ~= .064. Note that all of these numbers are approximate and slightly underestimate the overhead.

Prior to figuring out how much memory was used I had a goal. If I could reduce the overhead by 5% (from 6.4% to 1.4% in this case) then it would be worth fixing. I don't think it is feasible to do that for 16kb pages. It might be possible to reduce the overhead from 12% to 7% for 8kb pages (1144 to 636 bytes), but 8kb pages might not improve performance for other reasons.

What accounts for the 1144 byte overhead per page?

• 752 bytes for buf_block_struct that maintains per-page metadata
• 256 bytes for the adaptive hash index in btr_search_sys->hash_index
• 40 bytes for lock_sys->rec_hash
• 32 bytes for dict_sys->table_hash
• 32 bytes for dict_sys->table_id_hash
• 16 bytes for buf_pool->page_hash
• 16 bytes for buf_pool->zip_hash

There are a few simple things to fix, but they won't make a huge difference. InnoDB defines a boolean type, ibool, that uses ulint (unsigned long) and sizeof(long) is 8 on x86-64. Some of the structs above use ulint, lint, ulong and long where uint and int are sufficient.

The biggest problem is the bloat from mutex_t (mutex_struct) and rw_lock_t (rw_lock_struct). Each buf_block_struct instance contains one of each. The sizeof(mutex_struct) is 64 but it contains a pointer to os_event_struct which uses 128 bytes. The sizeof(rw_lock_struct) is 112 but it contains pointers to two os_event_struct instances. Thus (64 + 128) + (112 + 128 + 128) = 560 of the 1144 bytes are from this. That is too much memory given how frequently these are used.

Why does mutex struct use 172 bytes?

• os_event_t event - 128 bytes for the struct and 8 bytes for the pointer to it
• lock_word_t lock_word (4 bytes)
• ulint waiters (8 bytes)
• char* cfile_name (8 bytes)
• ulint cline (8 bytes)
• ulong count_os_wait (8 bytes)
• pointers for a double-linked list of all mutex_struct instances (16 bytes)

Why does rw_lock_struct use 368 bytes?

• os_event_t event, wait_ex_event - 256 bytes for the structs and 16 bytes for pointers
• lint lock_word (8 bytes)
• ulint waiters (8 bytes)
• ulint recursive (8 bytes)
• pointers for a double-linked list of all rw_lock_struct instances (16 bytes)
• ulint count_os_wait (8 bytes)
• char *cfile_name, last_s_file_name, last_x_file_name (24 bytes)
• unsigned:14 cline, last_s_line, last_x_line

The memory can be reduced by a small amount when several of the fields are switched from 8-byte to 4-byte ints. The cline fields can be removed if mutex and rw-lock creators are identified by name rather than file:line.

The big win comes from removing the os_event_t fields and that requires an architectural change. I don't think that will get done until there are benefits beyond reduce the memory overhead.