Dealing with Assertion failure in file fut0lst.ic : addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA

I recently wrote an article on dealing with an assertion failure in log/log0recv.c, specifically !page || (ibool)!!page_is_comp(page) == dict_table_is_comp(index->table).

I mention it because this occurred after a system outage, and I just encountered another system outage (either HDD power outage or some other serious HDD event), with a completely different assertion failure and error message. Similar to the one above, it’s also kind of obscure, so I wanted to post this so people searching for it will find this.

For reference, the first outage assertion failure was this:

InnoDB: Assertion failure in thread 139838283589376 in file
log/log0recv.c line 1094
InnoDB: Failing assertion: !page || (ibool)!!page_is_comp(page) ==
dict_table_is_comp(index->table)

Here is the new assertion failure:

111201 16:45:00 InnoDB: Assertion failure in thread 4500 in
file fut0lst.ic line 83
InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset >=
FIL_PAGE_DATA

As you can see, they are completely different, yet both due to similar, catastrophic events:

In fact, there are a number of relevant/useful snippets from this error log (in case you see one or more of these):

InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Warning: a page in the doublewrite buffer is not within space
InnoDB: bounds; space id 0 page number 2248, page 22 in doublewrite buf.
InnoDB: Warning: a page in the doublewrite buffer is not within space
InnoDB: bounds; space id 0 page number 1775, page 31 in doublewrite buf.
...
111201 16:45:00 InnoDB: ERROR: We were only able to scan the log up to
InnoDB: 670800896, but a checkpoint was at 670801309.
InnoDB: It is possible that the database is now corrupt!
111201 16:45:00 InnoDB: Error: page 7 log sequence number 924466223
InnoDB: is in the future! Current system log sequence number 670801309.
...
111201 16:45:00 InnoDB: Assertion failure in thread 4500 in file
fut0lst.ic line 83
InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset
>= FIL_PAGE_DATA
...
111201 16:45:00 - mysqld got exception 0xc0000005 ;
...
000000013F6A4C6F mysqld.exe!my_osmaperr()
000000013F69B9A0 mysqld.exe!my_osmaperr()
000000013F4727D6 mysqld.exe!?ha_initialize_handlerton@@
YAHPEAUst_plugin_int@@@Z()
000000013F46C172 mysqld.exe!?plugin_lock_by_name@@
YAPEAUst_plugin_int@@PEAVTHD@@PEBUst_mysql_lex_string@@H@Z()
000000013F4713E9 mysqld.exe!?plugin_init@@YAHPEAHPEAPEADH@Z()
000000013F45BDA7 mysqld.exe!handle_shutdown()
000000013F45C91A mysqld.exe!?win_main@@YAHHPEAPEAD@Z()
000000013F45CD90 mysqld.exe!?mysql_service@@YAHPEAX@Z()
000000013F45D0A3 mysqld.exe!?mysqld_main@@YAHHPEAPEAD@Z()
000000013F7FBB27 mysqld.exe!my_mb_ctype_mb()
00000000777F652D kernel32.dll!BaseThreadInitThunk()
0000000077A2C521 ntdll.dll!RtlUserThreadStart()

As you can see, the corruption was severe … so severe that InnoDB wouldn’t even start with innodb_force_recovery set to 6!

As discussed in the previous post, setting innodb_doublewrite=1, innodb_flush_log_at_trx_commit=1, sync_binlog=1, and having a battery backed cache can be the best bets against such issues.

However, I will say that in this case, innodb_flush_log_at_trx_commit=1 and innodb_doublewrite=1 were both set, and binary logging had been disabled, leaving only the battery backed cache to be desired.

Luckily, this was a test environment (and recovery from a backup was possible). Otherwise, using the InnoDB Recovery Tool would be necessary, which can be an undertaking.

Morals of the story: take regular backups (as always) and invest in a battery backed cache.