Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 442346

Summary: dev-db/mysql-5.5.28 - InnoDB: Assertion failure in thread 140230384047936 in file trx0rseg.c line 337
Product: Gentoo Linux Reporter: Martin von Gagern <Martin.vGagern>
Component: Current packagesAssignee: Gentoo Linux MySQL bugs team <mysql-bugs>
Status: RESOLVED TEST-REQUEST    
Severity: normal    
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
URL: http://bugs.mysql.com/bug.php?id=67528
Whiteboard:
Package list:
Runtime testing required: ---
Attachments: emerge --info
my.cnf

Description Martin von Gagern 2012-11-08 12:49:49 UTC
Created attachment 328804 [details]
emerge --info

I've recently updated from mysql 5.1 to the newly unmasked mysql 5.5. Now my mysqld won't start. /var/log/mysql/mysqld.err reports:

121108  7:00:31 InnoDB: The InnoDB memory heap is disabled
121108  7:00:31 InnoDB: Mutexes and rw_locks use GCC atomic builtins
121108  7:00:31 InnoDB: Compressed tables use zlib 1.2.7
121108  7:00:31 InnoDB: Using Linux native AIO
121108  7:00:31 InnoDB: Initializing buffer pool, size = 16.0M
121108  7:00:31 InnoDB: Completed initialization of buffer pool
121108  7:00:31 InnoDB: highest supported file format is Barracuda.
121108  7:00:31  InnoDB: Warning: allocated tablespace 5, old maximum was 0
121108  7:00:31  InnoDB: Assertion failure in thread 140230384047936 in file trx0rseg.c line 337
InnoDB: Failing assertion: page_no != FIL_NULL
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
06:00:31 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 134073 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x1e)[0x78143e]
/usr/sbin/mysqld(handle_fatal_signal+0x357)[0x672b17]
/lib64/libpthread.so.0(+0x10fb0)[0x7f89eddcafb0]
/lib64/libc.so.6(gsignal+0x35)[0x7f89ed30e155]
/lib64/libc.so.6(abort+0x185)[0x7f89ed30f625]
/usr/sbin/mysqld[0x82ce89]
/usr/sbin/mysqld[0x82f5f5]
/usr/sbin/mysqld[0x81c122]
/usr/sbin/mysqld[0x7e9450]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x41)[0x674961]
/usr/sbin/mysqld[0x59d2d6]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0x7ef)[0x5a10cf]
/usr/sbin/mysqld[0x52bed9]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x383)[0x52fe43]
/lib64/libc.so.6(__libc_start_main+0xed)[0x7f89ed2fa94d]
/usr/sbin/mysqld[0x527e55]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.


Running mysql_upgrade, as advised by the ebuild, fails because the server is not running:

# mysql_upgrade 
Looking for 'mysql' as: mysql
Looking for 'mysqlcheck' as: mysqlcheck
Running 'mysqlcheck' with connection arguments: '--port=3306' '--socket=/var/run/mysqld/mysqld.sock' 
mysqlcheck: Got error: 2002: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) when trying to connect
FATAL ERROR: Upgrade failed


I can't rule out the possibility that my database might be corrupt, but the fact that it starts crashing after an upgrade seems indicative of a problem introduced by that upgrade.

Using gdb, I created a more useful version of the stack trace at the time the SIGABRT is generated:

#2  0x000000000082ce89 in trx_rseg_create ()
    at mysql/storage/innobase/trx/trx0rseg.c:337
#3  0x000000000082f5f5 in trx_sys_create_rsegs (n_rsegs=127)
    at mysql/storage/innobase/trx/trx0sys.c:1365
#4  0x000000000081c122 in innobase_start_or_create_for_mysql ()
    at mysql/storage/innobase/srv/srv0start.c:1836
#5  0x00000000007e9450 in innobase_init (p=<optimized out>)
    at mysql/storage/innobase/handler/ha_innodb.cc:2576
#6  0x0000000000674961 in ha_initialize_handlerton (plugin=0xfdc740)
    at mysql/sql/handler.cc:543
#7  0x000000000059d2d6 in plugin_initialize (plugin=0xfdc740)
    at mysql/sql/sql_plugin.cc:1100
#8  0x00000000005a10cf in plugin_init (argc=0xf768a4 <remaining_argc>, 
    argv=0xfa4418, flags=0)
    at mysql/sql/sql_plugin.cc:1386
#9  0x000000000052bed9 in init_server_components ()
    at mysql/sql/mysqld.cc:3904
#10 0x000000000052fe43 in mysqld_main (argc=32, argv=0xfa4418)
    at mysql/sql/mysqld.cc:4483
#11 0x00007ffff6ef694d in __libc_start_main (main=
    0x51a280 <main(int, char**)>, argc=2, ubp_av=0x7fffffffd008, 
    init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, 
    stack_end=0x7fffffffcff8) at libc-start.c:225
#12 0x0000000000527e55 in _start ()
Comment 1 Martin von Gagern 2012-11-08 15:47:38 UTC
After downgrading to mysql 5.1, the server started correctly, and a mysqldump of all tables succeeded as well. So I take it my data is intact, at least for 5.1. I've upgraded again, moved my data directory, configured mysql like a fresh install, and am currently playing back my dump.
Comment 2 Brian Evans Gentoo Infrastructure gentoo-dev 2012-11-08 15:48:43 UTC
That error looks a lot like http://bugs.mysql.com/bug.php?id=57387.

Can you please post your my.cnf and /etc/conf.d/mysql?
Comment 3 Martin von Gagern 2012-11-08 16:19:54 UTC
Created attachment 328828 [details]
my.cnf

(In reply to comment #2)
> That error looks a lot like http://bugs.mysql.com/bug.php?id=57387.

The failed assertion mentioned there looks the same. But that assertion was for the case where innodb_data_file_path did not contain autoextend. On my system I have innodb_data_file_path='ibdata1:10M:autoextend:max:128M'.

> Can you please post your my.cnf and /etc/conf.d/mysql?

Non-comment lines from /etc/conf.d/mysql:

MY_CNF="/etc/mysql/my.cnf"
MY_ARGS=""
STARTUP_TIMEOUT="900"
STARTUP_EARLY_TIMEOUT="1000"
STOP_TIMEOUT=120
[ "${SVCNAME}" != mysql ] && rc_provide="!mysql"
return 0

my.cnf attached. The switch from language to lc_messages helped avoid an error message from 5.5 (when starting the daemon directly from the command line), but caused 5.1 to fail to start. So I switched between both alternatives there. The most notable change to my my.cnf from the distribution default is perhaps the fact that I disabled the binary log. Not sure whether that's got anything to do with anything.
Comment 4 Brian Evans Gentoo Infrastructure gentoo-dev 2012-11-08 16:34:49 UTC
(In reply to comment #3)
> Created attachment 328828 [details]
> my.cnf
> 
> (In reply to comment #2)
> > That error looks a lot like http://bugs.mysql.com/bug.php?id=57387.
> 
> The failed assertion mentioned there looks the same. But that assertion was
> for the case where innodb_data_file_path did not contain autoextend. On my
> system I have innodb_data_file_path='ibdata1:10M:autoextend:max:128M'.
> 
The bug was for when the main tablespace hit the maximum because it inflated another 8MB.

What size is your ibdata1 file?
Does the assertion still happen if you increase the max to like 256M?
Comment 5 Martin von Gagern 2012-11-08 21:08:55 UTC
(In reply to comment #4)
> (In reply to comment #3)
> > Created attachment 328828 [details]
> > my.cnf
> > 
> > (In reply to comment #2)
> > > That error looks a lot like http://bugs.mysql.com/bug.php?id=57387.
> > 
> > The failed assertion mentioned there looks the same. But that assertion was
> > for the case where innodb_data_file_path did not contain autoextend. On my
> > system I have innodb_data_file_path='ibdata1:10M:autoextend:max:128M'.
> > 
> The bug was for when the main tablespace hit the maximum because it inflated
> another 8MB.
> 
> What size is your ibdata1 file?

128MiB exactly. Which seems kind of large, seeing as in the database restored from my dump, that same file is only 19M. But I must confess, I'm not really sure what that file contains. Does that file contain all the InnoDB tables from this MySQL instance?

> Does the assertion still happen if you increase the max to like 256M?

No, in that case, mysqld starts and ibdata1 grows to 142606336 bytes. mysql_upgrade succeeded, and the size stays as it is.

Trying to find out what this file is all about, I found http://stackoverflow.com/q/3456159/1468366 . Sounds like reclaiming space from that file is a real problem, and perhaps I'd be better off with my newly restored dump instead of the old cruft. Perhaps I should even consider using innodb_file_per_table.

What I find really disturbing is the discrepancy between this observed behaviour and the one described in http://dev.mysql.com/doc/refman/5.5/en/full-disk.html. From the manual, I'd have expected proper error messages whenever some resource, be it disk or tablespace, gets exhausted. The fact that a full disc should be handled more gracefully than a reached configuration limit seems really really strange. And a failed assertion together with a backtrace and a SIGABRT really feel inappropriate for such a configuration issue.
Comment 6 Brian Evans Gentoo Infrastructure gentoo-dev 2012-11-08 21:27:43 UTC
(In reply to comment #5)
> (In reply to comment #4)
> > (In reply to comment #3)
> > The bug was for when the main tablespace hit the maximum because it inflated
> > another 8MB.
> > 
> > What size is your ibdata1 file?
> 
> 128MiB exactly. Which seems kind of large, seeing as in the database
> restored from my dump, that same file is only 19M. But I must confess, I'm
> not really sure what that file contains. Does that file contain all the
> InnoDB tables from this MySQL instance?

The thing about Innodb is the main file never shrinks unless you dump the entire system and reload from scratch.
innodb_file_per_table will help keep the size down.


> 
> > Does the assertion still happen if you increase the max to like 256M?
> 
> No, in that case, mysqld starts and ibdata1 grows to 142606336 bytes.
> mysql_upgrade succeeded, and the size stays as it is.
> 
> Trying to find out what this file is all about, I found
> http://stackoverflow.com/q/3456159/1468366 . Sounds like reclaiming space
> from that file is a real problem, and perhaps I'd be better off with my
> newly restored dump instead of the old cruft. Perhaps I should even consider
> using innodb_file_per_table.

Exactly.

> 
> What I find really disturbing is the discrepancy between this observed
> behaviour and the one described in
> http://dev.mysql.com/doc/refman/5.5/en/full-disk.html. From the manual, I'd
> have expected proper error messages whenever some resource, be it disk or
> tablespace, gets exhausted. The fact that a full disc should be handled more
> gracefully than a reached configuration limit seems really really strange.
> And a failed assertion together with a backtrace and a SIGABRT really feel
> inappropriate for such a configuration issue.

This is really an InnoDB issue which needs to be taken upstream.  It should fail more gracefully.  I expect it has to do with initialization failure going from newly created or updating an old tablespace.
Comment 7 Martin von Gagern 2012-11-08 22:04:30 UTC
(In reply to comment #6)
> This is really an InnoDB issue which needs to be taken upstream.  It should
> fail more gracefully.

Reported upstream as http://bugs.mysql.com/bug.php?id=67528
Comment 8 Jorge Manuel B. S. Vicetto Gentoo Infrastructure gentoo-dev 2013-04-28 02:44:17 UTC
(In reply to comment #7)
> (In reply to comment #6)
> > This is really an InnoDB issue which needs to be taken upstream.  It should
> > fail more gracefully.
> 
> Reported upstream as http://bugs.mysql.com/bug.php?id=67528

Can you please test with mysql-5.6.10 as requested in the upstream bug report?
Comment 9 Martin von Gagern 2013-04-29 09:46:28 UTC
(In reply to comment #8)
> Can you please test with mysql-5.6.10 as requested in the upstream bug
> report?

I'm still waiting for an ebuild for mysql-5.6. So far, 5.5.30 is the latest in portage.
Comment 10 Jorge Manuel B. S. Vicetto Gentoo Infrastructure gentoo-dev 2013-04-29 11:48:33 UTC
(In reply to comment #9)
> (In reply to comment #8)
> > Can you please test with mysql-5.6.10 as requested in the upstream bug
> > report?
> 
> I'm still waiting for an ebuild for mysql-5.6. So far, 5.5.30 is the latest
> in portage.

Please test the ebuild in the mysql overlay[1].

 [1] - http://git.overlays.gentoo.org/gitweb/?p=proj/mysql.git
Comment 11 Brian Evans Gentoo Infrastructure gentoo-dev 2014-06-04 02:55:48 UTC
Marking as TEST-REQUEST as no feedback has been given for over a year