Created attachment 328804 [details] emerge --info I've recently updated from mysql 5.1 to the newly unmasked mysql 5.5. Now my mysqld won't start. /var/log/mysql/mysqld.err reports: 121108 7:00:31 InnoDB: The InnoDB memory heap is disabled 121108 7:00:31 InnoDB: Mutexes and rw_locks use GCC atomic builtins 121108 7:00:31 InnoDB: Compressed tables use zlib 1.2.7 121108 7:00:31 InnoDB: Using Linux native AIO 121108 7:00:31 InnoDB: Initializing buffer pool, size = 16.0M 121108 7:00:31 InnoDB: Completed initialization of buffer pool 121108 7:00:31 InnoDB: highest supported file format is Barracuda. 121108 7:00:31 InnoDB: Warning: allocated tablespace 5, old maximum was 0 121108 7:00:31 InnoDB: Assertion failure in thread 140230384047936 in file trx0rseg.c line 337 InnoDB: Failing assertion: page_no != FIL_NULL InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 06:00:31 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=16777216 read_buffer_size=262144 max_used_connections=0 max_threads=151 thread_count=0 connection_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 134073 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x1e)[0x78143e] /usr/sbin/mysqld(handle_fatal_signal+0x357)[0x672b17] /lib64/libpthread.so.0(+0x10fb0)[0x7f89eddcafb0] /lib64/libc.so.6(gsignal+0x35)[0x7f89ed30e155] /lib64/libc.so.6(abort+0x185)[0x7f89ed30f625] /usr/sbin/mysqld[0x82ce89] /usr/sbin/mysqld[0x82f5f5] /usr/sbin/mysqld[0x81c122] /usr/sbin/mysqld[0x7e9450] /usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x41)[0x674961] /usr/sbin/mysqld[0x59d2d6] /usr/sbin/mysqld(_Z11plugin_initPiPPci+0x7ef)[0x5a10cf] /usr/sbin/mysqld[0x52bed9] /usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x383)[0x52fe43] /lib64/libc.so.6(__libc_start_main+0xed)[0x7f89ed2fa94d] /usr/sbin/mysqld[0x527e55] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Running mysql_upgrade, as advised by the ebuild, fails because the server is not running: # mysql_upgrade Looking for 'mysql' as: mysql Looking for 'mysqlcheck' as: mysqlcheck Running 'mysqlcheck' with connection arguments: '--port=3306' '--socket=/var/run/mysqld/mysqld.sock' mysqlcheck: Got error: 2002: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) when trying to connect FATAL ERROR: Upgrade failed I can't rule out the possibility that my database might be corrupt, but the fact that it starts crashing after an upgrade seems indicative of a problem introduced by that upgrade. Using gdb, I created a more useful version of the stack trace at the time the SIGABRT is generated: #2 0x000000000082ce89 in trx_rseg_create () at mysql/storage/innobase/trx/trx0rseg.c:337 #3 0x000000000082f5f5 in trx_sys_create_rsegs (n_rsegs=127) at mysql/storage/innobase/trx/trx0sys.c:1365 #4 0x000000000081c122 in innobase_start_or_create_for_mysql () at mysql/storage/innobase/srv/srv0start.c:1836 #5 0x00000000007e9450 in innobase_init (p=<optimized out>) at mysql/storage/innobase/handler/ha_innodb.cc:2576 #6 0x0000000000674961 in ha_initialize_handlerton (plugin=0xfdc740) at mysql/sql/handler.cc:543 #7 0x000000000059d2d6 in plugin_initialize (plugin=0xfdc740) at mysql/sql/sql_plugin.cc:1100 #8 0x00000000005a10cf in plugin_init (argc=0xf768a4 <remaining_argc>, argv=0xfa4418, flags=0) at mysql/sql/sql_plugin.cc:1386 #9 0x000000000052bed9 in init_server_components () at mysql/sql/mysqld.cc:3904 #10 0x000000000052fe43 in mysqld_main (argc=32, argv=0xfa4418) at mysql/sql/mysqld.cc:4483 #11 0x00007ffff6ef694d in __libc_start_main (main= 0x51a280 <main(int, char**)>, argc=2, ubp_av=0x7fffffffd008, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffcff8) at libc-start.c:225 #12 0x0000000000527e55 in _start ()
After downgrading to mysql 5.1, the server started correctly, and a mysqldump of all tables succeeded as well. So I take it my data is intact, at least for 5.1. I've upgraded again, moved my data directory, configured mysql like a fresh install, and am currently playing back my dump.
That error looks a lot like http://bugs.mysql.com/bug.php?id=57387. Can you please post your my.cnf and /etc/conf.d/mysql?
Created attachment 328828 [details] my.cnf (In reply to comment #2) > That error looks a lot like http://bugs.mysql.com/bug.php?id=57387. The failed assertion mentioned there looks the same. But that assertion was for the case where innodb_data_file_path did not contain autoextend. On my system I have innodb_data_file_path='ibdata1:10M:autoextend:max:128M'. > Can you please post your my.cnf and /etc/conf.d/mysql? Non-comment lines from /etc/conf.d/mysql: MY_CNF="/etc/mysql/my.cnf" MY_ARGS="" STARTUP_TIMEOUT="900" STARTUP_EARLY_TIMEOUT="1000" STOP_TIMEOUT=120 [ "${SVCNAME}" != mysql ] && rc_provide="!mysql" return 0 my.cnf attached. The switch from language to lc_messages helped avoid an error message from 5.5 (when starting the daemon directly from the command line), but caused 5.1 to fail to start. So I switched between both alternatives there. The most notable change to my my.cnf from the distribution default is perhaps the fact that I disabled the binary log. Not sure whether that's got anything to do with anything.
(In reply to comment #3) > Created attachment 328828 [details] > my.cnf > > (In reply to comment #2) > > That error looks a lot like http://bugs.mysql.com/bug.php?id=57387. > > The failed assertion mentioned there looks the same. But that assertion was > for the case where innodb_data_file_path did not contain autoextend. On my > system I have innodb_data_file_path='ibdata1:10M:autoextend:max:128M'. > The bug was for when the main tablespace hit the maximum because it inflated another 8MB. What size is your ibdata1 file? Does the assertion still happen if you increase the max to like 256M?
(In reply to comment #4) > (In reply to comment #3) > > Created attachment 328828 [details] > > my.cnf > > > > (In reply to comment #2) > > > That error looks a lot like http://bugs.mysql.com/bug.php?id=57387. > > > > The failed assertion mentioned there looks the same. But that assertion was > > for the case where innodb_data_file_path did not contain autoextend. On my > > system I have innodb_data_file_path='ibdata1:10M:autoextend:max:128M'. > > > The bug was for when the main tablespace hit the maximum because it inflated > another 8MB. > > What size is your ibdata1 file? 128MiB exactly. Which seems kind of large, seeing as in the database restored from my dump, that same file is only 19M. But I must confess, I'm not really sure what that file contains. Does that file contain all the InnoDB tables from this MySQL instance? > Does the assertion still happen if you increase the max to like 256M? No, in that case, mysqld starts and ibdata1 grows to 142606336 bytes. mysql_upgrade succeeded, and the size stays as it is. Trying to find out what this file is all about, I found http://stackoverflow.com/q/3456159/1468366 . Sounds like reclaiming space from that file is a real problem, and perhaps I'd be better off with my newly restored dump instead of the old cruft. Perhaps I should even consider using innodb_file_per_table. What I find really disturbing is the discrepancy between this observed behaviour and the one described in http://dev.mysql.com/doc/refman/5.5/en/full-disk.html. From the manual, I'd have expected proper error messages whenever some resource, be it disk or tablespace, gets exhausted. The fact that a full disc should be handled more gracefully than a reached configuration limit seems really really strange. And a failed assertion together with a backtrace and a SIGABRT really feel inappropriate for such a configuration issue.
(In reply to comment #5) > (In reply to comment #4) > > (In reply to comment #3) > > The bug was for when the main tablespace hit the maximum because it inflated > > another 8MB. > > > > What size is your ibdata1 file? > > 128MiB exactly. Which seems kind of large, seeing as in the database > restored from my dump, that same file is only 19M. But I must confess, I'm > not really sure what that file contains. Does that file contain all the > InnoDB tables from this MySQL instance? The thing about Innodb is the main file never shrinks unless you dump the entire system and reload from scratch. innodb_file_per_table will help keep the size down. > > > Does the assertion still happen if you increase the max to like 256M? > > No, in that case, mysqld starts and ibdata1 grows to 142606336 bytes. > mysql_upgrade succeeded, and the size stays as it is. > > Trying to find out what this file is all about, I found > http://stackoverflow.com/q/3456159/1468366 . Sounds like reclaiming space > from that file is a real problem, and perhaps I'd be better off with my > newly restored dump instead of the old cruft. Perhaps I should even consider > using innodb_file_per_table. Exactly. > > What I find really disturbing is the discrepancy between this observed > behaviour and the one described in > http://dev.mysql.com/doc/refman/5.5/en/full-disk.html. From the manual, I'd > have expected proper error messages whenever some resource, be it disk or > tablespace, gets exhausted. The fact that a full disc should be handled more > gracefully than a reached configuration limit seems really really strange. > And a failed assertion together with a backtrace and a SIGABRT really feel > inappropriate for such a configuration issue. This is really an InnoDB issue which needs to be taken upstream. It should fail more gracefully. I expect it has to do with initialization failure going from newly created or updating an old tablespace.
(In reply to comment #6) > This is really an InnoDB issue which needs to be taken upstream. It should > fail more gracefully. Reported upstream as http://bugs.mysql.com/bug.php?id=67528
(In reply to comment #7) > (In reply to comment #6) > > This is really an InnoDB issue which needs to be taken upstream. It should > > fail more gracefully. > > Reported upstream as http://bugs.mysql.com/bug.php?id=67528 Can you please test with mysql-5.6.10 as requested in the upstream bug report?
(In reply to comment #8) > Can you please test with mysql-5.6.10 as requested in the upstream bug > report? I'm still waiting for an ebuild for mysql-5.6. So far, 5.5.30 is the latest in portage.
(In reply to comment #9) > (In reply to comment #8) > > Can you please test with mysql-5.6.10 as requested in the upstream bug > > report? > > I'm still waiting for an ebuild for mysql-5.6. So far, 5.5.30 is the latest > in portage. Please test the ebuild in the mysql overlay[1]. [1] - http://git.overlays.gentoo.org/gitweb/?p=proj/mysql.git
Marking as TEST-REQUEST as no feedback has been given for over a year
