There is a very serious bug in either mysql ebuild or one of the libraries it is build upon. I've sent the bug report to the MySQL developers, but after doing some testing I've found that MySQL binary distribution is unaffected of this. So the bug must either lie in the ebuild of mysql or one of the libraries it depends on. The setup the SQL server is running in is a web cluster of 9 computers that access the SQL server, which is a dual Xeon server dedicated to run mysqld. It is our main production setup, so we had to replace the crucial server with an old single processor machine, we've got our system to use the crashing SQL server as a proxy SQL server so that all queries are forwarded to both servers, this allows us to run test with the crashing server. We are probably the only ones that will be able to reproduce the crash, so if you have anything you want to test, I'll do it. The crash only exists on our dual Xeon server, so it seems to be related to SMP machines only. Also it happens very often, during a 10 second periode of uptime for our SMP server the crash occured three times, so we have no problem reproducing it. For more information please read the MySql bug report here: http://bugs.mysql.com/bug.php?id=2332 I managed to get a backtrace from gdb: (gdb) backtrace full #0 0x4007c083 in get_field () from /usr/lib/libwrap.so.0 No symbol table info available. #1 0x4007bbf2 in process_options () from /usr/lib/libwrap.so.0 No symbol table info available. #2 0x4007b31d in table_match () from /usr/lib/libwrap.so.0 No symbol table info available. #3 0x4007b15b in hosts_access () from /usr/lib/libwrap.so.0 No symbol table info available. #4 0x0810de24 in handle_connections_sockets () No symbol table info available. #5 0x0810d69e in main () No symbol table info available. #6 0x402c38dc in __libc_start_main () from /lib/libc.so.6 No symbol table info available. I have been able to reproduce the crash with CFLAGS stripped down to: CFLAGS=-O3 -pipe My use flags are: USE="-X -qt -gnome -gtk innodb mmx odbc perl postgres" Adding +debug to the mysql build will prevent the crash from happening. Reproducible: Always Steps to Reproduce: 1. I've only managed to reproduce it on our own server cluster 2. 3. Actual Results: Mysqld crashed with a signal 11, backtrace included in description. Expected Results: No crash please. Portage 2.0.49-r20 (default-x86-1.4, gcc-3.2.3, glibc-2.3.2-r3, 2.4.22-gentoo-r2) ================================================================= System uname: 2.4.22-gentoo-r2 i686 Intel(R) Xeon(TM) CPU 2.40GHz Gentoo Base System version 1.4.3.10 distcc 2.11.1 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled] ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CFLAGS="-O3 -march=pentium4 -mcpu=pentium4 -funroll-loops -fprefetch-loop-arrays -pipe" CHOST="i686-pc-linux-gnu" COMPILER="gcc3" CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3/share/config /usr/share/config /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/env.d" CXXFLAGS="-O3 -march=pentium4 -mcpu=pentium4 -funroll-loops -fprefetch-loop-arrays -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="autoaddcvs ccache sandbox" GENTOO_MIRRORS="http://gentoo.bzimage.dk/pub/gentoo ftp://ftp.uninett.no/pub/linux/Gentoo" MAKEOPTS="-j3" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="" SYNC="rsync://gentoo.bzimage.dk/gentoo-portage" USE="apm arts avi berkdb crypt cups encode foomaticdb gdbm gif gpm gtk2 imlib innodb jpeg kde libg++ libwww mad mikmod mmx motif mpeg mysql ncurses nls odbc oggvorbis opengl oss pam pdflib perl png postgres python quicktime readline sdl slang spell ssl svga tcpd truetype x86 xml2 xmms xv zlib"
Okay after doing more debuggin I've found that this bug is tcp_wrappers specific! Happens only when /etc/hosts.allow or /etc/hosts.deny is not pressent, this in turn causes the "string"!? To be undefined. This looks like a _very_ serious flaw in tcp_wrappers, and could have the same impact on a lot of programs! Better backtrace (-g3 -O0), notice "string" is out of bounds: 0x4007ca63 in get_field (string=0x73796d2f <Address 0x73796d2f out of bounds>, last=0xbfffdf7c) at options.c:587 587 options.c: No such file or directory. in options.c (gdb) bt full #0 0x4007ca63 in get_field (string=0x73796d2f <Address 0x73796d2f out of bounds>, last=0xbfffdf7c) at options.c:587 src = 0x73796d2f <Address 0x73796d2f out of bounds> dst = 0x73796d2f <Address 0x73796d2f out of bounds> ret = 0x73796d2f <Address 0x73796d2f out of bounds> ch = 1073784592 #1 0x4007befc in process_options (options=0x73796d2f <Address 0x73796d2f out of bounds>, request=0xbfffe860) at options.c:142 key = 0x1 <Address 0x1 out of bounds> value = 0x403d2910 "" curr_opt = 0x4000a710 "ZY\207\004$
Okay after doing more debuggin I've found that this bug is tcp_wrappers specific! Happens only when /etc/hosts.allow or /etc/hosts.deny is not pressent, this in turn causes the "string"!? To be undefined. This looks like a _very_ serious flaw in tcp_wrappers, and could have the same impact on a lot of programs! Better backtrace (-g3 -O0), notice "string" is out of bounds: 0x4007ca63 in get_field (string=0x73796d2f <Address 0x73796d2f out of bounds>, last=0xbfffdf7c) at options.c:587 587 options.c: No such file or directory. in options.c (gdb) bt full #0 0x4007ca63 in get_field (string=0x73796d2f <Address 0x73796d2f out of bounds>, last=0xbfffdf7c) at options.c:587 src = 0x73796d2f <Address 0x73796d2f out of bounds> dst = 0x73796d2f <Address 0x73796d2f out of bounds> ret = 0x73796d2f <Address 0x73796d2f out of bounds> ch = 1073784592 #1 0x4007befc in process_options (options=0x73796d2f <Address 0x73796d2f out of bounds>, request=0xbfffe860) at options.c:142 key = 0x1 <Address 0x1 out of bounds> value = 0x403d2910 "" curr_opt = 0x4000a710 "ZY\207\004$Â\b" next_opt = 0xbfffe7e8 "\030èÿ¿|²\a@±å\a@`èÿ¿<\003" last = 0x4007e71d "" op = (struct option *) 0xbfffe860 bf = "warning: cannot open %s: %m\0\0\0\0\0Ã\0\0\0\0\0\0\0$ô\001@ØK\002@\224\231\221\bÈ\234\221\b¸¿ÿ¿WÊ\001@ô\tE\bx\235\221\bháÿ¿Îå2\bä\tE\bÈ\234\221\bÀ\233\221\b ×ÿ¿\0\0\0\0È×ÿ¿P\b\0\0Ì×ÿ¿ñ\0\0\0Ä×ÿ¿\021\0\0\0\0\0\0\0À\0\0\0`Öÿ¿\0\0\0\0dÖÿ¿\036\0\0\0lÖÿ¿\021\0\0\0hÖÿ¿\021\0\0\0xÕÿ¿\b\0\0\0|Õÿ¿\b\0\0\0h×ÿ¿ \0\0\0\0\0\0\0þþ\a\001\0\002\001i"... #2 0x4007b4ed in table_match (table=0x4007e5b1 "/etc/hosts.deny", request=0xbfffe860) at hosts_access.c:173 fp = (FILE *) 0x0 sv_list = "ysql/columns_priv.MYI\0ÿ¿ÿ\001\0\0\b\0\0\0@§\220\b\030àÿ¿(âÿ¿!6\bpäÿ¿\020àÿ¿\001\0\0\0ã\0226\b./mysql/columns_priv\0ql/\0üÿÿ\003", '\0' <repeats 11 times>, "\227\0\0\0\227\0\0\0ä\003\0\0\001\0\0\0\003\0\0\0\0\0\0\0Ðd\221\bÀd\221\b\002\0\0\0\002\0\0\0\210[\221\bÿÿÿÿ\020\0\0\0\231Sü?\0\0\0\0\0\0\0\0\005\0\0\0\227\0\0\0Àa\221\b ãÿ¿Ò¡/@\0\0\0\0Rí\a@³æÿ¿Ø"... cl_list = 0x6d2f6c71 <Address 0x6d2f6c71 out of bounds> sh_cmd = 0x73796d2f <Address 0x73796d2f out of bounds> match = -1 saved_context = {file = 0x0, line = 0} #3 0x4007b27c in hosts_access (request=0xbfffe860) at hosts_access.c:119 verdict = 0 #4 0x0811013f in handle_connections_sockets ()
Oh... you might wonder why there is an extra parameter named "last", it's because we suspected the static char * in get_fields() to be not thread safe. However this has no impact on the "string" parameter.
Signal 11's that are only reproducible on 1 machine sound like a hardware issue. I run mysql on a SMP machine and have yet to have a problem like this. I also do not have a hosts.allow or hosts.deny... Here's Signal 11 info... http://www.bitwizard.nl/sig11/
the tcpwrappers flaw was fixed elsewhere, and I can't reproduce this on my dual Xeons at work.
