Our database server randomly crashes when the load raises. i don't have the complete panic log, only the provided screen shot from our KVM switch. We're running postgres 8.0.3 on this server; no X is installed. unfortunally this is our production server so we're very unhappy with this situation. any suggestion or hints to stop this random crashes will be greatfully accepted! i try to add some infos here, if you need more, please request and i'll provide all you need. Memory: total used free shared buffers cached Mem: 16281964 2217164 14064800 0 61196 1846136 -/+ buffers/cache: 309832 15972132 Swap: 31254416 0 31254416 Portage 2.0.54 (default-linux/amd64/2005.1, gcc-3.4.4, glibc-2.3.5-r2, 2.6.15-gentoo-r5 x86_64) ================================================================= System uname: 2.6.15-gentoo-r5 x86_64 AMD Opteron(tm) Processor 275 Gentoo Base System version 1.6.14 dev-lang/python: 2.3.5-r2, 2.4.2 sys-apps/sandbox: 1.2.12 sys-devel/autoconf: 2.13, 2.59-r6 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r1 sys-devel/binutils: 2.16.1 sys-devel/libtool: 1.5.22 virtual/os-headers: 2.6.11-r2 ACCEPT_KEYWORDS="amd64" AUTOCLEAN="yes" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=k8 -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3/share/config /usr/share/config /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d" CXXFLAGS="-march=k8 -O2 -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="autoconfig distlocks sandbox sfperms strict" GENTOO_MIRRORS="http://gentoo.osuosl.org/ ftp://gentoo.risq.qc.ca/ http://gentoo.ccccom.com ftp://gentoo.ccccom.com http://gentoo.inode.at/ http://gd.tuwien.ac.at/opsys/linux/gentoo/ ftp://gd.tuwien.ac.at/opsys/linux/gentoo/ http://ftp.belnet.be/mirror/rsync.gentoo.org/gentoo/ ftp://ftp.tu-clausthal.de/pub/linux/gentoo/ ftp://sunsite.informatik.rwth-aachen.de/pub/Linux/gentoo http://linux.rz.ruhr-uni-bochum.de/download/gentoo-mirror/ ftp://linux.rz.ruhr-uni-bochum.de/gentoo-mirror/ http://ftp.uni-erlangen.de/pub/mirrors/gentoo ftp://ftp.uni-erlangen.de/pub/mirrors/gentoo http://ftp6.uni-erlangen.de/pub/mirrors/gentoo ftp://ftp6.uni-erlangen.de/pub/mirrors/gentoo ftp://ftp.join.uni-muenster.de/pub/linux/distributions/gentoo ftp://ftp.wh2.tu-dresden.de/pub/mirrors/gentoo ftp://ftp.join.uni-muenster.de/pub/linux/distributions/gentoo ftp://ftp6.uni-muenster.de/pub/linux/distributions/gentoo ftp://ftp.ipv6.uni-muenster.de/pub/linux/distributions/gentoo http://mirrors.sec.informatik.tu-darmstadt.de/gentoo/ http://ftp-stud.fht-esslingen.de/pub/Mirrors/gentoo/ ftp://ftp-stud.fht-esslingen.de/pub/Mirrors/gentoo/ ftp://ftp.gentoo.mesh-solutions.com/gentoo/ http://pandemonium.tiscali.de/pub/gentoo/ ftp://pandemonium.tiscali.de/pub/gentoo/ ftp://ftp.rz.tu-bs.de/pub/mirror/ftp.gentoo.org/gentoo-distfiles/" MAKEOPTS="-j5" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="amd64 apache2 avi bash-completion berkdb bitmap-fonts bzip2 cal caps crypt cups curl dba dbase dbm dbx eds emboss encode exif expat fftw flash flatfile foomaticdb fortran gd gdbm gif gmp gnutls gpm gstreamer iconv imagemagick imap imlib innodb ipv6 jabber java jikes jpeg ldap libwww lm_sensors lzw lzw-tiff m17n-lib mcal mhash ming mmap mng mp3 mpeg ncurses netcdf nls nptl odbc ogg openal opengl pam pcntl pcre pdflib perl php png posix postgres prelude python quicktime readline recode ruby sasl sdl sharedext sharedmem shorten simplexml sndfile snmp soap sockets sox speex spell spl ssl svg sysvipc tcpd theora tidy tiff tokenizer truetype truetype-fonts type1-fonts udev unicode usb userlocales vhosts vorbis wddx wmf xml xml2 xmlrpc xpm xsl xv xvid yaz yeo zlib userland_GNU kernel_linux elibc_glibc" Unset: ASFLAGS, CTARGET, LANG, LC_ALL, LDFLAGS, LINGUAS, PORTDIR_OVERLAY
Created attachment 80375 [details] kernel config
Created attachment 80376 [details] kernel panic screen shot
Created attachment 80377 [details] "sysctl -a" output
Created attachment 80378 [details] /proc/cpuinfo
Created attachment 80379 [details] kernel config
Created attachment 80381 [details] "sysctl -a" output
Created attachment 80382 [details] /proc/cpuinfo
Please enable CONFIG_KALLSYMS and post a new screenshot. It's almost impossible to diagnose this otherwise (kallsyms will add some useful text into those meaningless numbers). How often does the crash occur?
Created attachment 80404 [details] dmesg output
crashes occurs "usually" about every 3 weeks, but this week on monday morning and (after rebooting with 2.6.15.r5) tuesday evening. now i turned off swap. at the moment i recompile the kernel with CONFIG_KALLSYMS enabled but i cannot reboot before 8pm (GMT 7pm).
Ok. You should also upgrade to the latest development kernel (currently 2.6.16-rc4) as the problem may have been fixed. I'm going to close this bug for now as it sounds like we might be waiting weeks for a new crash screenshot. Please reopen when you do have one.
@Daniel: which sources do You mean? vanilla-sources-2.6.16-r4?
vanilla-sources-2.6.16-rc4
It crashed again. I'll attach some info.
Created attachment 87249 [details] console screen shot New screenshot, as requested the kernel was compiled with CONFIG_KALLSYMS=y.
Created attachment 87250 [details] stat output I log once a minute the output from utime an the content from /proc/meminfo and /proc/vmstat by a cron job. This is the last log before the system crashed.
Would it be possible to setup a serial console or netconsole to capture the full error message? You can find documentation on how to do so in Documentation/serial-console.txt and Documentation/networking/netconsole.txt, under your kernel source tree. It would also be a good idea to try with the latest vanilla sources (currently 2.6.21.1). If you can reproduce with that, and get the full error message, then you will have a much better chance of getting help from LKML (assuming we can't identify the problem here).
I do not admister this system any longer, so i cannot provide more information, sorry. So You may close the bug. But i had a very similar problem on my x86 notebook using an suspend2 kernel a while ago. I had random crashes every now and then after resuming with some USB devives (external mouse) not attached as they were before i suspended, but the system seemed to run ok after a real reboot. Some days later i tried to to emerge a new bash and the system crashed reproducible every time at the same point of compiling. The same occurs on other packages as well. The reason was a corrupt reiser3 /tmp file system (i got denied permissions on some files and dirs even as root). Repairing this (and by the way the other) reiser3 filesystems solved the problem. But i couldn't check it on the server; as i left the company the machine was running smooth for a couple of months (but without changing anything except rebooting a new kernel version after every crash).
OK. Thanks for the update. If your notebook still exhibits those problems on the latest version of a supported kernel (e.g. gentoo-sources) then please file a new bug.
