We have a system set to create coredumps on process crashes which is usually fine and very helpful for debugging purposes. But since one of the last updates to the kernel we are seeing a complete kernel crash if a multiprocess server like apache dies with SIGABRT (this sometimes happens due to unknwon reasons, but this is not what this bug report is about): This crash creates a huge amount of coredumps on a reiserfs partition (about 50, each 5-15 MB in size) in a very short time. Most of the times this leads to a crash of the kernel. The backtrace on the screen isn't very helpful because it mostly contains "?????????" symbols and feels like one mile long. It seems to be possible to crash the kernel by creating a huge amount of write requests in a very short time on reiserfs - not sure if it can be triggered by normal user-space alone or if the coredump writer must be involved. The problem is: The server is running (and crashing) at a remote site, not accessable physically and runs in production so I cannot force trying reproducing the problem on it. There is also no local test system with the same configuration. But we need a fix since the whole problem chain can be triggered from user space (creating huge amounts of coredumps). Temporary we will turn off coredumps. Reproducible: Always Steps to Reproduce: 1. Make kernel create coredumps on reiserfs partition 2. Force creation of many (big) coredumps at the same time 3. Kernel crashes with unreadable backtrace (only "????????" symbols), sometimes followed by continous dumping a seemingly random characters Actual Results: Kernel crashes after a few seconds with unusable backtrace, sometimes followed by continous dumping a seemingly random characters. Expected Results: System should slow down only, thrashing hard disk, increasing load. Involved system setup: * Dual Xeon with HT CPU (2x2 cores) * SMP hardened kernel with grsecurity Involved process: Due to a bisbehaving (yet unidentified) apache module, on apache restart, all child processes may crash with SIGABRT and each process creates a coredump. One process (probably master) crashes with SIGSEGV. Apache version 2.2.16 with about 50-70 running children. The crashes can be simulated by sending SIGABRT to all apache processes at once. Involved coredump setup: kernel.core_pattern = /var/tmp/core.%p-%s-%e fs.suid_dumpable = 2 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.core.netdev_max_backlog = 2500 /etc/limits: * C20480 /etc/security/limits.conf: * soft core 40960
Let's get more information: 1) Can you give me an earlier hardened-sources version where this does not happen? If its not a pain, can you test the latest hardened-sources-2.6.32-r27. Does it happen with the corresponding vanilla 2.6.32.24. 2) Can you get me your emerge --info and kernel .config files. 3) Have you tested on ext4 or some other format? Is it definitely reiser only? 4) Can you give me clear steps to reproduce the problem. It sounds like the coredumping precipitates the problem but is not the immediate cause. Can you reproduce the same kernel crash by say, writing a script to produce many small files? 5) I'm going to set up a vm as close to your system as possible and try to reproduce here. Anything else you think I need to get a similar situation here?
Created attachment 254717 [details] .config of running kernel (from /proc/config.gz) This is my kernel config of the server having the problem.
Created attachment 254719 [details] emerge --info Dump of "emerge --info"
(In reply to comment #1) > 1) Can you give me an earlier hardened-sources version where this does not > happen? If its not a pain, can you test the latest > hardened-sources-2.6.32-r27. Does it happen with the corresponding vanilla > 2.6.32.24. The last version which didn't have stability problems was: Tue Mar 3 18:36:12 2009 >>> sys-kernel/hardened-sources-2.6.27-r8 We already had some problems with stability in: Sun May 17 13:49:19 2009 >>> sys-kernel/hardened-sources-2.6.28-r7 Sun May 31 18:38:26 2009 >>> sys-kernel/hardened-sources-2.6.28-r9 Wed Jul 21 11:37:24 2010 >>> sys-kernel/hardened-sources-2.6.32-r9 Wed Jul 21 20:18:24 2010 >>> sys-kernel/hardened-sources-2.6.32-r9 Thu Sep 2 23:52:22 2010 >>> sys-kernel/hardened-sources-2.6.32-r9 I'm not sure if these kernels already had the same problems - we weren't able to reduce it to the mass-creation of coredumps at that time. Stability, however, slightly improved from 2.6.28 to 2.6.32 - but again: Probably there were other factors involved, one was that 2.6.32 was compiled with gcc 3.4.6 (from the old-stable hardened toolchain) and that isn't really supported by Gentoo, so I switched to gcc 4.3.4 and stability improved. But we were having occasional freezes since 2.6.28 which we had no clue for until lately when we discovered that forcing mass-creation of coredumps forced the freeze/crash in maybe >80% of trials. The system is pretty loaded (constantly above 2.5, with peaks up to 20-25, and mean load around 4-5). > 2) Can you get me your emerge --info and kernel .config files. Attached. Thanks for investigation. > 3) Have you tested on ext4 or some other format? Is it definitely reiser only? No, I cannot do that - it's a production system. But I'll try to reproduce the same system configuration in a VM and see if I can force the crash there. The system consists of only XFS and ReiserFS - no ext[34] support used. Since I'm overdue with some projects, an implementation of a testing environment will take a while. > 4) Can you give me clear steps to reproduce the problem. It sounds like the > coredumping precipitates the problem but is not the immediate cause. Can you > reproduce the same kernel crash by say, writing a script to produce many small > files? The system is already pretty loaded with constant IO - the harddisks are never idle. It constantly creates small files and unlinks them. I don't think that could easily trigger the crash (although that could explain the instability we had during the last weeks and months). However, you can easily trigger the bug by forcing crashes of many processes at the same time. Here's our setup: * Apache 2.2.16 with mod_php, mod_passenger (3.0.0), mod_itk, mod_disk_cache, mod_rewrite (and probably some more) * Our apache runs with constantly more than 20 processes, usually around 50, eg currently: $ sudo pgrep apache2 | wc -l 114 * Sending "killall -ABRT apache2" triggers the bug, given the coredump configuration mentioned in the OP. Our apache processes usually receive SIGABRT on "/etc/apache2 reload" probably due to a bug in mod_passenger. The system starts to create coredumps in /var/tmp and after a few seconds: BANG! No more ping, no more ssh. When the remote site support reaches our server it usually scrolls seemingly random characters around the screen or is frozen on a backtrace with mostly "????????" symbols - so unusable. After reboot, the system replays journal of reiserfs and a reiserfsck yields no defects. I'm watching this increasing instability since removal of the BKL touches more and more of the VFS layers in the kernel (but we also see increasing performance on this system due to this). > 5) I'm going to set up a vm as close to your system as possible and try to > reproduce here. Anything else you think I need to get a similar situation > here? Probably the above is all you need. You could tell apache to spawn at least 100 child processes, using a non-threaded mpm. mod_passenger may be involved. I can give more hints after I've set up a VM myself which can trigger the problem. PM me if you'd think you could use a SSH shell to the server. Or just leave a note here if you need further configuration info. Thanks for you efforts.
This time the crash seemed to happen while doing a recursive chown on a directory with many sub directories and files (an apache document root with multiple php applications installed). However, I'm not sure if this was an incident or the trigger. This directory is also stored on reiserfs, but different partition the the coredump files.
(In reply to comment #5) > This time the crash seemed to happen while doing a recursive chown on a > directory with many sub directories and files (an apache document root with > multiple php applications installed). However, I'm not sure if this was an > incident or the trigger. This directory is also stored on reiserfs, but > different partition the the coredump files. > I've tried to reproduce this in a vm, but I have not succeeded. There are some differences between your system and the vm, primarily the emulated devices. Also, I used ext4 as my root, while using XFS and ReiserFS partitions for testing. Unfortunately yours is a production system, so we can't experiment otherwise I'd suggest switching to a vanilla kernel of the same level and seeing if the problem isn't userland eg. how your world target was compiled. Have you been able to reproduce this on any other system that you *can* experiment with?
(In reply to comment #6) > Unfortunately yours is a production system, so we can't experiment otherwise > I'd suggest switching to a vanilla kernel of the same level and seeing if the > problem isn't userland eg. how your world target was compiled. > > Have you been able to reproduce this on any other system that you *can* > experiment with? Due to instability we are building a new system now and clone the existing system into a VM with only one monolithic XFS filesystem. If the problem still persists, we have at least a way to simply reboot the machine without a service technican physically accessing the machine (which he did already 10 times now, and that reaches the price for new hardware). After a grace period the left hardware can be used for testing purpose. I could offer you SSH access then, after disabling outbound communication (like mails etc). This will take around 10-14 weekdays.
I've cloned the filesystem of the failing server into an ESXi VM (using rsync) which consists of only one monolithic XFS partition (except /boot which is still reiserfs). It looks like the problem is now gone. So I strongly suspect reiserfs being the culprit. The new hardware is of similar specs: Still 4 cores but newer Xeon processors (Core2 based). The kernel was reconfigured accordingly. RAM has been increased from 7 GB physically to 8 GB virtually. The system also performs much better now (at least 3 times the IOPS, load stays below 1).
I've never been able to reproduce this with reiserfs. If this is still an issue, please reopen.
