After enabling full Journal mode for the Ext3 partitions (using tune2fs -O has_journal -o journal_data /dev/sdXY) and running some IO intensive operations, the running program will usually segfault at random intervals. Reproducible: Sometimes Steps to Reproduce: 1.Enable Full Journal mode on the Ext3 partitions. 2.Reboot the machine. 3.Start some IO intensive operations, such as emerging a large number of packages, starting a BitTorrent download. Actual Results: kernel BUG at mm/filemap.c:124! invalid opcode: 0000 [#1] PREEMPT SMP Modules linked in: snd_seq_midi snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_event snd_seq_midi_emul snd_seq w83627hf hwmon_vid hwmon nvidia(P) snd_emu10k1 snd_rawmidi snd_ac97_codec ac97_bus snd_pcm snd_seq_device snd_timer snd_page_alloc snd_util_mem snd_hwdep snd i2c_i801 CPU: 1 EIP: 0060:[<c043be56>] Tainted: P VLI EFLAGS: 00210046 (2.6.23-hardened-r4 #6) EIP is at __remove_from_page_cache+0x2e/0x35 eax: fffffff0 ebx: f4add350 ecx: c1c0dea0 edx: 00000010 esi: c1a9daa0 edi: 00000080 ebp: 00000001 esp: f67a5e54 ds: 0068 es: 0068 fs: 00d8 gs: 0033 ss: 0068 Process emerge (pid: 28811, ti=f67a4000 task=f70a4550 task.ti=f67a4000) Stack: f4add360 c1a9daa0 c043be7f c1a9daa0 00000000 c0442a5d c1a9daa0 c0442b7f 00001000 00000000 00000000 f389b240 ffffffff 00000000 f4add350 0000000e 00000000 c1af1540 c1a9daa0 c1aae280 c1a86c40 c1b28aa0 c1a995c0 c1a6fc20 Call Trace: [<c043be7f>] remove_from_page_cache+0x22/0x2b [<c0442a5d>] truncate_complete_page+0x2b/0x3f [<c0442b7f>] truncate_inode_pages_range+0xdf/0x2ba [<c04928a3>] ext3_delete_inode+0x0/0xc1 [<c0442d71>] truncate_inode_pages+0x17/0x1a [<c04928b7>] ext3_delete_inode+0x14/0xc1 [<c04928a3>] ext3_delete_inode+0x0/0xc1 [<c046c0a0>] generic_delete_inode+0x5e/0xc6 [<c046c295>] iput+0x5b/0x5d [<c0463c66>] do_unlinkat+0x13d/0x180 [<c04033f8>] restore_all+0x0/0x18 [<c04033d2>] syscall_call+0x7/0xb ======================= Code: 53 8b 58 10 8b 50 14 8d 43 04 e8 09 ec 0a 00 c7 46 10 00 00 00 00 ba 05 00 00 00 ff 4b 38 89 f0 e8 fc 94 00 00 83 7e 08 00 78 04 <0f> 0b eb fe 5b 5e c3 56 89 c6 53 8b 50 10 8b 00 a8 01 75 04 0f EIP: [<c043be56>] __remove_from_page_cache+0x2e/0x35 SS:ESP 0068:f67a5e54 note: emerge[28811] exited with preempt_count 1 Expected Results: Emerge should have been successful and no segfault should have occurred. I am running a Hardened profile but I do not think the problem is Hardened-related. I also found the following which suggests a possible fix: http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg243670.html My emerge --info Portage 2.1.3.19 (hardened/x86/2.6, gcc-3.4.6, glibc-2.6.1-r0, 2.6.23-hardened-r4 i686) ================================================================= System uname: 2.6.23-hardened-r4 i686 Intel(R) Pentium(R) 4 CPU 2.60GHz Timestamp of tree: Fri, 01 Feb 2008 23:30:04 +0000 ccache version 2.4 [enabled] app-shells/bash: 3.2_p17-r1 dev-java/java-config: 1.3.7, 2.1.4 dev-lang/python: 2.4.4-r6 dev-python/pycrypto: 2.0.1-r6 dev-util/ccache: 2.4-r7 sys-apps/baselayout: 1.12.10-r5 sys-apps/sandbox: 1.2.18.1-r2 sys-devel/autoconf: 2.13, 2.61-r1 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10 sys-devel/binutils: 2.18-r1 sys-devel/gcc-config: 1.4.0-r4 sys-devel/libtool: 1.5.24 virtual/os-headers: 2.6.23-r3 ACCEPT_KEYWORDS="x86" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -march=pentium4 -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc" CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/terminfo /etc/texmf/web2c /etc/udev/rules.d" CXXFLAGS="-O2 -march=pentium4 -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="ccache distlocks metadata-transfer sandbox sfperms strict unmerge-orphans userfetch" GENTOO_MIRRORS="http://distfiles.gentoo.org http://distro.ibiblio.org/pub/linux/distributions/gentoo" LANG="en_US.UTF-8" LINGUAS="en ja" MAKEOPTS="-j2" PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --filter=H_**/files/digest-*" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/portage/local/layman/enlightenment /usr/portage/local/layman/enlightenment /usr/local/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="X a52 aac acpi alsa apache2 bash-completion berkdb bzip2 cairo cdr cjk cracklib crypt dbus dvd encode gpgme gtk gtk2 hal hardened iconv jpeg lzo mad mailwrapper matroska midi mmx mp3 mysql ncurses nls nptl nptlonly ogg opengl pam pcre pic png python quicktime rar readline rtc ruby sdl skey sse sse2 ssl startup-notification suhosin svg tcpd theora threads truetype unicode urandom usb vorbis win32codecs x264 x86 xml xorg xv xvid zip zlib" ALSA_CARDS="emu10k1" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="mouse keyboard" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en ja" USERLAND="GNU" VIDEO_CARDS="nv" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LDFLAGS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Moving the call to cancel_dirty_page() so that it is after the do_invalidatepage() seem to fix the problem. A diff between the original file and the one with the changes follows (taken from http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg243670.html): diff --git a/mm/truncate.c b/mm/truncate.c index cadc156..2974903 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -98,11 +98,11 @@ truncate_complete_page(struct address_space *mapping, struct page *page) if (page->mapping != mapping) return; - cancel_dirty_page(page, PAGE_CACHE_SIZE); - if (PagePrivate(page)) do_invalidatepage(page, 0); + cancel_dirty_page(page, PAGE_CACHE_SIZE); + remove_from_page_cache(page); ClearPageUptodate(page); ClearPageMappedToDisk(page); I have emerged a number of packages and did a number of IO intensive operations without noticing any errors.
This may be fixed in 2.6.24. Zaid, would you care to test vanilla-sources-2.6.24 to confirm? The function is as follows in the 2.6.24 sources: --- void __remove_from_page_cache(struct page *page) { struct address_space *mapping = page->mapping; radix_tree_delete(&mapping->page_tree, page->index); page->mapping = NULL; mapping->nrpages--; __dec_zone_page_state(page, NR_FILE_PAGES); BUG_ON(page_mapped(page)); /* * Some filesystems seem to re-dirty the page even after * the VM has canceled the dirty bit (eg ext3 journaling). * * Fix it up by doing a final dirty accounting check after * having removed the page entirely. */ if (PageDirty(page) && mapping_cap_account_dirty(mapping)) { dec_zone_page_state(page, NR_FILE_DIRTY); dec_bdi_stat(mapping->backing_dev_info, BDI_RECLAIMABLE); } } --- I can't see any sign of a pending fix in the stable queue (for 2.6.23 at least). Some questions I have are: 1) Does the fix for <=2.6.23 appear to be suitable? 2) If not, then is a patch available that rectifies the bug in a suitable manner? 3) Will a suitable patch be incorporated into the stable queue and will a new stable release be forthcoming in a timely fashion? 4) If not then will genpatches incorporate a patch in the meantime?
I wanted to do one last test with 2.6.23 before testing 2.6.24 so I reverted the changes that I did to mm/trucate.c and started an emerge job. The job did not segfault but, I waited until it was done, and no matter how many times I sync'ed, the nr_dirty would not reach 0 (it is actually never going below 35 as we can see from the following): cat /proc/vmstat |grep dirty nr_dirty 35 When I ran with the modified 2.6.23, if I issued a sync then it would go to zero so it is clear that the vanilla 2.6.23 does not act in the same way. I grabbed the latest vanilla 2.6.24 sources from kernel.org and will be compiling them shortly.
I just tested with Vanilla 2.6.24 and I do not see the problem. I ran a few md5sum's on 700 Mb+ files, re-emerged a few packages, did emerge --sync and emerged another package then did `sync` and nr_dirty went to 0. I am back to the unmodified 2.6.23 but I disabled full journaling (left it with journal_data_ordered) and I'll see if the problem surfaces in this case as well.
OK. Changing the URL to that of the bug entry in upstream's bugzilla. I've also submitted the patch for inclusion in the stable queue. Let's see if they approve. I'll post back here if I have any news. Note also that 2.6.20-rc2 >< 2.6.23.14 are affected at the time of writing.
Response from upstream: On Sun 03-02-08 23:29:12, Kerin Millar wrote: > I am seeking approval for Björn's patch to be added to the stable > queue in response to a downstream bug report: > http://bugs.gentoo.org/show_bug.cgi?id=20869 which, upon further > investigation, I discovered to be the topic of kernel bug 9182. The > attachment is UTF-8 encoded with no BOM. Yes, it makes sence to fix this in stable kernels and the patch is fine. Honza -- Jan Kara <jack@suse.cz> SUSE Labs, CR Kernel team, please add and retain in genpatches for 2.6.20 - 2.6.23 branches (until such time as it is incorporated in a stable release for a given branch _and_ genpatches is updated to follow said stable release).
Added to 2.6.23 stable queue: http://tinyurl.com/35rqzs Upstream don't usually care about anything older than one release prior to the current version marked stable so I don't think it will go into the stable queues for older versions. In view of this, please remember to add it to any older genpatches branches that are still being used by kernel ebuilds in portage.
Thanks, the 2.6.23 review cycle is underway so this will be included automatically in the next release
Thanks Daniel, that's good news. Unfortunately, this does not appear to be the end of the matter. The reporter, Zaid, left me a message via IRC reporting what appears to be the same bug in 2.6.24 (vanilla). I responded, suggesting that he append a report here and that he might also try to reproduce it with an untainted kernel (to avoid issues if it needs to go upstream). I suspect that he may not have seen my response so I'm taking the liberty to paste the report here on his behalf (see below). How should we proceed? ------------[ cut here ]------------ kernel BUG at mm/filemap.c:126! invalid opcode: 0000 [#1] PREEMPT SMP Modules linked in: snd_seq_midi snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_event snd_seq_midi_emul snd_seq w83627hf hwmon_vid nvidia(P) snd_emu10k1 snd_rawmidi snd_ac97_codec ac97_bus snd_pcm snd_seq_device snd_timer snd_page_alloc snd_util_mem snd_hwdep snd Pid: 921, comm: enlightenment Tainted: P (2.6.24 #1) EIP: 0060:[<c0845998>] EFLAGS: 00210046 CPU: 0 EIP is at __remove_from_page_cache+0x8c/0x90 EAX: 00000000 EBX: c1ae3920 ECX: 00000002 EDX: 00000010 ESI: dbdea350 EDI: 00000079 EBP: ffffffff ESP: f7771d44 DS: 0068 ES: 0068 FS: 00d8 GS: 0000 SS: 0068 Process enlightenment (pid: 921, ti=f7770000 task=daaecff0 task.ti=f7770000) Stack: dbdea37c dbdea360 c1ae3920 c08459ba c1ae3920 00000009 c084e85d c1ae3920 c084e9ef 00001000 00000000 00000000 f7de3800 00000000 00000000 dbdea350 0000000e 00000000 c1ae7ee0 c1ae7f40 c1ae7fa0 c1ad8060 c1ad80c0 c1ad8120 Call Trace: [<c08459ba>] remove_from_page_cache+0x1e/0x2b [<c084e85d>] truncate_complete_page+0x24/0x41 [<c084e9ef>] truncate_inode_pages_range+0x12c/0x30d [<c08a85cb>] ext3_delete_inode+0x0/0x136 [<c084ebe7>] truncate_inode_pages+0x17/0x1b [<c08a85ea>] ext3_delete_inode+0x1f/0x136 [<c08a85cb>] ext3_delete_inode+0x0/0x136 [<c087c74d>] generic_delete_inode+0x5f/0xc7 [<c087c961>] iput+0x53/0x5f [<c08797fb>] d_kill+0x36/0x52 [<c0879884>] dput+0x6d/0xe7 [<c086a198>] __fput+0x104/0x16a [<c0857433>] remove_vma+0x39/0x54 [<c08598b8>] exit_mmap+0xcf/0x118 [<c081c01e>] mmput+0x33/0x94 [<c086d2a9>] exec_mmap+0x117/0x308 [<c086d948>] flush_old_exec+0x56/0x20c [<c086d180>] kernel_read+0x50/0x62 [<c08956f6>] load_elf_binary+0x391/0xf5d [<c085662e>] handle_mm_fault+0x2ab/0x34f [<c08529ff>] kunmap_high+0x59/0x9b [<c0895365>] load_elf_binary+0x0/0xf5d [<c086ddff>] search_binary_handler+0xb2/0x21f [<c086e092>] do_execve+0x126/0x17e [<c08017b5>] sys_execve+0x2f/0x82 [<c0802d52>] syscall_call+0x7/0xb [<c0a60000>] __pfkey_xfrm_state2msg+0x717/0xb42 ======================= Code: 75 05 ba ff ff ff ff 8d 14 d5 08 00 00 00 83 c0 1c 89 14 24 b9 ff ff ff ff ba ff ff ff ff e8 1a b0 0c 00 53 9d 83 c4 04 5b 5e c3 <0f> 0b eb fe 56 89 c6 53 8b 50 10 8b 00 a8 01 74 1a 8d 5a 10 89 EIP: [<c0845998>] __remove_from_page_cache+0x8c/0x90 SS:ESP 0068:f7771d44 ---[ end trace 1ac82b2ecbf36146 ]--- note: enlightenment[921] exited with preempt_count 1
Thanks Kerin for your previous post with the error log. It seems that my machine locked up overnight so I didn't get any messages on IRC. The kernel that I am running is a Vanilla 2.6.24 but it is patched with the latest Pax patch (test11) from Grsecurity and I also have a Proprietary module loaded (nvidia). I will compile a fresh vanilla 2.6.24 with no patches and no nvidia module loaded then leave it running to see if the issue resurfaces. Note: The machine ran without any issues for the last 2 days so I'm hoping we'll see the issue in a couple of days as well.
The fix was incorporated into 2.6.23.15 and, subsequently, the genpatches 2.6.23-8 patchset.
Looks like this isn't end of story yet - I've same bug on 2.6.36-hardened-r9. I'm also using ext3 with journaling (but it was formated using default mkfs.ext3, I didn't used tune2fs), and I also see this bug usually for emerge process while installing a lot of packages (also sometimes bug triggered by revdep-rebuild). Here is yesterday's logs while doing `emerge -ke world`: First: 2011-03-05_12:54:35.70103 <0>------------[ cut here ]------------ 2011-03-05_12:54:35.70106 kern.crit: kernel BUG at mm/filemap.c:128! 2011-03-05_12:54:35.70107 <0>invalid opcode: 0000 [#1] SMP 2011-03-05_12:54:35.70108 <0>last sysfs file: /sys/devices/virtual/vtconsole/vtcon0/uevent 2011-03-05_12:54:35.70109 kern.warn: Modules linked in: act_police cls_fw cls_u32 sch_ingress sch_tbf sch_sfq sch_prio sch_cbq sch_htb nvidia(P) vmnet vmblock vmci vmmon sky2 8139too skge 2011-03-05_12:54:35.70110 kern.warn: 2011-03-05_12:54:35.70111 kern.warn: Pid: 11233, comm: emerge Tainted: P 2.6.36-hardened-r9 #2 P5B-Deluxe/System Product Name 2011-03-05_12:54:35.70112 kern.warn: EIP: 0060:[<c10743be>] EFLAGS: 00210002 CPU: 0 2011-03-05_12:54:35.70112 kern.warn: EAX: 00000001 EBX: c366cce0 ECX: 00000018 EDX: 00000009 2011-03-05_12:54:35.70113 kern.warn: ESI: eac929c8 EDI: 0000000e EBP: dfcefd18 ESP: dfcefd0c 2011-03-05_12:54:35.70114 kern.warn: DS: 0068 ES: 0068 FS: 00d8 GS: 00e0 SS: 0068 2011-03-05_12:54:35.70115 <0>Process emerge (pid: 11233, ti=dfcee000 task=e79c5410 task.ti=dfcee000) 2011-03-05_12:54:35.70116 <0>Stack: 2011-03-05_12:54:35.70116 kern.warn: 0000000d c366cce0 eac929c8 dfcefd28 c10743f7 c366cce0 eac929c8 dfcefd44 2011-03-05_12:54:35.70117 kern.warn: <0> c107bfce 00001000 00000000 00000000 c366cce0 0000000d dfcefdb0 c107c0fa 2011-03-05_12:54:35.70118 kern.warn: <0> 0000000e 00000000 00000000 00000000 eac929c8 ffffffff 0000000e 00000000 2011-03-05_12:54:35.70122 <0>Call Trace: 2011-03-05_12:54:35.70122 kern.warn: [<c10743f7>] ? remove_from_page_cache+0x27/0x40 2011-03-05_12:54:35.70123 kern.warn: [<c107bfce>] ? truncate_inode_page+0x7e/0xc0 2011-03-05_12:54:35.70124 kern.warn: [<c107c0fa>] ? truncate_inode_pages_range+0xea/0x2b0 2011-03-05_12:54:35.70125 kern.warn: [<c107c2da>] ? truncate_inode_pages+0x1a/0x20 2011-03-05_12:54:35.70126 kern.warn: [<c1120085>] ? ext3_evict_inode+0x35/0x170 2011-03-05_12:54:35.70126 kern.warn: [<c10b439a>] ? evict+0x1a/0xa0 2011-03-05_12:54:35.70127 kern.warn: [<c10b4c49>] ? iput+0x159/0x240 2011-03-05_12:54:35.70128 kern.warn: [<c10b1317>] ? dentry_iput+0x67/0xb0 2011-03-05_12:54:35.70129 kern.warn: [<c10b1415>] ? d_kill+0x35/0x60 2011-03-05_12:54:35.70129 kern.warn: [<c10b236e>] ? dput+0x6e/0x140 2011-03-05_12:54:35.70130 kern.warn: [<c10ab570>] ? sys_renameat+0x1f0/0x2c0 2011-03-05_12:54:35.70131 kern.warn: [<c10a2868>] ? cp_new_stat64+0x118/0x170 2011-03-05_12:54:35.70132 kern.warn: [<c10ab668>] ? sys_rename+0x28/0x30 2011-03-05_12:54:35.70133 kern.warn: [<c13f4edc>] ? syscall_call+0x7/0xb 2011-03-05_12:54:35.70133 kern.warn: [<c13f0000>] ? snd_emu10k1_init_efx+0x1a49/0x27ee 2011-03-05_12:54:35.70134 kern.warn: [<c1090000>] ? do_mremap+0x270/0x460 2011-03-05_12:54:35.70135 kern.warn: [<c13f4ef5>] ? restore_all+0x0/0x18 2011-03-05_12:54:35.70136 <0>Code: d5 08 00 00 00 89 14 24 ba ff ff ff ff e8 cb eb 16 00 53 9d 83 c4 04 5b 5e c9 c3 66 90 ba 16 00 00 00 89 d8 e8 f4 ee 00 00 eb 89 <0f> 0b eb fe 8d b4 26 00 00 00 00 8d bc 27 00 00 00 00 55 89 e5 2011-03-05_12:54:35.70138 <0>EIP: [<c10743be>] __remove_from_page_cache+0xae/0xc0 SS:ESP 0068:dfcefd0c 2011-03-05_12:54:35.70138 kern.warn: ---[ end trace 8ee95cd55cf8c673 ]--- 2011-03-05_12:54:40.98304 <0>------------[ cut here ]------------ 2011-03-05_12:54:40.98307 kern.crit: kernel BUG at mm/filemap.c:128! 2011-03-05_12:54:40.98310 <0>invalid opcode: 0000 [#2] SMP 2011-03-05_12:54:40.98311 <0>last sysfs file: /sys/devices/virtual/vtconsole/vtcon0/uevent 2011-03-05_12:54:40.98312 kern.warn: Modules linked in: act_police cls_fw cls_u32 sch_ingress sch_tbf sch_sfq sch_prio sch_cbq sch_htb nvidia(P) vmnet vmblock vmci vmmon sky2 8139too skge 2011-03-05_12:54:40.98313 kern.warn: 2011-03-05_12:54:40.98313 kern.warn: Pid: 12665, comm: emerge Tainted: P D 2.6.36-hardened-r9 #2 P5B-Deluxe/System Product Name 2011-03-05_12:54:40.98314 kern.warn: EIP: 0060:[<c10743be>] EFLAGS: 00210002 CPU: 0 2011-03-05_12:54:40.98315 kern.warn: EAX: 00000034 EBX: c207e580 ECX: 00000018 EDX: 00000009 2011-03-05_12:54:40.98316 kern.warn: ESI: d2e5beb8 EDI: 0000007a EBP: ed925d28 ESP: ed925d1c 2011-03-05_12:54:40.98317 kern.warn: DS: 0068 ES: 0068 FS: 00d8 GS: 00e0 SS: 0068 2011-03-05_12:54:40.98317 <0>Process emerge (pid: 12665, ti=ed924000 task=ed8d3bd0 task.ti=ed924000) 2011-03-05_12:54:40.98318 <0>Stack: 2011-03-05_12:54:40.98319 kern.warn: 0000007a c207e580 d2e5beb8 ed925d38 c10743f7 c207e580 d2e5beb8 ed925d54 2011-03-05_12:54:40.98320 kern.warn: <0> c107bfce 00001000 00000000 00000000 c207e580 00000009 ed925dc0 c107c0fa 2011-03-05_12:54:40.98321 kern.warn: <0> 0000000e 00000000 00000000 00000000 d2e5beb8 ffffffff 0000000e 00000000 2011-03-05_12:54:40.98322 <0>Call Trace: 2011-03-05_12:54:40.98323 kern.warn: [<c10743f7>] ? remove_from_page_cache+0x27/0x40 2011-03-05_12:54:40.98324 kern.warn: [<c107bfce>] ? truncate_inode_page+0x7e/0xc0 2011-03-05_12:54:40.98325 kern.warn: [<c107c0fa>] ? truncate_inode_pages_range+0xea/0x2b0 2011-03-05_12:54:40.98325 kern.warn: [<c107c2da>] ? truncate_inode_pages+0x1a/0x20 2011-03-05_12:54:40.98326 kern.warn: [<c1120085>] ? ext3_evict_inode+0x35/0x170 2011-03-05_12:54:40.98327 kern.warn: [<c10b439a>] ? evict+0x1a/0xa0 2011-03-05_12:54:40.98328 kern.warn: [<c10b4c49>] ? iput+0x159/0x240 2011-03-05_12:54:40.98329 kern.warn: [<c10b1317>] ? dentry_iput+0x67/0xb0 2011-03-05_12:54:40.98329 kern.warn: [<c10b1415>] ? d_kill+0x35/0x60 2011-03-05_12:54:40.98331 kern.warn: [<c10b236e>] ? dput+0x6e/0x140 2011-03-05_12:54:40.98332 kern.warn: [<c10a0236>] ? fput+0x176/0x240 2011-03-05_12:54:40.98333 kern.warn: [<c108c2e5>] ? remove_vma+0x35/0x60 2011-03-05_12:54:40.98334 kern.warn: [<c108c40e>] ? exit_mmap+0xfe/0x140 2011-03-05_12:54:40.98334 kern.warn: [<c102fe16>] ? mmput+0x36/0xc0 2011-03-05_12:54:40.98335 kern.warn: [<c1033bbd>] ? exit_mm+0xdd/0x100 2011-03-05_12:54:40.98336 kern.warn: [<c1035cb4>] ? do_exit+0x604/0x6d0 2011-03-05_12:54:40.98337 kern.warn: [<c1035db9>] ? do_group_exit+0x39/0xa0 2011-03-05_12:54:40.98337 kern.warn: [<c1035e33>] ? sys_exit_group+0x13/0x20 2011-03-05_12:54:40.98338 kern.warn: [<c13f4edc>] ? syscall_call+0x7/0xb 2011-03-05_12:54:40.98339 <0>Code: d5 08 00 00 00 89 14 24 ba ff ff ff ff e8 cb eb 16 00 53 9d 83 c4 04 5b 5e c9 c3 66 90 ba 16 00 00 00 89 d8 e8 f4 ee 00 00 eb 89 <0f> 0b eb fe 8d b4 26 00 00 00 00 8d bc 27 00 00 00 00 55 89 e5 2011-03-05_12:54:40.98341 <0>EIP: [<c10743be>] __remove_from_page_cache+0xae/0xc0 SS:ESP 0068:ed925d1c 2011-03-05_12:54:40.98342 kern.warn: ---[ end trace 8ee95cd55cf8c674 ]--- 2011-03-05_12:54:40.98342 kern.alert: Fixing recursive fault but reboot is needed! Second: 2011-03-06_00:05:35.26533 <0>------------[ cut here ]------------ 2011-03-06_00:05:35.26537 kern.crit: kernel BUG at mm/filemap.c:128! 2011-03-06_00:05:35.26538 <0>invalid opcode: 0000 [#1] SMP 2011-03-06_00:05:35.26539 <0>last sysfs file: /sys/devices/virtual/vtconsole/vtcon0/uevent 2011-03-06_00:05:35.26542 kern.warn: Modules linked in: act_police cls_fw cls_u32 sch_ingress sch_tbf sch_sfq sch_prio sch_cbq sch_htb nvidia(P) vmnet vmblock vmci vmmon sky2 8139too skge 2011-03-06_00:05:35.26543 kern.warn: 2011-03-06_00:05:35.26544 kern.warn: Pid: 25574, comm: emerge Tainted: P 2.6.36-hardened-r9 #2 P5B-Deluxe/System Product Name 2011-03-06_00:05:35.26545 kern.warn: EIP: 0060:[<c10743be>] EFLAGS: 00210046 CPU: 1 2011-03-06_00:05:35.26546 kern.warn: EAX: 00000000 EBX: c239f540 ECX: 00000018 EDX: 00000009 2011-03-06_00:05:35.26547 kern.warn: ESI: f7374d68 EDI: 00000007 EBP: df8efd88 ESP: df8efd7c 2011-03-06_00:05:35.26548 kern.warn: DS: 0068 ES: 0068 FS: 00d8 GS: 00e0 SS: 0068 2011-03-06_00:05:35.26548 <0>Process emerge (pid: 25574, ti=df8ee000 task=ee1ba410 task.ti=df8ee000) 2011-03-06_00:05:35.26549 <0>Stack: 2011-03-06_00:05:35.26550 kern.warn: 00000006 c239f540 f7374d68 df8efd98 c10743f7 c239f540 f7374d68 df8efdb4 2011-03-06_00:05:35.26551 kern.warn: <0> c107bfce 00001000 00000000 00000000 c239f540 00000006 df8efe20 c107c0fa 2011-03-06_00:05:35.26552 kern.warn: <0> 0000000e 00000000 00000000 00000000 f7374d68 ffffffff 0000000e 00000000 2011-03-06_00:05:35.26553 <0>Call Trace: 2011-03-06_00:05:35.26553 kern.warn: [<c10743f7>] ? remove_from_page_cache+0x27/0x40 2011-03-06_00:05:35.26554 kern.warn: [<c107bfce>] ? truncate_inode_page+0x7e/0xc0 2011-03-06_00:05:35.26555 kern.warn: [<c107c0fa>] ? truncate_inode_pages_range+0xea/0x2b0 2011-03-06_00:05:35.26557 kern.warn: [<c107c2da>] ? truncate_inode_pages+0x1a/0x20 2011-03-06_00:05:35.26559 kern.warn: [<c1120085>] ? ext3_evict_inode+0x35/0x170 2011-03-06_00:05:35.26559 kern.warn: [<c10b439a>] ? evict+0x1a/0xa0 2011-03-06_00:05:35.26560 kern.warn: [<c10b4c49>] ? iput+0x159/0x240 2011-03-06_00:05:35.26561 kern.warn: [<c10b1317>] ? dentry_iput+0x67/0xb0 2011-03-06_00:05:35.26562 kern.warn: [<c10b1415>] ? d_kill+0x35/0x60 2011-03-06_00:05:35.26562 kern.warn: [<c10b236e>] ? dput+0x6e/0x140 2011-03-06_00:05:35.26563 kern.warn: [<c10ab570>] ? sys_renameat+0x1f0/0x2c0 2011-03-06_00:05:35.26564 kern.warn: [<c10a2e72>] ? sys_lstat64+0x22/0x30 2011-03-06_00:05:35.26565 kern.warn: [<c10ab668>] ? sys_rename+0x28/0x30 2011-03-06_00:05:35.26565 kern.warn: [<c13f4edc>] ? syscall_call+0x7/0xb 2011-03-06_00:05:35.26566 <0>Code: d5 08 00 00 00 89 14 24 ba ff ff ff ff e8 cb eb 16 00 53 9d 83 c4 04 5b 5e c9 c3 66 90 ba 16 00 00 00 89 d8 e8 f4 ee 00 00 eb 89 <0f> 0b eb fe 8d b4 26 00 00 00 00 8d bc 27 00 00 00 00 55 89 e5 2011-03-06_00:05:35.26567 <0>EIP: [<c10743be>] __remove_from_page_cache+0xae/0xc0 SS:ESP 0068:df8efd7c 2011-03-06_00:05:35.26568 kern.warn: ---[ end trace c704168e8575db2c ]---
Sorry, forgot about my emerge --info, not sure it relevant for this bug, but just in case: Portage 2.1.9.25 (hardened/linux/x86, gcc-4.4.5, glibc-2.11.3-r0, 2.6.36-hardened-r9 i686) ================================================================= System uname: Linux-2.6.36-hardened-r9-i686-Intel-R-_Core-TM-2_CPU_6600_@_2.40GHz-with-gentoo-1.12.14 Timestamp of tree: Sat, 05 Mar 2011 08:30:01 +0000 app-shells/bash: 4.1_p9 dev-java/java-config: 2.1.11-r3 dev-lang/python: 2.6.6-r2, 3.1.3-r1 dev-util/cmake: 2.8.1-r2 sys-apps/baselayout: 1.12.14-r1 sys-apps/sandbox: 2.4 sys-devel/autoconf: 2.13, 2.65-r1 sys-devel/automake: 1.9.6-r3, 1.10.3, 1.11.1 sys-devel/binutils: 2.20.1-r1 sys-devel/gcc: 4.4.5 sys-devel/gcc-config: 1.4.1 sys-devel/libtool: 2.2.10 sys-devel/make: 3.81-r2 virtual/os-headers: 2.6.36.1 (sys-kernel/linux-headers) ACCEPT_KEYWORDS="x86" ACCEPT_LICENSE="*" CBUILD="i686-pc-linux-gnu" CFLAGS="-march=prescott -O2 -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /service /usr/inferno/keydb /usr/inferno/lib /usr/inferno/services /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/config /usr/share/openvpn/easy-rsa /var/log /var/qmail/alias /var/qmail/control" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-march=prescott -O2 -pipe" DISTDIR="/usr/portage-distfiles" EMERGE_DEFAULT_OPTS="--with-bdeps=y" FEATURES="assume-digests binpkg-logs distlocks fixlafiles fixpackages news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox" GENTOO_MIRRORS="ftp://ftp.df.lth.se/pub/gentoo/ http://ftp.df.lth.se/pub/gentoo/ http://gentoo.telcom.net.ua/" LANG="ru_RU.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LINGUAS="en ru" MAKEOPTS="-j1" PKGDIR="/usr/portage-packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_EXTRA_OPTS="--exclude ChangeLog --delete-excluded" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/var/lib/layman/powerman /var/lib/layman/sunrise /var/lib/layman/kde-sunset /var/lib/layman/vmware /usr/local/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="X Xaw3d a52 aac acl acpi aim alsa apache2 asf avi bash-completion berkdb bitmap-fonts bzip2 cddb cdr chm cli cracklib crypt cscope cue curl cxx dbus dga divx4linux djvu dlloader dri dts dvd dvdr dvdread encode fastcgi ffmpeg flac flash gd gdbm gif gnutls gpg gtk gtk2 hardened hddtemp iconv icq idn imagemagick imap imlib irc jabber javascript jpeg kde lm_sensors lzo mad mailbox mbox mmx mng modules motif mp3 mpeg msn mudflap musepack mysql ncurses nls nptl nptlonly ogg opengl openmp oss pam pcre perl pic png pppd pwdb python qt qt3support qt4 quicktime readline rss rtc samba sdl session spell sse sse2 ssl ssse3 svg sysfs tcltk tcpd theora tiff truetype truetype-fonts type1-fonts unicode urandom vim-pager vim-syntax vim-with-x vorbis wavpack win32codecs x86 xinetd xorg xv xvid yahoo zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="log_config vhost_alias autoindex alias rewrite dir deflate filter mime negotiation auth_basic authn_file authz_host authz_user authz_groupfile cgi actions headers env setenvif" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en ru" LIRC_DEVICES="serial" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="vesa fbdev nv nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, FFLAGS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS
This bug ran its course, concerning - as it did - a specific regression in a specific release and relating to a specific upstream bug. Under the circumstances, I'd suggest filing a new bug - even though the symptoms appear similar. I found a comparitively recent discussion from the LKML which is, unfortunately, inconclusive: http://www.gossamer-threads.com/lists/linux/kernel/1309854 If anyone can fix it, Hugh Dickin can, so it should really be taken upstream. Try to reproduce in the latest stable release beforehand though - that's 2.6.37.2 at the time of writing.
Hmm, there's a patch in the 6th comment in the aforementioned thread. You could try that also.
