I'd post this to the kernel.org bugzilla but I see that the grsec patch makes lots of changes to do_page_fault(), and I can't find a grsec bugzilla. I saw the following crash recently, while the system was under pretty heavy load: an emerge of gcc, a java application with ~350 threads and moderate I/O, some nfs and smb accesses as well as the usual desktop apps. This is on a single core amd64 system with 4G of RAM. The "as" in the first GPF (there were 3) was presumably from the gcc compile. Also, according to syslog there was a ~30s gap between GPF 2 and 3. The system stayed up and running apps were responsive to commands, but any new exec would segfault. I did a few memtest passes to verify that the memory was OK (it was), and I'm trying to reproduce the problem at the moment. Regardless, the kernel shouldn't have crashed. Is there an upstream contact for grsec/pax? general protection fault: 0000 [1] PREEMPT CPU 0 Modules linked in: nls_iso8859_1 nfs nfsd lockd sunrpc exportfs radeon drm iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack ip_tables xt_tcpudp xt_owner ip6table_filter ip6_tables x_tables snd_seq_midi snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_emul snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss tun ftdi_sio usbserial usblp usb_storage it87 hwmon_vid cryptoloop rtc_cmos rtc_core usbhid rtc_lib hid snd_emu10k1 snd_rawmidi firmware_class snd_ac97_codec ac97_bus snd_pcm snd_seq_device snd_timer snd_page_alloc k8temp hwmon snd_util_mem snd_hwdep snd emu10k1_gp gameport soundcore sg ehci_hcd ohci_hcd i2c_nforce2 i2c_core fan thermal processor button Pid: 6690, comm: as Tainted: G W 2.6.26-hardened-r3 #1 RIP: 0010:[<ffffffff8025d2b3>] [<ffffffff8025d2b3>] __rmqueue_smallest+0xc3/0x1c0 RSP: 0000:ffff810008521b88 EFLAGS: 00010012 RAX: ffffffff80812af8 RBX: ffffffff80812ad8 RCX: ffffe20000816ab8 RDX: fff7e20002185e30 RSI: 0000000000000000 RDI: ffffffff80812a80 RBP: ffffe20001420348 R08: ffffe20001420370 R09: ff2002ffff2002ff R10: ff1001ffff1001ff R11: 0000000000000058 R12: 0000000000000000 R13: 0000000000000020 R14: 0000000000000000 R15: 0000000000000000 FS: 00002dfc12b66f50(0000) GS:ffffffff80840000(0000) knlGS:00000000ea5406d0 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00002dfc11e0e000 CR3: 00000000aebb5000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process as (pid: 6690, threadinfo ffff810008520000, task ffff81000845b9d0) Stack: ffffe20000816ab8 ffffffff80812a80 0000000000000002 0000000000000000 000000000000001f ffffffff8025d880 0000000000000001 0000000000000000 0000000000000000 ffffe20000816ab8 000000000000001a 0000000000000002 Call Trace: [<ffffffff8025d880>] ? __rmqueue+0x30/0x2d0 [<ffffffff8025dd02>] ? rmqueue_bulk+0x52/0xc0 [<ffffffff8025f72c>] ? get_page_from_freelist+0x1dc/0x5a0 [<ffffffff8025fbd7>] ? __alloc_pages_internal+0xe7/0x4a0 [<ffffffff80262ae4>] ? __pagevec_lru_add_active+0x104/0x130 [<ffffffff8026ab55>] ? handle_mm_fault+0x2e5/0x7e0 [<ffffffff80217d6b>] ? do_page_fault+0x45b/0xb20 [<ffffffff8026fb5c>] ? vma_merge+0x1bc/0x2a0 [<ffffffff80270780>] ? mmap_region+0x3f0/0x5e0 [<ffffffff80207227>] ? arch_get_unmapped_area+0xd7/0x280 [<ffffffff80270e15>] ? do_mmap_pgoff+0x4a5/0x500 [<ffffffff80548d6e>] ? thread_return+0x4c/0x2fe [<ffffffff8054b0d9>] ? error_exit+0x0/0x51 Code: 01 10 ff 49 0f af f3 49 c1 e5 04 49 b9 ff 02 20 ff ff 02 20 ff 49 8d 44 35 00 4c 8b 44 07 58 49 8d 68 d8 48 8b 55 28 48 8b 45 30 <48> 89 42 08 48 89 10 4c 89 4d 30 4c 89 55 28 41 0f ba 70 d8 12 RIP [<ffffffff8025d2b3>] __rmqueue_smallest+0xc3/0x1c0 RSP <ffff810008521b88> ---[ end trace 847e4d83bdf41d97 ]--- note: as[6690] exited with preempt_count 2 BUG: scheduling while atomic: as/6690/0x10000003 Pid: 6690, comm: as Tainted: G D W 2.6.26-hardened-r3 #1 Call Trace: [<ffffffff80548e40>] thread_return+0x11e/0x2fe [<ffffffff80221840>] __cond_resched+0x20/0x50 [<ffffffff805490e5>] _cond_resched+0x35/0x50 [<ffffffff80269dc5>] unmap_vmas+0x685/0x790 [<ffffffff8026e56d>] exit_mmap+0x6d/0x130 [<ffffffff80224245>] mmput+0x25/0xe0 [<ffffffff8022a96b>] do_exit+0x19b/0x7a0 [<ffffffff80220c32>] __wake_up_common+0x52/0x80 [<ffffffff802039f4>] oops_end+0x74/0x80 [<ffffffff8054b0d9>] error_exit+0x0/0x51 [<ffffffff8025d2b3>] __rmqueue_smallest+0xc3/0x1c0 [<ffffffff8025dd02>] rmqueue_bulk+0x52/0xc0 [<ffffffff8025d880>] __rmqueue+0x30/0x2d0 [<ffffffff8025dd02>] rmqueue_bulk+0x52/0xc0 [<ffffffff8025f72c>] get_page_from_freelist+0x1dc/0x5a0 [<ffffffff8025fbd7>] __alloc_pages_internal+0xe7/0x4a0 [<ffffffff80262ae4>] __pagevec_lru_add_active+0x104/0x130 [<ffffffff8026ab55>] handle_mm_fault+0x2e5/0x7e0 [<ffffffff80217d6b>] do_page_fault+0x45b/0xb20 [<ffffffff8026fb5c>] vma_merge+0x1bc/0x2a0 [<ffffffff80270780>] mmap_region+0x3f0/0x5e0 [<ffffffff80207227>] arch_get_unmapped_area+0xd7/0x280 [<ffffffff80270e15>] do_mmap_pgoff+0x4a5/0x500 [<ffffffff80548d6e>] thread_return+0x4c/0x2fe [<ffffffff8054b0d9>] error_exit+0x0/0x51 general protection fault: 0000 [2] PREEMPT CPU 0 Modules linked in: nls_iso8859_1 nfs nfsd lockd sunrpc exportfs radeon drm iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack ip_tables xt_tcpudp xt_owner ip6table_filter ip6_tables x_tables snd_seq_midi snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_emul snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss tun ftdi_sio usbserial usblp usb_storage it87 hwmon_vid cryptoloop rtc_cmos rtc_core usbhid rtc_lib hid snd_emu10k1 snd_rawmidi firmware_class snd_ac97_codec ac97_bus snd_pcm snd_seq_device snd_timer snd_page_alloc k8temp hwmon snd_util_mem snd_hwdep snd emu10k1_gp gameport soundcore sg ehci_hcd ohci_hcd i2c_nforce2 i2c_core fan thermal processor button Pid: 6665, comm: sh Tainted: G D W 2.6.26-hardened-r3 #1 RIP: 0010:[<ffffffff8025ef43>] [<ffffffff8025ef43>] free_pages_bulk+0x2d3/0x430 RSP: 0018:ffff8100110d9d28 EFLAGS: 00010046 RAX: ffffe20000816ab8 RBX: ffffe20000f43b20 RCX: 0000000000000000 RDX: fff7e20002185e30 RSI: ffffe20001420348 RDI: 0000000000000000 RBP: ffffe20001420310 R08: 4000000000040008 R09: ffff8100011012c0 R10: 0000000000000001 R11: 000000000000000e R12: 0000000000000058 R13: ff2002ffff2002ff R14: ffffffff80812a80 R15: 0000000000000000 FS: 00003310a76a4b00(0000) GS:ffffffff80840000(0000) knlGS:00000000e7969b90 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00003310a7345020 CR3: 000000004828c000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process sh (pid: 6665, threadinfo ffff8100110d8000, task ffff81000845b440) Stack: 0000003800000010 0000000000000000 0000000000000001 ffffffff80812ac8 00000001110d9db8 ffffffff80812ee0 0000000200000001 0000000000000000 0000000c0845b688 ffffe20000f43b20 0000000000000520 ffffffff80812ac8 Call Trace: [<ffffffff8025f3e3>] ? free_hot_cold_page+0x1d3/0x210 [<ffffffff80269beb>] ? unmap_vmas+0x4ab/0x790 [<ffffffff8026e56d>] ? exit_mmap+0x6d/0x130 [<ffffffff80224245>] ? mmput+0x25/0xe0 [<ffffffff8022a96b>] ? do_exit+0x19b/0x7a0 [<ffffffff8022afa4>] ? do_group_exit+0x34/0xc0 [<ffffffff8022b042>] ? sys_exit_group+0x12/0x20 [<ffffffff8020296b>] ? system_call_after_swapgs+0x7b/0x80 Code: 89 f0 74 04 48 8b 46 10 8b 40 08 85 c0 0f 85 54 01 00 00 48 8b 46 30 48 8b 56 28 41 bc 58 00 00 00 49 bd ff 02 20 ff ff 02 20 ff <48> 89 42 08 48 89 10 48 b8 ff 01 10 ff ff 01 10 ff 48 89 46 28 RIP [<ffffffff8025ef43>] free_pages_bulk+0x2d3/0x430 RSP <ffff8100110d9d28> ---[ end trace 847e4d83bdf41d97 ]--- Fixing recursive fault but reboot is needed! BUG: scheduling while atomic: sh/6665/0x00000005 Pid: 6665, comm: sh Tainted: G D W 2.6.26-hardened-r3 #1 Call Trace: [<ffffffff80548e40>] thread_return+0x11e/0x2fe [<ffffffff8022af35>] do_exit+0x765/0x7a0 [<ffffffff80220c32>] __wake_up_common+0x52/0x80 [<ffffffff802039f4>] oops_end+0x74/0x80 [<ffffffff8054b0d9>] error_exit+0x0/0x51 [<ffffffff8025ef43>] free_pages_bulk+0x2d3/0x430 [<ffffffff8025f01f>] free_pages_bulk+0x3af/0x430 [<ffffffff8025f3e3>] free_hot_cold_page+0x1d3/0x210 [<ffffffff80269beb>] unmap_vmas+0x4ab/0x790 [<ffffffff8026e56d>] exit_mmap+0x6d/0x130 [<ffffffff80224245>] mmput+0x25/0xe0 [<ffffffff8022a96b>] do_exit+0x19b/0x7a0 [<ffffffff8022afa4>] do_group_exit+0x34/0xc0 [<ffffffff8022b042>] sys_exit_group+0x12/0x20 [<ffffffff8020296b>] system_call_after_swapgs+0x7b/0x80 general protection fault: 0000 [3] PREEMPT CPU 0 Modules linked in: nls_iso8859_1 nfs nfsd lockd sunrpc exportfs radeon drm iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack ip_tables xt_tcpudp xt_owner ip6table_filter ip6_tables x_tables snd_seq_midi snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_emul snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss tun ftdi_sio usbserial usblp usb_storage it87 hwmon_vid cryptoloop rtc_cmos rtc_core usbhid rtc_lib hid snd_emu10k1 snd_rawmidi firmware_class snd_ac97_codec ac97_bus snd_pcm snd_seq_device snd_timer snd_page_alloc k8temp hwmon snd_util_mem snd_hwdep snd emu10k1_gp gameport soundcore sg ehci_hcd ohci_hcd i2c_nforce2 i2c_core fan thermal processor button Pid: 20536, comm: java Tainted: G D W 2.6.26-hardened-r3 #1 RIP: 0010:[<ffffffff8025d2b3>] [<ffffffff8025d2b3>] __rmqueue_smallest+0xc3/0x1c0 RSP: 0018:ffff810100097968 EFLAGS: 00010012 RAX: ffffffff80812af8 RBX: ffffffff80812ad8 RCX: ffffe20000816ab8 RDX: fff7e20002185e30 RSI: 0000000000000000 RDI: ffffffff80812a80 RBP: ffffe20001420348 R08: ffffe20001420370 R09: ff2002ffff2002ff R10: ff1001ffff1001ff R11: 0000000000000058 R12: 0000000000000000 R13: 0000000000000020 R14: 0000000000000000 R15: 0000000000000000 FS: 000000005dbbd950(0063) GS:ffffffff80840000(0000) knlGS:00000000ea5406d0 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000072b3320 CR3: 000000012f0d2000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process java (pid: 20536, threadinfo ffff810100096000, task ffff81004366b750) Stack: ffffe20000816ab8 ffffffff80812a80 0000000000000002 0000000000000000 000000000000001f ffffffff8025d880 0000000000001000 0000000000000001 ffff8100b9b4a140 ffffe20000816ab8 0000000000000004 0000000000000002 Call Trace: [<ffffffff8025d880>] ? __rmqueue+0x30/0x2d0 [<ffffffff8025dd02>] ? rmqueue_bulk+0x52/0xc0 [<ffffffff8025f72c>] ? get_page_from_freelist+0x1dc/0x5a0 [<ffffffff8025fbd7>] ? __alloc_pages_internal+0xe7/0x4a0 [<ffffffff802af511>] ? __set_page_dirty+0x101/0x170 [<ffffffff802596bd>] ? __grab_cache_page+0x6d/0xb0 [<ffffffff802b116d>] ? block_write_begin+0x7d/0xe0 [<ffffffff80329a72>] ? ext2_write_begin+0x22/0x30 [<ffffffff80329a80>] ? ext2_get_block+0x0/0x800 [<ffffffff8025a6bb>] ? generic_file_buffered_write+0x1cb/0x7a0 [<ffffffff8022bdfe>] ? current_fs_time+0x1e/0x30 [<ffffffff8025b23e>] ? __generic_file_aio_write_nolock+0x27e/0x490 [<ffffffff8025b4b4>] ? generic_file_aio_write+0x64/0xe0 [<ffffffff8028278b>] ? do_sync_write+0xdb/0x120 [<ffffffff8023dd50>] ? autoremove_wake_function+0x0/0x30 [<ffffffff802831db>] ? vfs_write+0xcb/0x170 [<ffffffff80283321>] ? sys_pwrite64+0xa1/0xb0 [<ffffffff8054b0d9>] ? error_exit+0x0/0x51 [<ffffffff8020296b>] ? system_call_after_swapgs+0x7b/0x80 Code: 01 10 ff 49 0f af f3 49 c1 e5 04 49 b9 ff 02 20 ff ff 02 20 ff 49 8d 44 35 00 4c 8b 44 07 58 49 8d 68 d8 48 8b 55 28 48 8b 45 30 <48> 89 42 08 48 89 10 4c 89 4d 30 4c 89 55 28 41 0f ba 70 d8 12 RIP [<ffffffff8025d2b3>] __rmqueue_smallest+0xc3/0x1c0 RSP <ffff810100097968> ---[ end trace 847e4d83bdf41d97 ]--- note: java[20536] exited with preempt_count 2 general protection fault: 0000 [4] PREEMPT CPU 0 Modules linked in: nls_iso8859_1 nfs nfsd lockd sunrpc exportfs radeon drm iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack ip_tables xt_tcpudp xt_owner ip6table_filter ip6_tables x_tables snd_seq_midi snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_emul snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss tun ftdi_sio usbserial usblp usb_storage it87 hwmon_vid cryptoloop rtc_cmos rtc_core usbhid rtc_lib hid snd_emu10k1 snd_rawmidi firmware_class snd_ac97_codec ac97_bus snd_pcm snd_seq_device snd_timer snd_page_alloc k8temp hwmon snd_util_mem snd_hwdep snd emu10k1_gp gameport soundcore sg ehci_hcd ohci_hcd i2c_nforce2 i2c_core fan thermal processor button Pid: 5690, comm: firefox Tainted: G D W 2.6.26-hardened-r3 #1 RIP: 0010:[<ffffffff8025d2b3>] [<ffffffff8025d2b3>] __rmqueue_smallest+0xc3/0x1c0 RSP: 0018:ffff81012afa3918 EFLAGS: 00010012 RAX: ffffffff80812af8 RBX: ffffffff80812ad8 RCX: ffffffff80812ac8 RDX: fff7e20002185e30 RSI: 0000000000000000 RDI: ffffffff80812a80 RBP: ffffe20001420348 R08: ffffe20001420370 R09: ff2002ffff2002ff R10: ff1001ffff1001ff R11: 0000000000000058 R12: 0000000000000000 R13: 0000000000000020 R14: 0000000000000000 R15: 0000000000000000 FS: 00006d27a7dcf700(0000) GS:ffffffff80840000(0000) knlGS:00000000e7969b90 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000072b3320 CR3: 000000012b04f000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process firefox (pid: 5690, threadinfo ffff81012afa2000, task ffff81012af2a390) Stack: ffffffff80812ac8 ffffffff80812a80 0000000000000002 0000000000000000 000000000000001f ffffffff8025d880 ffff81012ec24000 ffffffff80427596 ffff81012fe2b380 ffffffff80812ac8 0000000000000000 0000000000000002 Call Trace: [<ffffffff8025d880>] ? __rmqueue+0x30/0x2d0 [<ffffffff80427596>] ? ata_sff_tf_load+0xf6/0x1c0 [<ffffffff8025dd02>] ? rmqueue_bulk+0x52/0xc0 [<ffffffff8025f72c>] ? get_page_from_freelist+0x1dc/0x5a0 [<ffffffff8025fbd7>] ? __alloc_pages_internal+0xe7/0x4a0 [<ffffffff802596bd>] ? __grab_cache_page+0x6d/0xb0 [<ffffffff8030f1de>] ? ext3_write_begin+0xae/0x1d0 [<ffffffff8031c3e3>] ? __journal_file_buffer+0xa3/0x1d0 [<ffffffff8031bf8c>] ? __ext3_journal_dirty_metadata+0x2c/0x70 [<ffffffff8025a6bb>] ? generic_file_buffered_write+0x1cb/0x7a0 [<ffffffff803198dd>] ? __ext3_journal_stop+0x2d/0x60 [<ffffffff8029bb12>] ? file_update_time+0xf2/0x160 [<ffffffff8025b23e>] ? __generic_file_aio_write_nolock+0x27e/0x490 [<ffffffff8025b4b4>] ? generic_file_aio_write+0x64/0xe0 [<ffffffff8030bb93>] ? ext3_file_write+0x23/0xd0 [<ffffffff8028278b>] ? do_sync_write+0xdb/0x120 [<ffffffff80201b05>] ? do_notify_resume+0x85/0x980 [<ffffffff802624aa>] ? pagevec_lookup_tag+0x1a/0x30 [<ffffffff8023dd50>] ? autoremove_wake_function+0x0/0x30 [<ffffffff Reproducible: Sometimes Steps to Reproduce: Unknown at the moment.
I posted about this on the grsec forums here: http://forums.grsecurity.net/viewtopic.php?f=3&t=2059 The PAX guy says it's probably a stray bit error in memory, and I've been unable to reproduce it. If it happens again I'll try to reproduce it with vanilla, if that fails I'll reopen this bug with cc the pax/grsec people.
