Bug 242378 – sys-kernel/hardened-sources-2.6.26-r3: general protection fault in __rmqueue_smallest after page fault

• Product: Gentoo Linux
• Status: RESOLVED INVALID

[Description happyfool 2008-10-16 18:48:25 UTC]

I'd post this to the kernel.org bugzilla but I see that the grsec patch makes lots of changes to do_page_fault(), and I can't find a grsec bugzilla.

I saw the following crash recently, while the system was under pretty heavy load: an emerge of gcc, a java application with ~350 threads and moderate I/O, some nfs and smb accesses as well as the usual desktop apps. This is on a single core amd64 system with 4G of RAM.

The "as" in the first GPF (there were 3) was presumably from the gcc compile. Also, according to syslog there was a ~30s gap between GPF 2 and 3. The system stayed up and running apps were responsive to commands, but any new exec would segfault.

I did a few memtest passes to verify that the memory was OK (it was), and I'm trying to reproduce the problem at the moment. Regardless, the kernel shouldn't have crashed. Is there an upstream contact for grsec/pax?

general protection fault: 0000 [1] PREEMPT 
CPU 0 
Modules linked in: nls_iso8859_1 nfs nfsd lockd sunrpc exportfs radeon drm iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack ip_tables xt_tcpudp xt_owner ip6table_filter ip6_tables x_tables snd_seq_midi snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_emul snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss tun ftdi_sio usbserial usblp usb_storage it87 hwmon_vid cryptoloop rtc_cmos rtc_core usbhid rtc_lib hid snd_emu10k1 snd_rawmidi firmware_class snd_ac97_codec ac97_bus snd_pcm snd_seq_device snd_timer snd_page_alloc k8temp hwmon snd_util_mem snd_hwdep snd emu10k1_gp gameport soundcore sg ehci_hcd ohci_hcd i2c_nforce2 i2c_core fan thermal processor button
Pid: 6690, comm: as Tainted: G        W 2.6.26-hardened-r3 #1
RIP: 0010:[<ffffffff8025d2b3>]  [<ffffffff8025d2b3>] __rmqueue_smallest+0xc3/0x1c0
RSP: 0000:ffff810008521b88  EFLAGS: 00010012
RAX: ffffffff80812af8 RBX: ffffffff80812ad8 RCX: ffffe20000816ab8
RDX: fff7e20002185e30 RSI: 0000000000000000 RDI: ffffffff80812a80
RBP: ffffe20001420348 R08: ffffe20001420370 R09: ff2002ffff2002ff
R10: ff1001ffff1001ff R11: 0000000000000058 R12: 0000000000000000
R13: 0000000000000020 R14: 0000000000000000 R15: 0000000000000000
FS:  00002dfc12b66f50(0000) GS:ffffffff80840000(0000) knlGS:00000000ea5406d0
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002dfc11e0e000 CR3: 00000000aebb5000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process as (pid: 6690, threadinfo ffff810008520000, task ffff81000845b9d0)
Stack:  ffffe20000816ab8 ffffffff80812a80 0000000000000002 0000000000000000
000000000000001f ffffffff8025d880 0000000000000001 0000000000000000
0000000000000000 ffffe20000816ab8 000000000000001a 0000000000000002
Call Trace:
[<ffffffff8025d880>] ? __rmqueue+0x30/0x2d0
[<ffffffff8025dd02>] ? rmqueue_bulk+0x52/0xc0
[<ffffffff8025f72c>] ? get_page_from_freelist+0x1dc/0x5a0
[<ffffffff8025fbd7>] ? __alloc_pages_internal+0xe7/0x4a0
[<ffffffff80262ae4>] ? __pagevec_lru_add_active+0x104/0x130
[<ffffffff8026ab55>] ? handle_mm_fault+0x2e5/0x7e0
[<ffffffff80217d6b>] ? do_page_fault+0x45b/0xb20
[<ffffffff8026fb5c>] ? vma_merge+0x1bc/0x2a0
[<ffffffff80270780>] ? mmap_region+0x3f0/0x5e0
[<ffffffff80207227>] ? arch_get_unmapped_area+0xd7/0x280
[<ffffffff80270e15>] ? do_mmap_pgoff+0x4a5/0x500
[<ffffffff80548d6e>] ? thread_return+0x4c/0x2fe
[<ffffffff8054b0d9>] ? error_exit+0x0/0x51


Code: 01 10 ff 49 0f af f3 49 c1 e5 04 49 b9 ff 02 20 ff ff 02 20 ff 49 8d 44 35 00 4c 8b 44 07 58 49 8d 68 d8 48 8b 55 28 48 8b 45 30 <48> 89 42 08 48 89 10 4c 89 4d 30 4c 89 55 28 41 0f ba 70 d8 12 
RIP  [<ffffffff8025d2b3>] __rmqueue_smallest+0xc3/0x1c0
RSP <ffff810008521b88>
---[ end trace 847e4d83bdf41d97 ]---
note: as[6690] exited with preempt_count 2
BUG: scheduling while atomic: as/6690/0x10000003
Pid: 6690, comm: as Tainted: G      D W 2.6.26-hardened-r3 #1

Call Trace:
[<ffffffff80548e40>] thread_return+0x11e/0x2fe
[<ffffffff80221840>] __cond_resched+0x20/0x50
[<ffffffff805490e5>] _cond_resched+0x35/0x50
[<ffffffff80269dc5>] unmap_vmas+0x685/0x790
[<ffffffff8026e56d>] exit_mmap+0x6d/0x130
[<ffffffff80224245>] mmput+0x25/0xe0
[<ffffffff8022a96b>] do_exit+0x19b/0x7a0
[<ffffffff80220c32>] __wake_up_common+0x52/0x80
[<ffffffff802039f4>] oops_end+0x74/0x80
[<ffffffff8054b0d9>] error_exit+0x0/0x51
[<ffffffff8025d2b3>] __rmqueue_smallest+0xc3/0x1c0
[<ffffffff8025dd02>] rmqueue_bulk+0x52/0xc0
[<ffffffff8025d880>] __rmqueue+0x30/0x2d0
[<ffffffff8025dd02>] rmqueue_bulk+0x52/0xc0
[<ffffffff8025f72c>] get_page_from_freelist+0x1dc/0x5a0
[<ffffffff8025fbd7>] __alloc_pages_internal+0xe7/0x4a0
[<ffffffff80262ae4>] __pagevec_lru_add_active+0x104/0x130
[<ffffffff8026ab55>] handle_mm_fault+0x2e5/0x7e0
[<ffffffff80217d6b>] do_page_fault+0x45b/0xb20
[<ffffffff8026fb5c>] vma_merge+0x1bc/0x2a0
[<ffffffff80270780>] mmap_region+0x3f0/0x5e0
[<ffffffff80207227>] arch_get_unmapped_area+0xd7/0x280
[<ffffffff80270e15>] do_mmap_pgoff+0x4a5/0x500
[<ffffffff80548d6e>] thread_return+0x4c/0x2fe
[<ffffffff8054b0d9>] error_exit+0x0/0x51

general protection fault: 0000 [2] PREEMPT 
CPU 0 
Modules linked in: nls_iso8859_1 nfs nfsd lockd sunrpc exportfs radeon drm iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack ip_tables xt_tcpudp xt_owner ip6table_filter ip6_tables x_tables snd_seq_midi snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_emul snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss tun ftdi_sio usbserial usblp usb_storage it87 hwmon_vid cryptoloop rtc_cmos rtc_core usbhid rtc_lib hid snd_emu10k1 snd_rawmidi firmware_class snd_ac97_codec ac97_bus snd_pcm snd_seq_device snd_timer snd_page_alloc k8temp hwmon snd_util_mem snd_hwdep snd emu10k1_gp gameport soundcore sg ehci_hcd ohci_hcd i2c_nforce2 i2c_core fan thermal processor button
Pid: 6665, comm: sh Tainted: G      D W 2.6.26-hardened-r3 #1
RIP: 0010:[<ffffffff8025ef43>]  [<ffffffff8025ef43>] free_pages_bulk+0x2d3/0x430
RSP: 0018:ffff8100110d9d28  EFLAGS: 00010046
RAX: ffffe20000816ab8 RBX: ffffe20000f43b20 RCX: 0000000000000000
RDX: fff7e20002185e30 RSI: ffffe20001420348 RDI: 0000000000000000
RBP: ffffe20001420310 R08: 4000000000040008 R09: ffff8100011012c0
R10: 0000000000000001 R11: 000000000000000e R12: 0000000000000058
R13: ff2002ffff2002ff R14: ffffffff80812a80 R15: 0000000000000000
FS:  00003310a76a4b00(0000) GS:ffffffff80840000(0000) knlGS:00000000e7969b90
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00003310a7345020 CR3: 000000004828c000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process sh (pid: 6665, threadinfo ffff8100110d8000, task ffff81000845b440)
Stack:  0000003800000010 0000000000000000 0000000000000001 ffffffff80812ac8
00000001110d9db8 ffffffff80812ee0 0000000200000001 0000000000000000
0000000c0845b688 ffffe20000f43b20 0000000000000520 ffffffff80812ac8
Call Trace:
[<ffffffff8025f3e3>] ? free_hot_cold_page+0x1d3/0x210
[<ffffffff80269beb>] ? unmap_vmas+0x4ab/0x790
[<ffffffff8026e56d>] ? exit_mmap+0x6d/0x130
[<ffffffff80224245>] ? mmput+0x25/0xe0
[<ffffffff8022a96b>] ? do_exit+0x19b/0x7a0
[<ffffffff8022afa4>] ? do_group_exit+0x34/0xc0
[<ffffffff8022b042>] ? sys_exit_group+0x12/0x20
[<ffffffff8020296b>] ? system_call_after_swapgs+0x7b/0x80


Code: 89 f0 74 04 48 8b 46 10 8b 40 08 85 c0 0f 85 54 01 00 00 48 8b 46 30 48 8b 56 28 41 bc 58 00 00 00 49 bd ff 02 20 ff ff 02 20 ff <48> 89 42 08 48 89 10 48 b8 ff 01 10 ff ff 01 10 ff 48 89 46 28 
RIP  [<ffffffff8025ef43>] free_pages_bulk+0x2d3/0x430
RSP <ffff8100110d9d28>
---[ end trace 847e4d83bdf41d97 ]---
Fixing recursive fault but reboot is needed!
BUG: scheduling while atomic: sh/6665/0x00000005
Pid: 6665, comm: sh Tainted: G      D W 2.6.26-hardened-r3 #1

Call Trace:
[<ffffffff80548e40>] thread_return+0x11e/0x2fe
[<ffffffff8022af35>] do_exit+0x765/0x7a0
[<ffffffff80220c32>] __wake_up_common+0x52/0x80
[<ffffffff802039f4>] oops_end+0x74/0x80
[<ffffffff8054b0d9>] error_exit+0x0/0x51
[<ffffffff8025ef43>] free_pages_bulk+0x2d3/0x430
[<ffffffff8025f01f>] free_pages_bulk+0x3af/0x430
[<ffffffff8025f3e3>] free_hot_cold_page+0x1d3/0x210
[<ffffffff80269beb>] unmap_vmas+0x4ab/0x790
[<ffffffff8026e56d>] exit_mmap+0x6d/0x130
[<ffffffff80224245>] mmput+0x25/0xe0
[<ffffffff8022a96b>] do_exit+0x19b/0x7a0
[<ffffffff8022afa4>] do_group_exit+0x34/0xc0
[<ffffffff8022b042>] sys_exit_group+0x12/0x20
[<ffffffff8020296b>] system_call_after_swapgs+0x7b/0x80

general protection fault: 0000 [3] PREEMPT 
CPU 0 
Modules linked in: nls_iso8859_1 nfs nfsd lockd sunrpc exportfs radeon drm iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack ip_tables xt_tcpudp xt_owner ip6table_filter ip6_tables x_tables snd_seq_midi snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_emul snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss tun ftdi_sio usbserial usblp usb_storage it87 hwmon_vid cryptoloop rtc_cmos rtc_core usbhid rtc_lib hid snd_emu10k1 snd_rawmidi firmware_class snd_ac97_codec ac97_bus snd_pcm snd_seq_device snd_timer snd_page_alloc k8temp hwmon snd_util_mem snd_hwdep snd emu10k1_gp gameport soundcore sg ehci_hcd ohci_hcd i2c_nforce2 i2c_core fan thermal processor button
Pid: 20536, comm: java Tainted: G      D W 2.6.26-hardened-r3 #1
RIP: 0010:[<ffffffff8025d2b3>]  [<ffffffff8025d2b3>] __rmqueue_smallest+0xc3/0x1c0
RSP: 0018:ffff810100097968  EFLAGS: 00010012
RAX: ffffffff80812af8 RBX: ffffffff80812ad8 RCX: ffffe20000816ab8
RDX: fff7e20002185e30 RSI: 0000000000000000 RDI: ffffffff80812a80
RBP: ffffe20001420348 R08: ffffe20001420370 R09: ff2002ffff2002ff
R10: ff1001ffff1001ff R11: 0000000000000058 R12: 0000000000000000
R13: 0000000000000020 R14: 0000000000000000 R15: 0000000000000000
FS:  000000005dbbd950(0063) GS:ffffffff80840000(0000) knlGS:00000000ea5406d0
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000072b3320 CR3: 000000012f0d2000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process java (pid: 20536, threadinfo ffff810100096000, task ffff81004366b750)
Stack:  ffffe20000816ab8 ffffffff80812a80 0000000000000002 0000000000000000
000000000000001f ffffffff8025d880 0000000000001000 0000000000000001
ffff8100b9b4a140 ffffe20000816ab8 0000000000000004 0000000000000002
Call Trace:
[<ffffffff8025d880>] ? __rmqueue+0x30/0x2d0
[<ffffffff8025dd02>] ? rmqueue_bulk+0x52/0xc0
[<ffffffff8025f72c>] ? get_page_from_freelist+0x1dc/0x5a0
[<ffffffff8025fbd7>] ? __alloc_pages_internal+0xe7/0x4a0
[<ffffffff802af511>] ? __set_page_dirty+0x101/0x170
[<ffffffff802596bd>] ? __grab_cache_page+0x6d/0xb0
[<ffffffff802b116d>] ? block_write_begin+0x7d/0xe0
[<ffffffff80329a72>] ? ext2_write_begin+0x22/0x30
[<ffffffff80329a80>] ? ext2_get_block+0x0/0x800
[<ffffffff8025a6bb>] ? generic_file_buffered_write+0x1cb/0x7a0
[<ffffffff8022bdfe>] ? current_fs_time+0x1e/0x30
[<ffffffff8025b23e>] ? __generic_file_aio_write_nolock+0x27e/0x490
[<ffffffff8025b4b4>] ? generic_file_aio_write+0x64/0xe0
[<ffffffff8028278b>] ? do_sync_write+0xdb/0x120
[<ffffffff8023dd50>] ? autoremove_wake_function+0x0/0x30
[<ffffffff802831db>] ? vfs_write+0xcb/0x170
[<ffffffff80283321>] ? sys_pwrite64+0xa1/0xb0
[<ffffffff8054b0d9>] ? error_exit+0x0/0x51
[<ffffffff8020296b>] ? system_call_after_swapgs+0x7b/0x80


Code: 01 10 ff 49 0f af f3 49 c1 e5 04 49 b9 ff 02 20 ff ff 02 20 ff 49 8d 44 35 00 4c 8b 44 07 58 49 8d 68 d8 48 8b 55 28 48 8b 45 30 <48> 89 42 08 48 89 10 4c 89 4d 30 4c 89 55 28 41 0f ba 70 d8 12 
RIP  [<ffffffff8025d2b3>] __rmqueue_smallest+0xc3/0x1c0
RSP <ffff810100097968>
---[ end trace 847e4d83bdf41d97 ]---
note: java[20536] exited with preempt_count 2
general protection fault: 0000 [4] PREEMPT 
CPU 0 
Modules linked in: nls_iso8859_1 nfs nfsd lockd sunrpc exportfs radeon drm iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack ip_tables xt_tcpudp xt_owner ip6table_filter ip6_tables x_tables snd_seq_midi snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_emul snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss tun ftdi_sio usbserial usblp usb_storage it87 hwmon_vid cryptoloop rtc_cmos rtc_core usbhid rtc_lib hid snd_emu10k1 snd_rawmidi firmware_class snd_ac97_codec ac97_bus snd_pcm snd_seq_device snd_timer snd_page_alloc k8temp hwmon snd_util_mem snd_hwdep snd emu10k1_gp gameport soundcore sg ehci_hcd ohci_hcd i2c_nforce2 i2c_core fan thermal processor button
Pid: 5690, comm: firefox Tainted: G      D W 2.6.26-hardened-r3 #1
RIP: 0010:[<ffffffff8025d2b3>]  [<ffffffff8025d2b3>] __rmqueue_smallest+0xc3/0x1c0
RSP: 0018:ffff81012afa3918  EFLAGS: 00010012
RAX: ffffffff80812af8 RBX: ffffffff80812ad8 RCX: ffffffff80812ac8
RDX: fff7e20002185e30 RSI: 0000000000000000 RDI: ffffffff80812a80
RBP: ffffe20001420348 R08: ffffe20001420370 R09: ff2002ffff2002ff
R10: ff1001ffff1001ff R11: 0000000000000058 R12: 0000000000000000
R13: 0000000000000020 R14: 0000000000000000 R15: 0000000000000000
FS:  00006d27a7dcf700(0000) GS:ffffffff80840000(0000) knlGS:00000000e7969b90
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000072b3320 CR3: 000000012b04f000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process firefox (pid: 5690, threadinfo ffff81012afa2000, task ffff81012af2a390)
Stack:  ffffffff80812ac8 ffffffff80812a80 0000000000000002 0000000000000000
000000000000001f ffffffff8025d880 ffff81012ec24000 ffffffff80427596
ffff81012fe2b380 ffffffff80812ac8 0000000000000000 0000000000000002
Call Trace:
[<ffffffff8025d880>] ? __rmqueue+0x30/0x2d0
[<ffffffff80427596>] ? ata_sff_tf_load+0xf6/0x1c0
[<ffffffff8025dd02>] ? rmqueue_bulk+0x52/0xc0
[<ffffffff8025f72c>] ? get_page_from_freelist+0x1dc/0x5a0
[<ffffffff8025fbd7>] ? __alloc_pages_internal+0xe7/0x4a0
[<ffffffff802596bd>] ? __grab_cache_page+0x6d/0xb0
[<ffffffff8030f1de>] ? ext3_write_begin+0xae/0x1d0
[<ffffffff8031c3e3>] ? __journal_file_buffer+0xa3/0x1d0
[<ffffffff8031bf8c>] ? __ext3_journal_dirty_metadata+0x2c/0x70
[<ffffffff8025a6bb>] ? generic_file_buffered_write+0x1cb/0x7a0
[<ffffffff803198dd>] ? __ext3_journal_stop+0x2d/0x60
[<ffffffff8029bb12>] ? file_update_time+0xf2/0x160
[<ffffffff8025b23e>] ? __generic_file_aio_write_nolock+0x27e/0x490
[<ffffffff8025b4b4>] ? generic_file_aio_write+0x64/0xe0
[<ffffffff8030bb93>] ? ext3_file_write+0x23/0xd0
[<ffffffff8028278b>] ? do_sync_write+0xdb/0x120
[<ffffffff80201b05>] ? do_notify_resume+0x85/0x980
[<ffffffff802624aa>] ? pagevec_lookup_tag+0x1a/0x30
[<ffffffff8023dd50>] ? autoremove_wake_function+0x0/0x30
[<ffffffff

Reproducible: Sometimes

Steps to Reproduce:
Unknown at the moment.

[Comment 1 happyfool 2008-10-17 19:42:11 UTC]

I posted about this on the grsec forums here:
http://forums.grsecurity.net/viewtopic.php?f=3&t=2059
The PAX guy says it's probably a stray bit error in memory, and I've been unable to reproduce it. If it happens again I'll try to reproduce it with vanilla, if that fails I'll reopen this bug with cc the pax/grsec people.