When i compile kernel with BPF_JIT=y and do echo 1 > /proc/sys/net/core/bpf_jit_enable i'm getting "general protection fault" after a couple minutes of work. E.g. 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.552323] general protection fault: 0000 [#12] SMP 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.553361] CPU 0 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.553587] Modules linked in: tun cryptd aes_x86_64 aes_generic cbc dm_crypt zram(C) sit tunnel4 xt_TCPMSS ipt_REJ ECT xt_multiport xt_set ip_set nfnetlink nf_conntrack_ipv6 nf_defrag_ipv6 ip6t_rt xt_state iptable_raw ip6table_mangle iptable_mangle iptable_nat nf_nat nf_conntrack_ ipv4 nf_conntrack nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter xfs exportfs dm_mod ipv6 8139too sr_mod 8139cp cdrom evdev 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] Pid: 6504, comm: shutdown Tainted: G D C 3.1.8-hardened #2 Xen HVM domU 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] RIP: 0010:[<ffffffff810d3837>] [<ffffffff810d3837>] kmem_cache_alloc+0x47/0xc0 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] RSP: 0018:ffff8800038ddd68 EFLAGS: 00010202 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] RAX: 0000000000000000 RBX: ffff88001e6a5400 RCX: 00000000000009c4 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] RDX: 00000000000d68ec RSI: 000000000000f860 RDI: ffffffff8126e556 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] RBP: ffff8800038ddd88 R08: ffff88001fc0f860 R09: 0000000000000000 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] R10: 0000000000000411 R11: 0000000000000246 R12: ffff88001e802700 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] R13: 168c06ae1ffe460e R14: 00000000000000d0 R15: 0000000000000000 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] FS: 0000038f7c6f0700(0000) GS:ffff88001fc00000(0000) knlGS:0000000000000000 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] CR2: 0000038f7c232b00 CR3: 000000000132b000 CR4: 00000000000006b0 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] Process shutdown (pid: 6504, threadinfo ffff88001ebdeff0, task ffff88001ebdec00) 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] Stack: 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] ffff88001e6a5400 0000000000000000 0000000000000001 ffffffff81a10680 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] ffff8800038dddb8 ffffffff8126e556 ffff8800038dddf8 0000000000000001 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] ffff88001e80e000 0000000000000001 ffff8800038dddd8 ffffffff810fa06f 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] Call Trace: 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] [<ffffffff8126e556>] sock_alloc_inode+0x46/0xe0 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] [<ffffffff810fa06f>] alloc_inode+0x2f/0xb0 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] [<ffffffff810fb93e>] new_inode_pseudo+0xe/0x60 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] [<ffffffff8126f1f9>] sock_alloc+0x19/0x70 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] [<ffffffff8126f8e0>] __sock_create+0xa0/0x260 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] [<ffffffff8102350c>] ? do_page_fault+0x1bc/0x510 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] [<ffffffff8126fafb>] sock_create+0x2b/0x40 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] [<ffffffff81272643>] sys_socket+0x93/0xd0 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] [<ffffffff8131c77b>] system_call_fastpath+0x18/0x1d 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] Code: 24 10 41 89 f6 48 8b 7d 08 4d 8b 04 24 65 4c 03 04 25 c8 b1 00 00 49 8b 50 08 4d 8b 28 4d 85 ed 7 4 64 49 63 44 24 20 49 8b 34 24 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] RIP [<ffffffff810d3837>] kmem_cache_alloc+0x47/0xc0 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.561436] RSP <ffff8800038ddd68> 2012-01-18T12:41:19.966686+01:00 meteor kernel: [ 861.601437] ---[ end trace 3ea8321190620043 ]--- 2012-01-18T12:43:08.210000+01:00 meteor kernel: imklog 5.6.5, log source = /proc/kmsg started. 2012-01-18T12:43:08.210000+01:00 meteor kernel: [ 0.000000] Linux version 3.1.8-hardened (root@meteor) (gcc version 4.5.3 (Gentoo Hardened 4.5.3-r1 p1.0, pie-0.4.5 ) ) #2 SMP Wed Jan 18 11:10:02 CET 2012 2012-01-18T12:43:08.210000+01:00 meteor kernel: [ 0.000000] Command line: root=/dev/sda2 ro rootfstype=ext4 xen_emul_unplug=never panic=30 Yzcache 2012-01-18T12:43:08.210000+01:00 meteor kernel: [ 0.000000] BIOS-provided physical RAM map: Reproducible: Always
Created attachment 299227 [details] kernel config
I wrote version 3.1.7&3.1.8 because i tested them, i didn't try other version, probably they will be affected too.
Okay I'm cc-ing upstream on this one. Can you test 3.2.2-r1 which is the latest stable. You're probably right, but just in case something changed.
also can you try a vanilla kernel please?
On hardened 3.2.2-r1 i've started `tcpdump -i any -n` and i got: 2012-02-04T23:10:12.622385+01:00 meteor kernel: [ 690.704833] __sk_free: optmem leakage (245768 bytes) detected. 2012-02-04T23:10:15.172404+01:00 meteor kernel: [ 693.224942] general protection fault: 0000 [#1] SMP 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.225817] CPU 1 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.226042] Modules linked in: af_packet tun cryptd aes_x86_64 aes_generic cbc dm_crypt zram(C) sit tunnel4 xt_NFLOG nfnetlink_log xt_TCPMSS ipt_REJECT xt_multiport xt_set nf_conntrack_ipv6 nf_defrag_ipv6 ip6t_rt xt_state xt_NOTRACK ip_set nfnetlink iptable_raw ip6table_mangle iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter ip_tables xfs exportfs dm_mod ipv6 sr_mod 8139cp cdrom evdev 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] Pid: 3607, comm: nagios Tainted: G C 3.2.2-hardened-r1 #4 Xen HVM domU 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] RIP: 0010:[<ffffffff810d4c09>] [<ffffffff810d4c09>] __kmalloc+0x69/0x100 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] RSP: 0018:ffff88001457dbf8 EFLAGS: 00010286 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] RAX: 0000000000000000 RBX: 0000000017155588 RCX: ffff88000f70c45c 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] RDX: 00000000000b9fd6 RSI: 000000000000fe10 RDI: ffffffff8114f8b2 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] RBP: ffff88001457dc18 R08: ffff88001fd0fe10 R09: 0000000063686563 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] R10: 000000000b0b0b0b R11: 00000000520e2c72 R12: ffff88001e802700 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] R13: ff88001db58a8000 R14: 00000000000080d0 R15: ffff88001db580c0 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] FS: 0000031fd5d9f700(0000) GS:ffff88001fd00000(0000) knlGS:0000000000000000 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] CR2: 0000044684474000 CR3: 000000000132b000 CR4: 00000000000006b0 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] Process nagios (pid: 3607, threadinfo ffff88001ea96ef0, task ffff88001ea96b00) 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] Stack: 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] 0000000017155588 ffff88001457dd38 ffff880016f9da28 ffff88000f70c45c 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] ffff88001457dc68 ffffffff8114f8b2 17155588520e2c72 d569d49a3fd7941c 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] ffff88001457dc68 ffff88000f70c45c ffff88001457dd38 ffff880016f9da28 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] Call Trace: 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] [<ffffffff8114f8b2>] ext4_htree_store_dirent+0x32/0x140 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] [<ffffffff8115c761>] htree_dirblock_to_tree+0x131/0x190 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] [<ffffffff8115e9fb>] ext4_htree_fill_tree+0x14b/0x230 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] [<ffffffff810b4d7f>] ? handle_mm_fault+0x1cf/0x350 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] [<ffffffff810f37f0>] ? filldir64+0x280/0x280 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] [<ffffffff8114f6c6>] ext4_readdir+0x456/0x5c0 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] [<ffffffff810f37f0>] ? filldir64+0x280/0x280 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] [<ffffffff810f3d55>] vfs_readdir+0xc5/0xe0 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] [<ffffffff810f3e95>] sys_getdents+0xb5/0x190 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] [<ffffffff8131c13b>] system_call_fastpath+0x18/0x1d 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] Code: 87 95 00 00 00 48 8b 7d 08 4d 8b 04 24 65 4c 03 04 25 20 b8 00 00 49 8b 50 08 4d 8b 28 4d 85 ed 74 62 49 63 44 24 20 49 8b 34 24 <49> 8b 5c 05 00 48 8d 4a 01 4c 89 e8 65 48 0f c7 0e 0f 94 c0 84 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] RIP [<ffffffff810d4c09>] __kmalloc+0x69/0x100 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.234830] RSP <ffff88001457dbf8> 2012-02-04T23:10:15.182404+01:00 meteor kernel: [ 693.265009] ---[ end trace 7028de58d61c3686 ]--- 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.319253] general protection fault: 0000 [#2] SMP 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.319253] CPU 1 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.319253] Modules linked in: af_packet tun cryptd aes_x86_64 aes_generic cbc dm_crypt zram(C) sit tunnel4 xt_NFLOG nfnetlink_log xt_TCPMSS ipt_REJECT xt_multiport xt_set nf_conntrack_ipv6 nf_defrag_ipv6 ip6t_rt xt_state xt_NOTRACK ip_set nfnetlink iptable_raw ip6table_mangle iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter ip_tables xfs exportfs dm_mod ipv6 sr_mod 8139cp cdrom evdev 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] Pid: 17309, comm: dict Tainted: G D C 3.2.2-hardened-r1 #4 Xen HVM domU 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] RIP: 0010:[<ffffffff810d4dc7>] [<ffffffff810d4dc7>] kmem_cache_alloc+0x47/0xc0 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] RSP: 0018:ffff880018e33d08 EFLAGS: 00010286 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] RAX: 0000000000000000 RBX: ffff88000010b680 RCX: 0000000000000eb8 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] RDX: 00000000000b9fd6 RSI: 000000000000fe10 RDI: ffffffff81274a66 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] RBP: ffff880018e33d28 R08: ffff88001fd0fe10 R09: 00000041ed040650 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] R10: 0000000000000001 R11: 0000000000000246 R12: ffff88001e802700 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] R13: ff88001db58a8000 R14: 00000000000000d0 R15: 0000000000000000 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] FS: 000002f7034a8700(0000) GS:ffff88001fd00000(0000) knlGS:0000000000000000 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] CR2: 0000002c090ec028 CR3: 000000000132b000 CR4: 00000000000006b0 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] Process dict (pid: 17309, threadinfo ffff8800036546d0, task ffff8800036542e0) 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] Stack: 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] ffff88000010b680 0000000000000000 000003f981047350 000003f98104734c 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] ffff880018e33d58 ffffffff81274a66 0000000000000006 ffff88001e6a1900 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] ffff88001e80e000 000003f981047350 ffff880018e33d78 ffffffff810fb7af 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] Call Trace: 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] [<ffffffff81274a66>] sock_alloc_inode+0x46/0xe0 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] [<ffffffff810fb7af>] alloc_inode+0x2f/0xb0 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] [<ffffffff810fd07e>] new_inode_pseudo+0xe/0x60 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] [<ffffffff81275709>] sock_alloc+0x19/0x70 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] [<ffffffff81278ff6>] sys_accept4+0xa6/0x2a0 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] [<ffffffff8106527e>] ? getnstimeofday+0x5e/0xe0 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] [<ffffffff81065365>] ? do_gettimeofday+0x15/0x50 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] [<ffffffff812791fb>] sys_accept+0xb/0x20 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] [<ffffffff8131c13b>] system_call_fastpath+0x18/0x1d 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] Code: 24 10 41 89 f6 48 8b 7d 08 4d 8b 04 24 65 4c 03 04 25 20 b8 00 00 49 8b 50 08 4d 8b 28 4d 85 ed 74 64 49 63 44 24 20 49 8b 34 24 <49> 8b 5c 05 00 48 8d 4a 01 4c 89 e8 65 48 0f c7 0e 0f 94 c0 84 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] RIP [<ffffffff810d4dc7>] kmem_cache_alloc+0x47/0xc0 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.324831] RSP <ffff880018e33d08> 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.368689] ---[ end trace 7028de58d61c3687 ]--- 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.368689] general protection fault: 0000 [#3] SMP 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.368689] CPU 1 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.368689] Modules linked in: af_packet tun cryptd aes_x86_64 aes_generic cbc dm_crypt zram(C) sit tunnel4 xt_NFLOG nfnetlink_log xt_TCPMSS ipt_REJECT xt_multiport xt_set nf_conntrack_ipv6 nf_defrag_ipv6 ip6t_rt xt_state xt_NOTRACK ip_set nfnetlink iptable_raw ip6table_mangle iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter ip_tables xfs exportfs dm_mod ipv6 sr_mod 8139cp cdrom evdev 2012-02-04T23:10:15.272405+01:00 meteor kernel: [ 693.368689] [... other processes] On vanilla-3.2.2 i didn't reproduce problem yet (tcpdump didn't trigger problem).
i managed to reproduce the problem (it's a double free of the work struct i added to the bpf jit code), the next patch will fix it.
Good news, thanks.
Marcin did you verify any of the latest hardened-sources kernels to see that it was fixed?
I didn't notice there is new hardened kernel. On 3.2.5-hardened i can't reproduce issue. Thanks.
(In reply to comment #9) > I didn't notice there is new hardened kernel. On 3.2.5-hardened i can't > reproduce issue. > Thanks. Okay I'm going to close this resolved. Thanks pipacs :)