Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 254207 - net-firewall/ipset incompatible with 2.6.28
Summary: net-firewall/ipset incompatible with 2.6.28
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Core system (show other bugs)
Hardware: AMD64 Linux
: High normal (vote)
Assignee: Robin Johnson
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-01-08 10:20 UTC by Jochen Schlick
Modified: 2009-02-03 13:54 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
kernel config (ooops.config.thinkpad_sl500.090108.txt,81.33 KB, text/plain)
2009-01-08 19:37 UTC, Jochen Schlick
Details
dmesg output (after shorewall/iptables start) (ooops.dmesg.thinkpad_sl500.090108.txt,30.30 KB, text/plain)
2009-01-08 19:38 UTC, Jochen Schlick
Details
lsmod output (ooops.lsmod.thinkpad_sl500.090108.txt,6.34 KB, text/plain)
2009-01-08 19:39 UTC, Jochen Schlick
Details
/var/log/messages after /etc/init.d/shorewall start (messages.ooops.090110.txt,54.98 KB, text/plain)
2009-01-10 19:31 UTC, Jochen Schlick
Details
debug patch (nfdebug.patch,700 bytes, patch)
2009-01-11 11:33 UTC, Daniel Drake (RETIRED)
Details | Diff
iptables log of the new patched kernel - nocrash :-( (20090111.iptables_log.2628gentoodebug.nocrash.txt,3.80 KB, text/plain)
2009-01-11 16:16 UTC, Jochen Schlick
Details
messages of the new patched kernel - nocrash :-( (20090111.messages.2628gentoodebug.nocrash.txt,93.03 KB, text/plain)
2009-01-11 16:18 UTC, Jochen Schlick
Details
iptables log of the unpatched kernel (20090111.iptables_log.2628gentoo.origkernel.crash.txt,2.46 KB, text/plain)
2009-01-11 17:18 UTC, Jochen Schlick
Details
messages of the unpatched kernel (20090111.messages.2628gentoo.origkernel.crash.txt,7.92 KB, text/plain)
2009-01-11 17:20 UTC, Jochen Schlick
Details
messages of a 2.6.28 vanilla kernel (with nvidia) (20090112.messages.2628.vanilla.nvidia.crash.txt,55.37 KB, text/plain)
2009-01-12 22:08 UTC, Jochen Schlick
Details
messages of a 2.6.28 vanilla kernel (no nvidia) (20090112.messages.2628.vanilla.crash.txt,54.01 KB, text/plain)
2009-01-12 22:41 UTC, Jochen Schlick
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jochen Schlick 2009-01-08 10:20:33 UTC
BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
IP: [<ffffffffa0ce801c>] 0xffffffffa0ce801c
PGD 137051067 PUD 137c01067 PMD 0
Oops: 0000 [#1] SMP
last sysfs file: /sys/devices/system/cpu/cpu1/cache/index2/shared_cpu_map
CPU 0
Modules linked in: iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_TTL
ipt_ttl ipt_set ipt_REJECT ipt_REDIRECT ipt_NETMAP ipt_MASQUERADE ipt_LOG
ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah ipt_addrtype nf_nat_tftp
nf_nat_snmp_basic nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc
nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp nf_conntrack_amanda
nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip nf_conntrack_proto_sctp
nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netlink
nf_conntrack_netbios_ns nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp
ip_set_portmap ip_set_macipmap ip_set_ipmap ip_set_iphash ip_set xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG xt_multiport xt_MARK
xt_mark xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP
xt_dscp xt_dccp xt_conntrack xt_CONNMARK xt_connmark xt_CLASSIFY xt_tcpudp
xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack
iptable_mangle nfnetlink iptable_filter ip_tables x_tables aes_x86_64
aes_generic bridge stp llc snd_pcm_oss snd_mixer_oss snd_seq_dummy snd_seq_oss
snd_seq_midi_event snd_seq snd_seq_device ipv6 coretemp btusb uvcvideo
bluetooth nvidiafb compat_ioctl32 videodev fb_ddc i2c_algo_bit vgastate
v4l1_compat nvidia(P) firewire_ohci firewire_core snd_hda_intel arc4 ecb
ohci1394 iwlagn uhci_hcd ehci_hcd iwlcore snd_pcm intel_agp sg sdhci_pci
snd_timer ieee1394 usbcore r8169 snd_page_alloc sdhci mac80211 mmc_core
i2c_core mii video pcspkr serio_raw snd_hwdep ac cfg80211 output button snd
joydev battery
Pid: 6900, comm: iptables Tainted: P           2.6.28-gentoo #2
RIP: 0010:[<ffffffffa0ce801c>]  [<ffffffffa0ce801c>] 0xffffffffa0ce801c
RSP: 0018:ffff880133619c98  EFLAGS: 00010292
RAX: ffffffffa0ce8010 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000020 RSI: 0000000000000020 RDI: ffff880133619d48
RBP: ffff880133619ca8 R08: 0000000000000000 R09: 00000000fffffffe
R10: 0000000000000130 R11: 0000000000000000 R12: 0000000000000000
R13: ffff880133619d48 R14: ffffc2001225130a R15: 0000000000000000
FS:  00007f99cdae86f0(0000) GS:ffffffff80867d80(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000133672000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process iptables (pid: 6900, threadinfo ffff880133618000, task
ffff8801393c2c20)
Stack:
 ffff880133619ce8 ffffffffa0ce84a0 ffff880133619ce8 ffffffffa0b0da95
 00ff880137e158b0 ffff8801337a9d80 ffffc20012251308 ffffc20012250000
 ffffc2001225130a 0000000000000070 ffff880133619da8 ffffffffa0b1564e
Call Trace:
 [<ffffffffa0b0da95>] xt_check_match+0xf5/0x180 [x_tables]
 [<ffffffffa0b1564e>] translate_table+0x30e/0x680 [ip_tables]
 [<ffffffffa0b16924>] do_ipt_set_ctl+0x124/0x1a0 [ip_tables]
 [<ffffffff805b83b0>] nf_sockopt+0x60/0xa0
 [<ffffffff805b842c>] nf_setsockopt+0x1c/0x20
 [<ffffffff805c5ad2>] ip_setsockopt+0xa2/0xb0
 [<ffffffff805e23e5>] raw_setsockopt+0x25/0x70
 [<ffffffff8058c53f>] sock_common_setsockopt+0xf/0x20
 [<ffffffff8058a06b>] sys_setsockopt+0x7b/0xe0
 [<ffffffff8020c2db>] system_call_fastpath+0x16/0x1b
Code: <0f> b7 39 e8 6c a0 ef ff 66 ff c0 74 13 8b 53 1c b8 01 00 00 00 85
RIP  [<ffffffffa0ce801c>] 0xffffffffa0ce801c
 RSP <ffff880133619c98>
CR2: 0000000000000000
---[ end trace 32af231d28b68844 ]---




Reproducible: Always
Comment 1 George Kadianakis (RETIRED) gentoo-dev 2009-01-08 15:06:27 UTC
Could you attach your kernel config and your lsmod and dmesg output?

Thanks :)
Comment 2 Jochen Schlick 2009-01-08 19:37:06 UTC
Created attachment 177792 [details]
kernel config
Comment 3 Jochen Schlick 2009-01-08 19:38:02 UTC
Created attachment 177793 [details]
dmesg output (after shorewall/iptables start)
Comment 4 Jochen Schlick 2009-01-08 19:39:27 UTC
Created attachment 177795 [details]
lsmod output
Comment 5 George Kadianakis (RETIRED) gentoo-dev 2009-01-08 19:41:10 UTC
It would, also, be very helpful if you followed the steps below to isolate the line of code that is causing the oops. The instructions have to be followed in an identical kernel as the one that causes the crashes.

# emerge -n gdb
# cd /usr/src/linux #or wherever your kernel lies
# rm net/netfilter/x_tables.o
# make CONFIG_DEBUG_INFO=y net/netfilter/x_tables.o
# gdb net/netfilter/x_tables.o

Then at the gdb prompt:

# list *xt_check_match+0xf5

Then just post the output of the above command here!

Thanks :)
Comment 6 Jochen Schlick 2009-01-08 22:51:03 UTC
hope this helps


/usr/src/linux-2.6.28-gentoo # gdb net/netfilter/x_tables.o
GNU gdb 6.8
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu"...
(gdb) list *xt_check_match+0xf5
0x1a95 is in xt_check_match (net/netfilter/x_tables.c:357).
352                     printk("%s_tables: %s match: only valid for protocol %u\n",
353                            xt_prefix[par->family], par->match->name,
354                            par->match->proto);
355                     return -EINVAL;
356             }
357             if (par->match->checkentry != NULL && !par->match->checkentry(par))
358                     return -EINVAL;
359             return 0;
360     }
361     EXPORT_SYMBOL_GPL(xt_check_match);
(gdb) 
Comment 7 Markos Chandras (RETIRED) gentoo-dev 2009-01-09 11:33:24 UTC
Great output thanks.

Please make sure the below lines are set on /etc/shorewall/shorewall.conf file

LOGFILE=/var/log/messages
LOGFORMAT="Shorewall:%s:%s:"

If these line are like this already, attach your /var/log/messages. 

If they are not,bootup your system without starting shorewall. After booting successfully, run the shorewall by executing /etc/init.d/shorewall start

The your system , should crash. If so, reboot and attach here your /var/log/messages

Thank you
Comment 8 Daniel Drake (RETIRED) gentoo-dev 2009-01-09 16:36:51 UTC
Also, which is the last known working kernel?
Comment 9 Jochen Schlick 2009-01-09 19:59:50 UTC
sorry for the delay, but the thinkpad is not available at the moment (until tomorrow morning). but the current working kernel on this thinkpad is: linux-2.6.27-gentoo-r7

best regards
Comment 10 Jochen Schlick 2009-01-10 19:31:30 UTC
Created attachment 177991 [details]
/var/log/messages after /etc/init.d/shorewall start

there is no output from shorewall in /var/log/messages - only from iptables and the concerning oops after manual start of shorewall....
Comment 11 Daniel Drake (RETIRED) gentoo-dev 2009-01-11 11:17:43 UTC
Please create a file at /usr/local/bin/iptables_log and put this inside it:

-----------
#!/bin/bash
echo "$(date) $$ $*" >> /tmp/iptables_log
exec /sbin/iptables $*
-----------

make it executable:
chmod a+x /usr/local/bin/iptables_log

Then in shorewall.conf add:
IPTABLES=/usr/local/bin/iptables_log

Then start shorewall on an affected kernel, reproducing the crash. Post the new crash dump, and the contents of /usr/local/bin/iptables_log to this bug. Thanks!
Comment 12 Daniel Drake (RETIRED) gentoo-dev 2009-01-11 11:18:27 UTC
(In reply to comment #11)
> Then start shorewall on an affected kernel, reproducing the crash. Post the new
> crash dump, and the contents of /usr/local/bin/iptables_log to this bug.
> Thanks!


Should have read:
> Then start shorewall on an affected kernel, reproducing the crash. Post the new
> crash dump, and the contents of /tmp/iptables_log to this bug.
Comment 13 Daniel Drake (RETIRED) gentoo-dev 2009-01-11 11:33:18 UTC
Created attachment 178020 [details, diff]
debug patch

At the same time, please have this patch applied to 2.6.28 to add some more debug output. When including the updated crash dump per the instructions above, please also include the few preceding lines in the kernel log (which is what this patch will add)
Comment 14 Jochen Schlick 2009-01-11 16:11:36 UTC
ok, first of all shorewall expected then also a "iptables_log-restore" in /usr/local/bin, so I created one (a symlink to /sbin/iptables-restore). I hope this is correct.

I built a new kernel (2.6.28-gentoo.debug), rebooted several times but no oops :-(
(still investigating)
Comment 15 Jochen Schlick 2009-01-11 16:16:50 UTC
Created attachment 178083 [details]
iptables log of the new patched kernel - nocrash :-(
Comment 16 Jochen Schlick 2009-01-11 16:18:20 UTC
Created attachment 178084 [details]
messages of the new patched kernel - nocrash :-(
Comment 17 Jochen Schlick 2009-01-11 17:18:30 UTC
Created attachment 178087 [details]
iptables log of the unpatched kernel
Comment 18 Jochen Schlick 2009-01-11 17:20:15 UTC
Created attachment 178089 [details]
messages of the unpatched kernel
Comment 19 Daniel Drake (RETIRED) gentoo-dev 2009-01-11 20:08:36 UTC
How strange. The fact that the debug patch changes the behaviour probably means that it is a timing-sensitive issue.

The command it is crashing on is:
iptables -A fooX1594 -m set --set fooX1594 src -j ACCEPT

I think we should take this upstream as the next step. Before we do that, we need to reproduce this on an unpatched, untainted kernel. Please can you reproduce this on vanilla-sources-2.6.28, without the nvidia module, and post a new crash dump?
Comment 20 Jochen Schlick 2009-01-12 22:08:56 UTC
Created attachment 178271 [details]
messages of a 2.6.28 vanilla kernel (with nvidia)

sys-kernel/vanilla-sources 2.6.28 created with genkernel crashes also
Comment 21 Daniel Drake (RETIRED) gentoo-dev 2009-01-12 22:24:07 UTC
Thanks. That one still had the nvidia driver loaded though. Please configure your system so that the nvidia module doesn't get loaded at all, and post a new crash dump.
Comment 22 Jochen Schlick 2009-01-12 22:41:01 UTC
Created attachment 178287 [details]
messages of a 2.6.28 vanilla kernel (no nvidia)

not so fast ;-)

I removed the nvidia kernel module and called depmod for the vanilla kernel and it oopses again with the untainted kernel.

PS: I write this comment on the same laptop running the 2.6.28-gentoo.debug kernel and still no crash with this kernel.
Comment 23 Daniel Drake (RETIRED) gentoo-dev 2009-01-16 23:56:37 UTC
Perfect, thanks. Please now file an upstream bug for this at http://bugzilla.kernel.org. My suggestions:

- emphasize that this is a 2.6.28 regression (2.6.27 worked)
- note the iptables command which we have determined to be the trigger for the crash (explained above)
- attach your most recent dmesg there (non-tainted kernel with crash message)

I'll add some more technical details after you have done that. Please post the new bug URL here when done, thanks!
Comment 24 Jochen Schlick 2009-01-20 22:00:11 UTC
ok I created one on bugzilla.kernel.org


http://bugzilla.kernel.org/show_bug.cgi?id=12517
Comment 25 Jochen Schlick 2009-01-30 02:28:47 UTC
The reason for the oops was an incompatibility between the 2.6.28 kernels and the net-firewall/ipset package !!

deinstallation of /net-firewall/ipset fixes the problem.
Comment 26 Daniel Drake (RETIRED) gentoo-dev 2009-01-30 15:03:37 UTC
ah, that explains it
Comment 27 Peter Volkov (RETIRED) gentoo-dev 2009-01-31 16:21:26 UTC
Jochen I just added to the tree ipset-2.4.7. Could you verify that it does not crash 2.6.28 kernel?
Comment 28 Jochen Schlick 2009-02-03 00:08:29 UTC
installed, firewall restarted, ipset-modules are loaded  -  no crash.


Comment 29 Peter Volkov (RETIRED) gentoo-dev 2009-02-03 09:00:28 UTC
(In reply to comment #28)
> installed, firewall restarted, ipset-modules are loaded  -  no crash.

Thank you, Jochen. I think we should stabilize ipset-2.4.7 together with 2.6.28 kernel. Keeping this bug open untill then.
Comment 30 Peter Volkov (RETIRED) gentoo-dev 2009-02-03 13:54:40 UTC
This bug became quite long, so I've requested stabilization in bug 257483. Closing.