BUG: unable to handle kernel NULL pointer dereference at 0000000000000000 IP: [<ffffffffa0ce801c>] 0xffffffffa0ce801c PGD 137051067 PUD 137c01067 PMD 0 Oops: 0000 [#1] SMP last sysfs file: /sys/devices/system/cpu/cpu1/cache/index2/shared_cpu_map CPU 0 Modules linked in: iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_TTL ipt_ttl ipt_set ipt_REJECT ipt_REDIRECT ipt_NETMAP ipt_MASQUERADE ipt_LOG ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp ip_set_portmap ip_set_macipmap ip_set_ipmap ip_set_iphash ip_set xt_tcpmss xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_conntrack xt_CONNMARK xt_connmark xt_CLASSIFY xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack iptable_mangle nfnetlink iptable_filter ip_tables x_tables aes_x86_64 aes_generic bridge stp llc snd_pcm_oss snd_mixer_oss snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device ipv6 coretemp btusb uvcvideo bluetooth nvidiafb compat_ioctl32 videodev fb_ddc i2c_algo_bit vgastate v4l1_compat nvidia(P) firewire_ohci firewire_core snd_hda_intel arc4 ecb ohci1394 iwlagn uhci_hcd ehci_hcd iwlcore snd_pcm intel_agp sg sdhci_pci snd_timer ieee1394 usbcore r8169 snd_page_alloc sdhci mac80211 mmc_core i2c_core mii video pcspkr serio_raw snd_hwdep ac cfg80211 output button snd joydev battery Pid: 6900, comm: iptables Tainted: P 2.6.28-gentoo #2 RIP: 0010:[<ffffffffa0ce801c>] [<ffffffffa0ce801c>] 0xffffffffa0ce801c RSP: 0018:ffff880133619c98 EFLAGS: 00010292 RAX: ffffffffa0ce8010 RBX: 0000000000000000 RCX: 0000000000000000 RDX: 0000000000000020 RSI: 0000000000000020 RDI: ffff880133619d48 RBP: ffff880133619ca8 R08: 0000000000000000 R09: 00000000fffffffe R10: 0000000000000130 R11: 0000000000000000 R12: 0000000000000000 R13: ffff880133619d48 R14: ffffc2001225130a R15: 0000000000000000 FS: 00007f99cdae86f0(0000) GS:ffffffff80867d80(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000000133672000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process iptables (pid: 6900, threadinfo ffff880133618000, task ffff8801393c2c20) Stack: ffff880133619ce8 ffffffffa0ce84a0 ffff880133619ce8 ffffffffa0b0da95 00ff880137e158b0 ffff8801337a9d80 ffffc20012251308 ffffc20012250000 ffffc2001225130a 0000000000000070 ffff880133619da8 ffffffffa0b1564e Call Trace: [<ffffffffa0b0da95>] xt_check_match+0xf5/0x180 [x_tables] [<ffffffffa0b1564e>] translate_table+0x30e/0x680 [ip_tables] [<ffffffffa0b16924>] do_ipt_set_ctl+0x124/0x1a0 [ip_tables] [<ffffffff805b83b0>] nf_sockopt+0x60/0xa0 [<ffffffff805b842c>] nf_setsockopt+0x1c/0x20 [<ffffffff805c5ad2>] ip_setsockopt+0xa2/0xb0 [<ffffffff805e23e5>] raw_setsockopt+0x25/0x70 [<ffffffff8058c53f>] sock_common_setsockopt+0xf/0x20 [<ffffffff8058a06b>] sys_setsockopt+0x7b/0xe0 [<ffffffff8020c2db>] system_call_fastpath+0x16/0x1b Code: <0f> b7 39 e8 6c a0 ef ff 66 ff c0 74 13 8b 53 1c b8 01 00 00 00 85 RIP [<ffffffffa0ce801c>] 0xffffffffa0ce801c RSP <ffff880133619c98> CR2: 0000000000000000 ---[ end trace 32af231d28b68844 ]--- Reproducible: Always
Could you attach your kernel config and your lsmod and dmesg output? Thanks :)
Created attachment 177792 [details] kernel config
Created attachment 177793 [details] dmesg output (after shorewall/iptables start)
Created attachment 177795 [details] lsmod output
It would, also, be very helpful if you followed the steps below to isolate the line of code that is causing the oops. The instructions have to be followed in an identical kernel as the one that causes the crashes. # emerge -n gdb # cd /usr/src/linux #or wherever your kernel lies # rm net/netfilter/x_tables.o # make CONFIG_DEBUG_INFO=y net/netfilter/x_tables.o # gdb net/netfilter/x_tables.o Then at the gdb prompt: # list *xt_check_match+0xf5 Then just post the output of the above command here! Thanks :)
hope this helps /usr/src/linux-2.6.28-gentoo # gdb net/netfilter/x_tables.o GNU gdb 6.8 Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu"... (gdb) list *xt_check_match+0xf5 0x1a95 is in xt_check_match (net/netfilter/x_tables.c:357). 352 printk("%s_tables: %s match: only valid for protocol %u\n", 353 xt_prefix[par->family], par->match->name, 354 par->match->proto); 355 return -EINVAL; 356 } 357 if (par->match->checkentry != NULL && !par->match->checkentry(par)) 358 return -EINVAL; 359 return 0; 360 } 361 EXPORT_SYMBOL_GPL(xt_check_match); (gdb)
Great output thanks. Please make sure the below lines are set on /etc/shorewall/shorewall.conf file LOGFILE=/var/log/messages LOGFORMAT="Shorewall:%s:%s:" If these line are like this already, attach your /var/log/messages. If they are not,bootup your system without starting shorewall. After booting successfully, run the shorewall by executing /etc/init.d/shorewall start The your system , should crash. If so, reboot and attach here your /var/log/messages Thank you
Also, which is the last known working kernel?
sorry for the delay, but the thinkpad is not available at the moment (until tomorrow morning). but the current working kernel on this thinkpad is: linux-2.6.27-gentoo-r7 best regards
Created attachment 177991 [details] /var/log/messages after /etc/init.d/shorewall start there is no output from shorewall in /var/log/messages - only from iptables and the concerning oops after manual start of shorewall....
Please create a file at /usr/local/bin/iptables_log and put this inside it: ----------- #!/bin/bash echo "$(date) $$ $*" >> /tmp/iptables_log exec /sbin/iptables $* ----------- make it executable: chmod a+x /usr/local/bin/iptables_log Then in shorewall.conf add: IPTABLES=/usr/local/bin/iptables_log Then start shorewall on an affected kernel, reproducing the crash. Post the new crash dump, and the contents of /usr/local/bin/iptables_log to this bug. Thanks!
(In reply to comment #11) > Then start shorewall on an affected kernel, reproducing the crash. Post the new > crash dump, and the contents of /usr/local/bin/iptables_log to this bug. > Thanks! Should have read: > Then start shorewall on an affected kernel, reproducing the crash. Post the new > crash dump, and the contents of /tmp/iptables_log to this bug.
Created attachment 178020 [details, diff] debug patch At the same time, please have this patch applied to 2.6.28 to add some more debug output. When including the updated crash dump per the instructions above, please also include the few preceding lines in the kernel log (which is what this patch will add)
ok, first of all shorewall expected then also a "iptables_log-restore" in /usr/local/bin, so I created one (a symlink to /sbin/iptables-restore). I hope this is correct. I built a new kernel (2.6.28-gentoo.debug), rebooted several times but no oops :-( (still investigating)
Created attachment 178083 [details] iptables log of the new patched kernel - nocrash :-(
Created attachment 178084 [details] messages of the new patched kernel - nocrash :-(
Created attachment 178087 [details] iptables log of the unpatched kernel
Created attachment 178089 [details] messages of the unpatched kernel
How strange. The fact that the debug patch changes the behaviour probably means that it is a timing-sensitive issue. The command it is crashing on is: iptables -A fooX1594 -m set --set fooX1594 src -j ACCEPT I think we should take this upstream as the next step. Before we do that, we need to reproduce this on an unpatched, untainted kernel. Please can you reproduce this on vanilla-sources-2.6.28, without the nvidia module, and post a new crash dump?
Created attachment 178271 [details] messages of a 2.6.28 vanilla kernel (with nvidia) sys-kernel/vanilla-sources 2.6.28 created with genkernel crashes also
Thanks. That one still had the nvidia driver loaded though. Please configure your system so that the nvidia module doesn't get loaded at all, and post a new crash dump.
Created attachment 178287 [details] messages of a 2.6.28 vanilla kernel (no nvidia) not so fast ;-) I removed the nvidia kernel module and called depmod for the vanilla kernel and it oopses again with the untainted kernel. PS: I write this comment on the same laptop running the 2.6.28-gentoo.debug kernel and still no crash with this kernel.
Perfect, thanks. Please now file an upstream bug for this at http://bugzilla.kernel.org. My suggestions: - emphasize that this is a 2.6.28 regression (2.6.27 worked) - note the iptables command which we have determined to be the trigger for the crash (explained above) - attach your most recent dmesg there (non-tainted kernel with crash message) I'll add some more technical details after you have done that. Please post the new bug URL here when done, thanks!
ok I created one on bugzilla.kernel.org http://bugzilla.kernel.org/show_bug.cgi?id=12517
The reason for the oops was an incompatibility between the 2.6.28 kernels and the net-firewall/ipset package !! deinstallation of /net-firewall/ipset fixes the problem.
ah, that explains it
Jochen I just added to the tree ipset-2.4.7. Could you verify that it does not crash 2.6.28 kernel?
installed, firewall restarted, ipset-modules are loaded - no crash.
(In reply to comment #28) > installed, firewall restarted, ipset-modules are loaded - no crash. Thank you, Jochen. I think we should stabilize ipset-2.4.7 together with 2.6.28 kernel. Keeping this bug open untill then.
This bug became quite long, so I've requested stabilization in bug 257483. Closing.