Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 506882 - net-firewall/shorewall with kernel 3.13, 3.14 - panic in ipt_do_table in [ip_tables]
Summary: net-firewall/shorewall with kernel 3.13, 3.14 - panic in ipt_do_table in [ip_...
Status: RESOLVED UPSTREAM
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Core system (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Gentoo Kernel Bug Wranglers and Kernel Maintainers
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-04-06 04:02 UTC by Reuben Martin
Modified: 2014-08-17 04:40 UTC (History)
3 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
emerge info (emerge_info.txt,6.19 KB, text/plain)
2014-04-06 04:03 UTC, Reuben Martin
Details
Shorewall emerge info (emerge_shorewall_info.txt,6.54 KB, text/plain)
2014-04-07 01:54 UTC, Reuben Martin
Details
shorewall config files archive (shorewall.tar.gz,11.59 KB, application/x-gzip)
2014-04-07 01:55 UTC, Reuben Martin
Details
kernel-3.14.0 config (kernel_config,96.77 KB, text/plain)
2014-04-07 01:56 UTC, Reuben Martin
Details
Kernel Panic Photo (ScreenShot_Cropped.jpg,781.26 KB, image/jpeg)
2014-04-07 02:00 UTC, Reuben Martin
Details
iptables (iptables.txt,13.30 KB, text/plain)
2014-04-09 23:24 UTC, Reuben Martin
Details
iptables6 (ip6tables.txt,11.17 KB, text/plain)
2014-04-09 23:24 UTC, Reuben Martin
Details
hardware (hardware.txt,33.21 KB, text/plain)
2014-04-09 23:25 UTC, Reuben Martin
Details
dmesg (dmesg.txt,70.36 KB, text/plain)
2014-04-09 23:25 UTC, Reuben Martin
Details
kernel modules (modules.txt,4.31 KB, text/plain)
2014-04-09 23:55 UTC, Reuben Martin
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Reuben Martin 2014-04-06 04:02:58 UTC
I had been using kernel 3.12.x until todate when I wanted to try out 3.14. I have shorewall (shorewall-core + shorewall + shorewall6) set up on my system. However it appears to crash the kernel. I had origionally enabled the new nftables as a module and though that might be causing issues, but the same thing happens even after disabling nftables and rebuilding.

Same thing happened trying to test with 3.13.

If I turn off the shorewall services, the system boots fine and otherwise works as expected.

Reproducible: Always
Comment 1 Reuben Martin 2014-04-06 04:03:29 UTC
Created attachment 374358 [details]
emerge info
Comment 2 Jeroen Roovers (RETIRED) gentoo-dev 2014-04-06 13:45:04 UTC
What does really happen, then? Does the kernel panic? Does the init system hang? What happens if you start shorewall after boot time?
Comment 3 Reuben Martin 2014-04-06 15:08:15 UTC
Kernel panic. Starting manually after system is finished loading makes no difference.
Comment 4 Jeroen Roovers (RETIRED) gentoo-dev 2014-04-06 15:16:08 UTC
1) Please post your `emerge --info net-firewall/shorewall' output in a comment.
2) Userland tools shouldn't be able to do that, but we still don't have enough information. Could you get a screenshot or write down the messages?
Comment 5 Reuben Martin 2014-04-07 01:54:33 UTC
Created attachment 374442 [details]
Shorewall emerge info
Comment 6 Reuben Martin 2014-04-07 01:55:26 UTC
Created attachment 374444 [details]
shorewall config files archive
Comment 7 Reuben Martin 2014-04-07 01:56:03 UTC
Created attachment 374446 [details]
kernel-3.14.0 config
Comment 8 Reuben Martin 2014-04-07 02:00:17 UTC
Created attachment 374448 [details]
Kernel Panic Photo
Comment 9 Thomas Deutschmann (RETIRED) gentoo-dev 2014-04-09 14:48:54 UTC
I agree with Jeroen. shorewall is unable to cause this crash (shorewall is just a perl script which will call "iptables[6]" commands). This crash must be related to kernel and or net-firewall/iptables.

Can you please attache a dmesg, too? I am interested in your network hardware/driver.

Does your system always immediately crash when you start shorewall (so is this reproducible) or when does this happen?
Comment 10 Jeroen Roovers (RETIRED) gentoo-dev 2014-04-09 14:57:35 UTC
(In reply to Thomas D. from comment #9)
> Does your system always immediately crash when you start shorewall (so is
> this reproducible) or when does this happen?

Comment #3.
Comment 11 Thomas Deutschmann (RETIRED) gentoo-dev 2014-04-09 16:03:42 UTC
@ Reuben: What are the last lines you see after you run

  # shorewall start

(right, "shorewall start", don't use the runscript "/etc/init.d/shorewall"...)

If it will crash before you see a "done." line, please restart the system and run

  # shorewall trace start

This should give us a detailed output about the command which is causing the problems...

I'll try to reproduce the problem tomorrow.
Comment 12 Reuben Martin 2014-04-09 23:23:23 UTC
I'm aware that shorewall itself is not causing the crash. I just filed it under shorewall because that's the quickest way for me to reproduce it. There's lot of boiler plate rules that it generates, and I have no idea which rule (or combination of rules) triggers the panic. I'm willing to bet the issue stems from a bug introduced with the new nftables added in 3.13.

The panic is reproducible and consistently happens within a little over a second after shorewall finishes compiling the rules (i.e. reaches the "done" message).

I'm uploading the iptables as they are shown from "iptables(6) -L -v -n" under kernel 3.12.14 as well as the ouput from dmesg booting 3.14.0 with shorewall(6) disabled. And also hardware info from lshw.
Comment 13 Reuben Martin 2014-04-09 23:24:03 UTC
Created attachment 374632 [details]
iptables
Comment 14 Reuben Martin 2014-04-09 23:24:45 UTC
Created attachment 374634 [details]
iptables6
Comment 15 Reuben Martin 2014-04-09 23:25:10 UTC
Created attachment 374636 [details]
hardware
Comment 16 Reuben Martin 2014-04-09 23:25:35 UTC
Created attachment 374638 [details]
dmesg
Comment 17 Thomas Deutschmann (RETIRED) gentoo-dev 2014-04-09 23:38:37 UTC
I upgraded my test system to 3.14 (I used my configuration from 3.10 - I didn't make any adjustments, just "make oldconfig" from genkernel... so  CONFIG_NF_TABLES is not set, same like you). I am currently unable to reproduce your problem.

Could you please test with only shorewall (no shorewall6, so no iptables6...)? So we don't need to check if any IPv6 stuff is involved..

Also, from reading your rules file it looks like you are using  the xt_geoip module from net-firewall/xtables-addons. Could you please remove the rule for testing so we can make sure it is not the xt_geoip module...?
Comment 18 Reuben Martin 2014-04-09 23:41:28 UTC
Shorewall only loads ip4 rules. And that is what has been causing the panic. (when I start shorewall manually, shorewall6 has not yet been started)
Comment 19 Reuben Martin 2014-04-09 23:54:36 UTC
Ok, I disabled both the rule in the "rules" config, and well as the "GEOIPDIR" option in "shorewall.conf"/

Kernel still panics, with less delay.

I'm add the output of lsmod from under 4.12.14 with shorewall started, so that should give an idea of which modules are actually getting use.
Comment 20 Reuben Martin 2014-04-09 23:55:00 UTC
Created attachment 374640 [details]
kernel modules
Comment 21 Thomas Deutschmann (RETIRED) gentoo-dev 2014-04-10 00:30:14 UTC
I am now running your shorewall configuration (=this should load the same modules) and my system doesn't panic.

- Remove all the rules using helpers (SMBBI, mDNSbi..)
 - If this is not enough, remove all the rules - does it crash with an empty rules file? If not, re-add rule for rule...

- Maybe your system panic when a helper module will get traffic? Maybe you can temporarily isolate the system (pull the cables) to see if it won't panic if it won't see traffic...

However, try to get a crash dump.

- Maybe ipt_do_tables+0x144 is already enough to use objdump on the iptabels module to see something...

- You said 3.12.x was working for you. Maybe you can do a kernel bisect?
Comment 22 Reuben Martin 2014-04-10 04:38:43 UTC
Well the rabbit hole goes deeper...

I disabled all directives in the rules config, and unplugged the network cables before booting.

No dice, still panics.

I'm going to try and see if I can crash it directly tomorrow with iptables.

I've never tried to do kernel debugging. Where can I read up on how to obtain a crash dump? The panic error messages do consistently reference ipt_do_tables+0x144. What do I do with objdump?

I've heard kernel bisect can take a considerable amount of time.. So that seems like a last resort. However I've not done that either. I assume the rebuild / testing processes are automated somehow?
Comment 23 Reuben Martin 2014-04-10 04:42:11 UTC
I also forgot to ask, do I need to rebuild with some debug options enabled to make any sort of dump useful? My kernel config currently only has the bare minimum of debug options enabled.
Comment 24 Thomas Deutschmann (RETIRED) gentoo-dev 2014-04-10 15:10:29 UTC
Can you try without ebtables, too?

You cannot replace the r8169 driver, can you? E.g. use a different NIC (and disable this driver) or use the driver from http://www.realtek.com.tw/downloads/downloadsView.aspx?Langid=1&PNid=13&PFid=4&Level=5&Conn=4&DownTypeID=3&GetDown=false&Downloads=true ? This is just a feeling but if you search for r8169 and kernel panic you will see some reports for 3.13+ like https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=741667

Regarding kernel debugging/what you need in your kernel see https://wiki.gentoo.org/wiki/Kernel_Crash_Dumps

Regarding the use of objdump see https://wiki.ubuntu.com/Kernel/KernelDebuggingTricks#Objdump

Regarding bisect see https://wiki.gentoo.org/wiki/Kernel_git-bisect - It will take some time but it is not that hard like it sounds.
Comment 25 Reuben Martin 2014-04-13 23:30:26 UTC
Well... I would have posted a kernel dump by now, if the kernel didn't panic after being started the second time.

It might be because I'm loading the same kernel as the secondary. I'll try building a 3.12 debug kernel to load as the backup kernel and see if that helps.
Comment 26 Reuben Martin 2014-08-17 04:40:45 UTC
This has since been fixed as of kernel 3.16.