Bug 214646 – net-firewall/shorewall hindering NFS4 Performance

• Product: Gentoo Linux
• Status: RESOLVED NEEDINFO

[Description Roger 2008-03-25 07:04:57 UTC]

I believe this is the second time I've encountered this issue within the past 6 mos or more.  While performing some read/write tests between a NFS client and server, I noticed a severe speed was severely hindered on my 100mbits network with tests of writing a ~17MB file taking ~15 minutes, of which, should only take ~8-10 seconds.  

After restarting Shorewall (firewall), speeds immediately returned to normal.  Steps I took to restart the server were "/etc/init.d/shorewall stop && shorewall clear && /etc/init.d/shorewall start".  (I was trying to see if the configuration files were directly hindering performance.  Upon restarting shorewall, no speed issues were noted.)

This leads me to believe, Shorewall is borking someplace -- or more correctly put, one of the kernel modules concerning Netfilter is failing?


Reproducible: Always




Please note, I have no idea what is spawning this issue or how often it is occurring.  Hopefully, filing this bug, will help me remember I've seen this bug before!!!

[Comment 1 Roger 2008-03-28 07:43:41 UTC]

=sys-kernel/tuxonice-sources-2.6.24-r3

And, although probably not too relevant, the following versions:
=net-fs/nfs-utils-1.1.1
=net-libs/libnfsidmap-0.16
=net-libs/librpcsecgss-0.16
=net-libs/libgssglue-0.1


One thought does hit me, maybe one too many suspends of the kernel provokes this?

[Comment 2 Vieri 2008-03-31 10:35:57 UTC]

Does it only happen with NFS?
Can you test the same with another protocol?
Which shorewall version/compiler are you using?

Anyway, if you think that shorewall (or netfilter) is borked then it would be really useful if you could post upstream to the shorewall mailing list. And then post back here the result (if any) to help out others with the same/similar issue.

Thanks for your report.

[Comment 3 Roger 2008-03-31 17:37:42 UTC]

In process of posting upstream.

[Comment 4 Roger 2008-04-01 19:19:19 UTC]

Here's what I got from the shorewall mailing list:

Better problem documentation would include:

a) Output of "shorewall dump" at the time that the problem was encountered.
b) Output of "shorewall dump" after recovery measures were taken and 
performance restored.

"My bet is on a kernel, hardware, or network fabric issue that was
shaken loose by the brief traffic interruption that occurs when
shorewall starts. It could also be a traffic shaping corner case (some
flawed configurations have cascade failure modes, where everything
seizes up for certain traffic patterns but not others, and the effect
is self-continuing)."

Other significant data points:

 - what's on the network when the problem occurs? Steady-but-slow NFS
 traffic, fast-but-rare bursts, collisions, corrupted packets,
 something else entirely? Whenever I see something like this, I hit
 tcpdump -w first and think about it later; a packet dump explains
 most issues when you can study it at leisure.

 - relevant /proc/mounts entries. Do these change when the problem occurs?

 - physical network configuration

 - vmstat output when the problem occurs

But I doubt it's a shorewall problem.  (...since shorewall is only a frontend to configuring the kernel's netfilter parameters.)


My gut feeling, if I can remember in >6 months the next time I see this issue, is a "shorewall dump" before & after will more then likely show the source of the issue.  Until then, I'm sitting, waiting, & just hoping I can remember to do a "shorewall dump" prior to resorting to the last ditch effort of restarting shorewall!

Which is best for this issue: needinfo, remind, later?  (I usually search for any relevant open NFS bugs prior to debugging.)

[Comment 5 Vieri 2008-04-02 07:45:12 UTC]

Thanks for the feedback. I'm sure some Gentoo users will find it useful.

Please post back within a few months if you can. Meanwhile I'm CC'ing Peter about this bug's status (maybe severity and priority should be lowered; however, I wouldn't close the bug and re-open it when you witness the issue again because I think that your experience can help others).

[Comment 6 Peter Volkov (RETIRED) 2008-04-02 08:26:08 UTC]

Adding netmon in loop, just in case...

Roger, thank you for report. Until you'll manage to dig any additional information it's worth to resolve this bug as NEEDINFO. So, please, reopen as soon as you get any additional data.

[Comment 7 Roger 2008-04-02 09:05:18 UTC]

Your welcome.

(Saving this puppy as a custom saved search should help remind me.  If I don't see this after a 12 months, I'll close it.)