Bug 511510 – Kernel swaplock & D state hang results from using tcpdump on loaded system

• Product: Gentoo Linux
• Status: RESOLVED NEEDINFO

[Description Alex Cannon 2014-05-26 02:18:34 UTC]

A non permanent deadlock occurs in Linux 3.0.76-gentoo that is triggered by tcpdump. When running tcpdump, it immediately goes in to the "D" state and cannot be killed. iptraf-ng doesn't cause this. The system has 512MB of RAM, and during this time up to 250MB of RAM will show as "free" according to the "free" command, as if some kind of invisible kernel buffer is taking up this system RAM. This is a busy server with many different users and processes, and normally about 10MB of RAM is free, and 600MB of swap is in use. After about an hour, tcpdump can then be killed. If I suspend a bunch of processes and free up RAM, then I can get tcpdump killed in just a few minutes. Apparently, the kernel needs to finish whatever it is doing, and it needs RAM to do it.

Attached below is:
A normal copy of /proc/meminfo 
A copy of /proc/meminfo while tcpdump is deadlocked, showing unusual free memory.
3 Stack traces. 2 stack traces of tcpdump during the deadlock, and one from the dmesg 120 second deadlock warning.


Reproducible: Always

Steps to Reproduce:
1. Run tcpdump, on any interface including "lo".

Actual Results:  
tcpdump gets deadlocked in the "D" state.

Expected Results:  
Sending SIGKILL to tcpdump should terminate it.

[Comment 1 Alex Cannon 2014-05-26 02:20:07 UTC]

Portage 2.2.8-r1 (default/linux/x86/13.0, gcc-4.5.4, glibc-2.17, 3.0.76-gentoo i686)
=================================================================
System uname: Linux-3.0.76-gentoo-i686-Pentium_III_-Coppermine-with-gentoo-2.2
KiB Mem:      511516 total,     15080 free
KiB Swap:    2521068 total,   1842060 free
Timestamp of tree: Tue, 13 May 2014 05:30:01 +0000
ld GNU ld (GNU Binutils) 2.22
app-shells/bash:          4.2_p45
dev-java/java-config:     2.2.0
dev-lang/python:          2.6.8-r1, 2.7.5-r3, 3.1.5-r1, 3.2.5-r3, 3.3.3
dev-util/cmake:           2.8.9
dev-util/pkgconfig:       0.28
sys-apps/baselayout:      2.2
sys-apps/openrc:          0.11.8
sys-apps/sandbox:         2.6-r1
sys-devel/autoconf:       2.13, 2.69
sys-devel/automake:       1.11.6, 1.12.6, 1.13.4
sys-devel/binutils:       2.22-r1
sys-devel/gcc:            3.4.6-r2, 4.3.6-r1, 4.5.4
sys-devel/gcc-config:     1.7.3
sys-devel/libtool:        2.4-r1
sys-devel/make:           3.82-r4
sys-kernel/linux-headers: 3.7 (virtual/os-headers)
sys-libs/glibc:           2.17
Repositories: gentoo
ACCEPT_KEYWORDS="x86"
ACCEPT_LICENSE="* -@EULA Oracle-BCLA-JavaSE IBM-J1.6"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-Os -march=pentium3 -pipe"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-Os -march=pentium3 -pipe"
DISTDIR="/usr/portage/distfiles"
FCFLAGS="-O2 -march=i686 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync"
FFLAGS="-O2 -march=i686 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j1"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY=""
USE="7zip X a52 aac acc acl alsa apache2 berkdb bittorrent bzip2 chappa cli cracklib crypt cups cxx dbus dga divx4linux doc dri dvd dvdread emacs encode esd esound extras faad ffmpeg fltk fontconfig fortran gd gdbm gif gimp gnutella gtk html hwdb iconv imagemagick imap imlib ipv6 jadetex javascript jpeg kdeenablefinal live mad mmx mmxext mng modules motif mp2 mp3 mpeg multinet ncurses nls nptl offensive ogg oident openft openmp pam pcre png pnm proxy_http python qt3support readline scripting session snmp speex sse ssl suexec tcpd theora tiff truetype unicode usb vim-syntax vorbis x264 x86 xa xinetd xv xvid zlib" ABI_X86="32" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias ident proxy proxy_connect proxy_ftp proxy_http" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_3" RUBY_TARGETS="ruby19 ruby20" USERLAND="GNU" VIDEO_CARDS="fbdev glint intel mach64 mga nouveau nv r128 radeon savage sis tdfx trident vesa via vmware dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, SYNC, USE_PYTHON

[Comment 2 Alex Cannon 2014-05-26 02:22:43 UTC]

Created attachment 377642 [details]
Normal /proc/meminfo

[Comment 3 Alex Cannon 2014-05-26 02:25:00 UTC]

Created attachment 377644 [details]
During deadlock, unusual free memory shows up

[Comment 4 Alex Cannon 2014-05-26 02:29:27 UTC]

Created attachment 377646 [details]
Stack traces of tcpdump while deadlocked

During the deadlock, tcpdump is hardly using any memory according to top.

[Comment 5 Alex Cannon 2014-05-26 23:27:03 UTC]

I forgot to mention it's tcpdump-4.3.0. Tried 4.5.1 same issue. Maybe there is a bug related to operating system buffer size (tcpdump -B option) that makes it too big? iptraf-ng works perfectly. I realize the word deadlock isn't correct to describe it, it's just a D state hang that's causing unusually memory consumption or paging.

[Comment 6 Mike Pagano 2014-12-22 20:20:35 UTC]

Is this still an issue with later kernels?

[Comment 7 Alex Cannon 2015-04-27 23:33:45 UTC]

It just happened again using Linux-3.2.67 vanilla. Starting tcpdump causes almost 500MB of RAM to become free. It seems to only happen when memory is fairly loaded. Now the system has 1.25GB in it and about 300MB of swap is used, which is enough to trigger this bug.