It is possibly related to gentoo-sources OR open-vmware-tools. Here are some details: #uname -a Linux test 3.0.6-gentoo-mrbyte #1 SMP Wed Oct 5 02:14:44 CEST 2011 x86_64 Dual-Core AMD Opteron(tm) Processor 2214 HE AuthenticAMD GNU/Linux The problem started appearing soon after kernel update from 3.0.4-r2 to 3.0.6 and open-vm-tools to 0.0.20110923.491607. After a while, but not at once since the system boot "emerge" and other processes go into "D" (uninterruptable sleep) state forever. #ps auxww | grep -P "\sD" root 1046 0.0 0.0 104660 15092 ? D 01:00 0:00 /usr/bin/python2.7 /usr/bin/emerge -q --sync root 1047 0.0 0.0 4216 324 ? D 01:00 0:00 mkdir -p /var spool/cron/lastrun root 2596 0.0 0.0 4216 320 ? D 01:10 0:00 mkdir -p /var spool/cron/lastrun root 3346 0.0 0.0 106684 15008 ? D 01:13 0:00 /usr/bin/python2.7 /usr/bin/emerge -upD world root 4078 0.0 0.0 20652 2652 ? Ds 01:16 0:00 -bash root 4111 0.0 0.0 25620 1304 ? Ds 01:18 0:00 /sbin/rc reboot root 4159 0.0 0.0 4216 324 ? D 01:20 0:00 mkdir -p /var spool/cron/lastrun root 5750 0.0 0.0 37768 3712 ? D 01:30 0:00 mcedit /usr/sbin/run-crons root 5784 0.0 0.0 20652 2572 ? Ds 01:31 0:00 -bash root 5821 0.0 0.0 20548 2532 ? Ds 01:33 0:00 -bash root 5838 0.0 0.0 36948 2816 ? D 01:34 0:00 mcedit /root/scripts/nightly-maintenance.sh root 6095 0.1 0.0 133044 3068 ? Dsl Oct19 1:31 /usr/bin/vmtoolsd root 6118 0.0 0.0 19240 1072 ? D 01:58 0:00 fsck.ext3 /dev/sda2 root 6146 0.0 0.0 20548 2532 ? Ds 02:01 0:00 -bash root 6165 0.0 0.0 19240 1076 ? D 02:02 0:00 fsck.ext3 /dev/sda2 root 6248 0.0 0.0 20660 2656 ? Ds 02:10 0:00 -bash root 6826 0.0 0.0 0 0 tty2 Ds+ Oct19 0:00 [agetty] root 12863 0.0 0.0 20652 2556 ? Ds 09:21 0:00 -bash root 13682 0.0 0.0 20680 2492 ? Ds 09:46 0:00 -bash root 13736 0.0 0.0 20548 2544 ? Ds 09:46 0:00 -bash root 15364 0.1 0.0 106664 15004 ? D 10:28 0:00 /usr/bin/python2.7 /usr/bin/emerge -vp open-vm-tools root 16785 0.0 0.0 4216 324 ? D Oct19 0:00 mkdir -p /var spool/cron/lastrun root 18285 0.0 0.0 4216 320 ? D Oct19 0:00 mkdir -p /var spool/cron/lastrun root 19800 0.0 0.0 4216 324 ? D Oct19 0:00 mkdir -p /var spool/cron/lastrun root 21412 0.0 0.0 4216 320 ? D Oct19 0:00 mkdir -p /var spool/cron/lastrun root 22922 0.0 0.0 4216 320 ? D Oct19 0:00 mkdir -p /var spool/cron/lastrun root 24422 0.0 0.0 4216 320 ? D 00:00 0:00 mkdir -p /var spool/cron/lastrun root 25946 0.0 0.0 4216 320 ? D 00:10 0:00 mkdir -p /var spool/cron/lastrun root 27406 0.0 0.0 4216 324 ? D 00:20 0:00 mkdir -p /var spool/cron/lastrun root 28981 0.0 0.0 4216 324 ? D 00:30 0:00 mkdir -p /var spool/cron/lastrun root 30502 0.0 0.0 4216 320 ? D 00:40 0:00 mkdir -p /var spool/cron/lastrun root 31972 0.0 0.0 4216 316 ? D 00:50 0:00 mkdir -p /var spool/cron/lastrun When I tried to ls /sys/kernel, my bash process also hung. I enabled some trace and here's dmesg output: [82201.186033] mcedit D 0000000103338366 0 5838 5821 0x00000000 [82201.186033] ffff88013920fdb8 0000000000000086 0000105000000000 0000000000011000 [82201.186033] 0000000000011000 0000000000011000 0000000000011000 ffff88013e47d000 [82201.186033] 0000000000011000 ffff88013920ffd8 ffff88013920ffd8 0000000000011000 [82201.186033] Call Trace: [82201.186033] [<ffffffff81033d5d>] schedule+0x55/0x57 [82201.186033] [<ffffffff813138df>] __mutex_lock_common.clone.5+0x11c/0x17b [82201.186033] [<ffffffff8131394c>] __mutex_lock_slowpath+0xe/0x10 [82201.186033] [<ffffffff813136e2>] mutex_lock+0x16/0x27 [82201.186033] [<ffffffff810c3089>] lookup_create+0x1e/0x8f [82201.186033] [<ffffffff810c647f>] sys_mkdirat+0x3d/0xe9 [82201.186033] [<ffffffff810c2dcd>] ? path_put+0x1d/0x22 [82201.186033] [<ffffffff810d1126>] ? mntput_no_expire+0x27/0xc8 [82201.186033] [<ffffffff810c653e>] sys_mkdir+0x13/0x15 [82201.186033] [<ffffffff81314cd2>] system_call_fastpath+0x16/0x1b [82201.186033] fsck.ext3 D 00000001034a4a6d 0 6118 6117 0x00000004 [82201.186033] ffff880428d93c88 0000000000000086 ffff880400000000 0000000000011000 [82201.186033] 0000000000011000 0000000000011000 0000000000011000 ffff8804286b0000 [82201.186033] 0000000000011000 ffff880428d93fd8 ffff880428d93fd8 0000000000011000 [82201.186033] Call Trace: [82201.186033] [<ffffffff81033d5d>] schedule+0x55/0x57 [82201.186033] [<ffffffff813138df>] __mutex_lock_common.clone.5+0x11c/0x17b [82201.186033] [<ffffffff810c2c21>] ? exec_permission+0x7c/0x8a [82201.186033] [<ffffffff8131394c>] __mutex_lock_slowpath+0xe/0x10 [82201.186033] [<ffffffff813136e2>] mutex_lock+0x16/0x27 [82201.186033] [<ffffffff810c4b54>] do_last.clone.24+0x170/0x531 [82201.186033] [<ffffffff810c5e96>] path_openat+0xc4/0x328 [82201.186033] [<ffffffff810c5ca9>] ? user_path_at+0x66/0x95 [82201.186033] [<ffffffff810c6132>] do_filp_open+0x38/0x84 [82201.186033] [<ffffffff810cf7cf>] ? alloc_fd+0x6f/0x119 [82201.186033] [<ffffffff810ba28b>] do_sys_open+0x6f/0x108 [82201.186033] [<ffffffff810ba33f>] sys_open+0x1b/0x1d [82201.186033] [<ffffffff81314cd2>] system_call_fastpath+0x16/0x1b [82201.186033] bash D ffff88042eb2a000 0 6146 1 0x00000004 [82201.186033] ffff88042df19cc8 0000000000000086 ffff880400000001 0000000000011000 [82201.186033] 0000000000011000 0000000000011000 0000000000011000 ffff88042a9ff000 [82201.186033] 0000000000011000 ffff88042df19fd8 ffff88042df19fd8 0000000000011000 [82201.186033] Call Trace: [82201.186033] [<ffffffff81033d5d>] schedule+0x55/0x57 [82201.186033] [<ffffffff813138df>] __mutex_lock_common.clone.5+0x11c/0x17b [82201.186033] [<ffffffff8131394c>] __mutex_lock_slowpath+0xe/0x10 [82201.186033] [<ffffffff813136e2>] mutex_lock+0x16/0x27 [82201.186033] [<ffffffff810818b3>] generic_file_aio_write+0x3f/0xb5 [82201.186033] [<ffffffff810967a5>] ? pmd_offset+0x14/0x3b [82201.186033] [<ffffffff810ba65b>] do_sync_write+0xc6/0x103 [82201.186033] [<ffffffff8113bf64>] ? security_file_permission+0x29/0x2e [82201.186033] [<ffffffff810bacad>] vfs_write+0xa7/0xfa [82201.186033] [<ffffffff810ba312>] ? do_sys_open+0xf6/0x108 [82201.186033] [<ffffffff810baea5>] sys_write+0x45/0x69 [82201.186033] [<ffffffff81314cd2>] system_call_fastpath+0x16/0x1b [82201.186033] fsck.ext3 D 00000001034d28fb 0 6165 6162 0x00000004 [82201.186033] ffff88042a165c88 0000000000000086 ffff880400000000 0000000000011000 [82201.186033] 0000000000011000 0000000000011000 0000000000011000 ffff8804248f0000 [82201.186033] 0000000000011000 ffff88042a165fd8 ffff88042a165fd8 0000000000011000 [82201.186033] Call Trace: [82201.186033] [<ffffffff81033d5d>] schedule+0x55/0x57 [82201.186033] [<ffffffff813138df>] __mutex_lock_common.clone.5+0x11c/0x17b [82201.186033] [<ffffffff810c2c21>] ? exec_permission+0x7c/0x8a [82201.186033] [<ffffffff8131394c>] __mutex_lock_slowpath+0xe/0x10 [82201.186033] [<ffffffff813136e2>] mutex_lock+0x16/0x27 [82201.186033] [<ffffffff810c4b54>] do_last.clone.24+0x170/0x531 [82201.186033] [<ffffffff810c5e96>] path_openat+0xc4/0x328 [82201.186033] [<ffffffff810c5ca9>] ? user_path_at+0x66/0x95 [82201.186033] [<ffffffff810c6132>] do_filp_open+0x38/0x84 [82201.186033] [<ffffffff810cf7cf>] ? alloc_fd+0x6f/0x119 [82201.186033] [<ffffffff810ba28b>] do_sys_open+0x6f/0x108 [82201.186033] [<ffffffff810ba33f>] sys_open+0x1b/0x1d [82201.186033] [<ffffffff81314cd2>] system_call_fastpath+0x16/0x1b As mentioned, these issues started appearing soon after kernel upgrade and/or open-vm-tools upgrade. However, I am not excluding the problem to be on the side of VMware/hardware. Would be lovely to get some feedback on this nasty issue. Perhaps, this issue can also be escalated over to Linux kernel developers, I don't know. Andrew Reproducible: Always Steps to Reproduce: Rebooting Linux does not immediately cause any problems. I can run "emerge --sync" or "emerge -upD world" easily after reboot and install/update packages no problem. However, at night, during the cron job which runs "emerge -q --sync", the process goes into "D" state. So do some processes that I run from bash. Actual Results: The below processes are hanging: #ps auxww | grep -P "\sD" root 1046 0.0 0.0 104660 15092 ? D 01:00 0:00 /usr/bin/python2.7 /usr/bin/emerge -q --sync root 1047 0.0 0.0 4216 324 ? D 01:00 0:00 mkdir -p /var spool/cron/lastrun root 2596 0.0 0.0 4216 320 ? D 01:10 0:00 mkdir -p /var spool/cron/lastrun root 3346 0.0 0.0 106684 15008 ? D 01:13 0:00 /usr/bin/python2.7 /usr/bin/emerge -upD world root 4078 0.0 0.0 20652 2652 ? Ds 01:16 0:00 -bash root 4111 0.0 0.0 25620 1304 ? Ds 01:18 0:00 /sbin/rc reboot root 4159 0.0 0.0 4216 324 ? D 01:20 0:00 mkdir -p /var spool/cron/lastrun root 5750 0.0 0.0 37768 3712 ? D 01:30 0:00 mcedit /usr/sbin/run-crons root 5784 0.0 0.0 20652 2572 ? Ds 01:31 0:00 -bash root 5821 0.0 0.0 20548 2532 ? Ds 01:33 0:00 -bash root 5838 0.0 0.0 36948 2816 ? D 01:34 0:00 mcedit /root/scripts/nightly-maintenance.sh root 6095 0.1 0.0 133044 3068 ? Dsl Oct19 1:31 /usr/bin/vmtoolsd root 6118 0.0 0.0 19240 1072 ? D 01:58 0:00 fsck.ext3 /dev/sda2 root 6146 0.0 0.0 20548 2532 ? Ds 02:01 0:00 -bash root 6165 0.0 0.0 19240 1076 ? D 02:02 0:00 fsck.ext3 /dev/sda2 root 6248 0.0 0.0 20660 2656 ? Ds 02:10 0:00 -bash root 6826 0.0 0.0 0 0 tty2 Ds+ Oct19 0:00 [agetty] root 12863 0.0 0.0 20652 2556 ? Ds 09:21 0:00 -bash root 13682 0.0 0.0 20680 2492 ? Ds 09:46 0:00 -bash root 13736 0.0 0.0 20548 2544 ? Ds 09:46 0:00 -bash root 15364 0.1 0.0 106664 15004 ? D 10:28 0:00 /usr/bin/python2.7 /usr/bin/emerge -vp open-vm-tools root 16785 0.0 0.0 4216 324 ? D Oct19 0:00 mkdir -p /var spool/cron/lastrun root 18285 0.0 0.0 4216 320 ? D Oct19 0:00 mkdir -p /var spool/cron/lastrun root 19800 0.0 0.0 4216 324 ? D Oct19 0:00 mkdir -p /var spool/cron/lastrun root 21412 0.0 0.0 4216 320 ? D Oct19 0:00 mkdir -p /var spool/cron/lastrun root 22922 0.0 0.0 4216 320 ? D Oct19 0:00 mkdir -p /var spool/cron/lastrun root 24422 0.0 0.0 4216 320 ? D 00:00 0:00 mkdir -p /var spool/cron/lastrun root 25946 0.0 0.0 4216 320 ? D 00:10 0:00 mkdir -p /var spool/cron/lastrun root 27406 0.0 0.0 4216 324 ? D 00:20 0:00 mkdir -p /var spool/cron/lastrun root 28981 0.0 0.0 4216 324 ? D 00:30 0:00 mkdir -p /var spool/cron/lastrun root 30502 0.0 0.0 4216 320 ? D 00:40 0:00 mkdir -p /var spool/cron/lastrun root 31972 0.0 0.0 4216 316 ? D 00:50 0:00 mkdir -p /var spool/cron/lastrun There are no error messages in /var/log/messages, /var/log/syslog, and /var/log/kern.log.
Please post your `emerge --info CATEGORY/vmware-PKG' output too.
#emerge --info open-vm-tools Portage 2.1.10.29 (default/linux/amd64/10.0/server, gcc-4.5.3, glibc-2.13-r4, 3.0.4-gentoo-r2-mrbyte x86_64) ================================================================= System Settings ================================================================= System uname: Linux-3.0.4-gentoo-r2-mrbyte-x86_64-Dual-Core_AMD_Opteron-tm-_Processor_2214_HE-with-gentoo-2.1 Timestamp of tree: Thu, 20 Oct 2011 23:00:01 +0000 ccache version 3.1.6 [enabled] app-shells/bash: 4.2_p10 dev-lang/python: 2.7.2-r3, 3.2.2 dev-util/ccache: 3.1.6 dev-util/cmake: 2.8.6-r1 dev-util/pkgconfig: 0.26 sys-apps/baselayout: 2.1 sys-apps/openrc: 0.9.4 sys-apps/sandbox: 2.5 sys-devel/autoconf: 2.68 sys-devel/automake: 1.11.1-r1 sys-devel/binutils: 2.21.1-r1 sys-devel/gcc: 4.5.3-r1 sys-devel/gcc-config: 1.4.1-r1 sys-devel/libtool: 2.4-r4 sys-devel/make: 3.82-r3 sys-kernel/linux-headers: 2.6.39 (virtual/os-headers) sys-libs/glibc: 2.13-r4 Repositories: gentoo x-portage ACCEPT_KEYWORDS="amd64 amd64-fbsd x86 ~amd64 ~amd64-fbsd ~x86" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=opteron -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.2/ext-active/ /etc/php/apache2-php5.3/ext-active/ /etc/php/cgi-php5.2/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cli-php5.2/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-march=opteron -O2 -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="assume-digests binpkg-logs ccache distlocks ebuild-locks fixlafiles news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch" FFLAGS="" GENTOO_MIRRORS="http://distfiles.gentoo.org" LANG="POSIX" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LINGUAS="en ru" MAKEOPTS="-j5" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="3dnow 3dnowext \ acl acpi alias amd64 asis autoindex bash-completion bcmath bzip2 cache cgi clamav clflush cli cracklib crm114 crypt curl cxx dbus disk_cache domxml dri dspam expat extensions freetype ftp gd gdbm gettext glibc-omitfp gmp gpm hal hardened headers ht iconv imap ithreads jpeg kolab log_config maildir mailtrain mbstring mcrypt memcached memlimit mhash milter mime mimencode mm mmx mmxext mod_php modules mtrr mudflap multilib mysql mysqli ncurses nginx nls normalizemime nptl nptlonly openmp pam pcre pear png pop3d pppd qmail readline reiserfs session signature-log smp snmp sockets sqlite sse sse2 ssl sysfs syslog sysvipc tcpd threads tidymbstring tiff tokenizer truetype unicode userlocales utf8 valias vhost_alias vhosts vpopmail wddx xml xmlrpc xorg xsl xslt zlib zts" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions auth_basic authz_default authz_host authz_user authn_file alias autoindex deflate dir env expires filter headers log_config mime negotiation rewrite setenvif status" APACHE2_MPMS="event" CALLIGRA_FEATURES="kexi words flow plan stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en ru" NGINX_MODULES_HTTP="access autoindex browser charset empty_gif fastcgi flv geoip gzip map proxy rewrite stub_status upstream_ip_hash upload_progress" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="fbdev glint intel mach64 mga neomagic nouveau nv r128 radeon savage sis tdfx trident vesa via vmware dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS ================================================================= Package Settings ================================================================= app-emulation/open-vm-tools-0.0.20110923.491607 was built with the following: USE="(multilib) pic -X -doc -fuse -icu -unity -xinerama" Additionally, I noticed there's a warning message when installing open-vm-tools: >>> Configuring source in /var/tmp/portage/app-emulation/open-vm-tools-0.0.20110923.491607/work/open-vm-tools-2011.09.23-491607 ... * econf: updating open-vm-tools-2011.09.23-491607/config/config.sub with /usr/share/gnuconfig/config.sub * econf: updating open-vm-tools-2011.09.23-491607/config/config.guess with /usr/share/gnuconfig/config.guess ./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --libdir=/usr/lib64 --with-procps --with-dnet --without-kernel-modules --disable-docs --docdir=/usr/share/doc/open-vm-tools-0.0.20110923.491607 --without-x --without-gtk2 --without-gtkmm --without-icu --with-pic --disable-unity --disable-multimon configure: WARNING: unrecognized options: --disable-unity ^^^^
I've narrowed it down to the latest version of open-vm-tools (0.0.20110923.491607). Once I unmerge open-vm-tools, the issue is gone. I'll try to revert to the previous version of open-vm-tools (0.0.20110821.471295) and see what happens. I'd think from now on Vadim Kuznetsov could possibly take over this bug. Andrejs
So I reverted to open-vm-tools-0.0.20110821.471295.ebuild, and there are no issues for 48 hours. Please consider fixing the latest version, as it causes processes to go to uninterruptible sleep after a while. Andrejs
Is this still a problem?
(In reply to Andreas K. Hüttel from comment #5) > Is this still a problem? No reply.
