There is a very odd situation I keep running into on virtual machines with NFS root. If I do a world update, I get NFS device or resource busy errors and portage fails. If I merge the exact packages it succeeds. Portage itself is doing something to cause this. I NEVER get these error otherwise on the systems regardless of loads or compiling, etc. Just when portage is doing something very early on in the merge of the first package. Here is just one example. I cannot update world, but each package listed, updates/installs fine on its own. mail ~ # emerge -uDNkqv --keep-going world; etc-update; emerge -qav --depclean;revdep-rebuild -iq -- -qkv;eclean distfiles;eclean packages;emerge -uDNpv world; * IMPORTANT: 14 news items need reading for repository 'gentoo'. * Use eselect news to read news items. >>> Verifying ebuild manifests >>> Emerging binary (1 of 10) sys-process/procps-3.3.8-r2 >>> Jobs: 0 of 10 complete, 1 running Load avg: 0.71, 0.46, 0.27 Traceback (most recent call last): File "/usr/bin/emerge", line 50, in <module> retval = emerge_main() File "/usr/lib/portage/pym/_emerge/main.py", line 1031, in emerge_main return run_action(emerge_config) File "/usr/lib/portage/pym/_emerge/actions.py", line 4062, in run_action emerge_config.args, spinner) File "/usr/lib/portage/pym/_emerge/actions.py", line 453, in action_build retval = mergetask.merge() File "/usr/lib/portage/pym/_emerge/Scheduler.py", line 1019, in merge rval = self._merge() File "/usr/lib/portage/pym/_emerge/Scheduler.py", line 1408, in _merge self._main_loop() File "/usr/lib/portage/pym/_emerge/Scheduler.py", line 1379, in _main_loop self._event_loop.iteration() File "/usr/lib/portage/pym/portage/util/_eventloop/EventLoop.py", line 268, in iteration if not x.callback(f, event, *x.args): File "/usr/lib/portage/pym/portage/util/_async/PipeLogger.py", line 133, in _output_handler self._unregister_if_appropriate(event) File "/usr/lib/portage/pym/_emerge/AbstractPollTask.py", line 129, in _unregister_if_appropriate self.wait() File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 57, in wait self._wait_hook() File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 175, in _wait_hook self._exit_listener_stack.pop()(self) File "/usr/lib/portage/pym/_emerge/SpawnProcess.py", line 168, in _pipe_logger_exit self.wait() File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 57, in wait self._wait_hook() File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 175, in _wait_hook self._exit_listener_stack.pop()(self) File "/usr/lib/portage/pym/_emerge/EbuildPhase.py", line 266, in _ebuild_exit self.wait() File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 57, in wait self._wait_hook() File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 175, in _wait_hook self._exit_listener_stack.pop()(self) File "/usr/lib/portage/pym/_emerge/Binpkg.py", line 234, in _clean_exit portage.prepare_build_dirs(self.settings["ROOT"], self.settings, 1) File "/usr/lib/portage/pym/portage/package/ebuild/prepare_build_dirs.py", line 42, in prepare_build_dirs shutil.rmtree(clean_dir) File "/usr/lib/portage/pym/portage/__init__.py", line 259, in __call__ rval = self._func(*wrapped_args, **wrapped_kwargs) File "/usr/lib/python2.7/shutil.py", line 252, in rmtree onerror(os.remove, fullname, sys.exc_info()) File "/usr/lib/python2.7/shutil.py", line 250, in rmtree os.remove(fullname) OSError: [Errno 16] Device or resource busy: '/tmp/portage/sys-process/procps-3.3.8-r2/temp/.nfs000000000000e02a000159f8' Scanning Configuration files... Exiting: Nothing left to do; exiting. :) ^C Exiting on signal 2 ^C^C mail ~ # emerge -qkv procps * IMPORTANT: 14 news items need reading for repository 'gentoo'. * Use eselect news to read news items. >>> Emerging binary (1 of 1) sys-process/procps-3.3.8-r2 >>> Installing (1 of 1) sys-process/procps-3.3.8-r2 >>> Jobs: 1 of 1 complete Load avg: 0.75, 0.52, 0.31 !!! existing preserved libs found * IMPORTANT: 14 news items need reading for repository 'gentoo'. * Use eselect news to read news items.
Another example, seems to only be with binary packages. The ones that have to be compiled seem to be fine. mail ~ # emerge -uDNkqav world * IMPORTANT: 14 news items need reading for repository 'gentoo'. * Use eselect news to read news items. [binary U ] sys-apps/which-2.20-r1 [2.20] [binary U ] sys-apps/debianutils-4.4 [4.3.4] USE="-static" [binary U ] sys-apps/man-pages-3.55 [3.53] USE="nls" LINGUAS="-da -de -fr -it -ja -nl -pl -ro -ru -zh_CN" [ebuild U ] dev-perl/Return-Value-1.666.1-r1 [1.666.1] [ebuild U ] dev-perl/Email-MIME-ContentType-1.16.0 [1.15.0] USE="{-test}" [ebuild U ] dev-perl/HTTP-Negotiate-6.0.1-r1 [6.0.0] [binary U ] dev-libs/openssl-1.0.1f [1.0.1e-r1] USE="sse2 tls-heartbeat zlib -bindist -gmp -kerberos -rfc3779 -static-libs {-test} -vanilla" [ebuild U ] dev-perl/perl-ldap-0.570.0 [0.530.0] USE="ssl -sasl -xml" [binary R ] dev-lang/python-2.7.5-r3 USE="hardened ncurses readline ssl threads (wide-unicode) xml -berkdb* -build -doc -examples -gdbm -ipv6 -sqlite -tk -wininst" Would you like to merge these packages? [Yes/No] Yes >>> Verifying ebuild manifests >>> Emerging binary (1 of 9) sys-apps/which-2.20-r1 >>> Jobs: 0 of 9 complete, 1 running Load avg: 0.36, 0.56, 0.39 Traceback (most recent call last): File "/usr/bin/emerge", line 50, in <module> retval = emerge_main() File "/usr/lib/portage/pym/_emerge/main.py", line 1031, in emerge_main return run_action(emerge_config) File "/usr/lib/portage/pym/_emerge/actions.py", line 4062, in run_action emerge_config.args, spinner) File "/usr/lib/portage/pym/_emerge/actions.py", line 453, in action_build retval = mergetask.merge() File "/usr/lib/portage/pym/_emerge/Scheduler.py", line 1019, in merge rval = self._merge() File "/usr/lib/portage/pym/_emerge/Scheduler.py", line 1408, in _merge self._main_loop() File "/usr/lib/portage/pym/_emerge/Scheduler.py", line 1379, in _main_loop self._event_loop.iteration() File "/usr/lib/portage/pym/portage/util/_eventloop/EventLoop.py", line 268, in iteration if not x.callback(f, event, *x.args): File "/usr/lib/portage/pym/portage/util/_async/PipeLogger.py", line 133, in _output_handler self._unregister_if_appropriate(event) File "/usr/lib/portage/pym/_emerge/AbstractPollTask.py", line 129, in _unregister_if_appropriate self.wait() File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 57, in wait self._wait_hook() File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 175, in _wait_hook self._exit_listener_stack.pop()(self) File "/usr/lib/portage/pym/_emerge/SpawnProcess.py", line 168, in _pipe_logger_exit self.wait() File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 57, in wait self._wait_hook() File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 175, in _wait_hook self._exit_listener_stack.pop()(self) File "/usr/lib/portage/pym/_emerge/EbuildPhase.py", line 266, in _ebuild_exit self.wait() File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 57, in wait self._wait_hook() File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 175, in _wait_hook self._exit_listener_stack.pop()(self) File "/usr/lib/portage/pym/_emerge/Binpkg.py", line 234, in _clean_exit portage.prepare_build_dirs(self.settings["ROOT"], self.settings, 1) File "/usr/lib/portage/pym/portage/package/ebuild/prepare_build_dirs.py", line 42, in prepare_build_dirs shutil.rmtree(clean_dir) File "/usr/lib/portage/pym/portage/__init__.py", line 259, in __call__ rval = self._func(*wrapped_args, **wrapped_kwargs) File "/usr/lib/python2.7/shutil.py", line 252, in rmtree onerror(os.remove, fullname, sys.exc_info()) File "/usr/lib/python2.7/shutil.py", line 250, in rmtree os.remove(fullname) OSError: [Errno 16] Device or resource busy: '/tmp/portage/sys-apps/which-2.20-r1/temp/.nfs000000000000e05600015af1' mail ~ # emerge -qkv which debianutils man-pages * IMPORTANT: 14 news items need reading for repository 'gentoo'. * Use eselect news to read news items. >>> Emerging binary (1 of 3) sys-apps/which-2.20-r1 >>> Installing (1 of 3) sys-apps/which-2.20-r1 >>> Emerging binary (2 of 3) sys-apps/debianutils-4.4 >>> Installing (2 of 3) sys-apps/debianutils-4.4 >>> Recording sys-apps/debianutils in "world" favorites file... >>> Emerging binary (3 of 3) sys-apps/man-pages-3.55 >>> Jobs: 2 of 3 complete, 1 running Load avg: 0.94, 0.71, 0.48 >>> Jobs: 2 of 3 complete, 1 running Load avg: 0.95, 0.73, 0.49 >>> Installing (3 of 3) sys-apps/man-pages-3.55 >>> Jobs: 3 of 3 complete Load avg: 0.99, 0.82, 0.55 !!! existing preserved libs found * IMPORTANT: 14 news items need reading for repository 'gentoo'. * Use eselect news to read news items. mail ~ # emerge --info Portage 2.2.7 (hardened/linux/x86, gcc-4.7.3, glibc-2.17, 3.8.7-hardened i686) ================================================================= System uname: Linux-3.8.7-hardened-i686-Intel-R-_Xeon-TM-_MP_CPU_2.50GHz-with-gentoo-2.2 KiB Mem: 2072472 total, 218188 free KiB Swap: 0 total, 0 free Timestamp of tree: Sat, 25 Jan 2014 04:30:01 +0000 ld GNU ld (GNU Binutils) 2.23.2 app-shells/bash: 4.2_p45 dev-lang/python: 2.7.5-r3, 3.3.2-r2 dev-util/pkgconfig: 0.28 sys-apps/baselayout: 2.2 sys-apps/openrc: 0.12.4 sys-apps/sandbox: 2.6-r1 sys-devel/autoconf: 2.69 sys-devel/automake: 1.13.4 sys-devel/binutils: 2.23.2 sys-devel/gcc: 4.7.3-r1 sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4.2 sys-devel/make: 3.82-r4 sys-kernel/linux-headers: 3.9 (virtual/os-headers) sys-libs/glibc: 2.17
make.conf(5): distlocks Portage uses lockfiles to ensure competing instances don't clobber each other's files. This feature is enabled by default but may cause heartache on less intelligent remote filesystems like NFSv2 and some strangely config‐ ured Samba server (oplocks off, NFS re-export). A tool /usr/lib/portage/bin/clean_locks exists to help handle lock issues when a problem arises (normally due to a crash or disconnect). Did you try that? Also, please post your _entire_ `emerge --info' output in a comment.
No I have not tried distlocks, guess I can experiment with that, but only portage when doing world updates causes this issue. Not sure if it matters, but using NFSv3, not NFSv2. I have nfs lock off, maybe I need to disable that for portage. It happens on all of my VMs, but does not happen all the time. Here is emerge --info from one, the one I was showing the issue one when creating the bug. mail ~ # emerge --info Portage 2.2.7 (hardened/linux/x86, gcc-4.7.3, glibc-2.17, 3.8.7-hardened i686) ================================================================= System uname: Linux-3.8.7-hardened-i686-Intel-R-_Xeon-TM-_MP_CPU_2.50GHz-with-gentoo-2.2 KiB Mem: 2072472 total, 365080 free KiB Swap: 0 total, 0 free Timestamp of tree: Sat, 25 Jan 2014 04:30:01 +0000 ld GNU ld (GNU Binutils) 2.23.2 app-shells/bash: 4.2_p45 dev-lang/python: 2.7.5-r3, 3.3.2-r2 dev-util/pkgconfig: 0.28 sys-apps/baselayout: 2.2 sys-apps/openrc: 0.12.4 sys-apps/sandbox: 2.6-r1 sys-devel/autoconf: 2.69 sys-devel/automake: 1.13.4 sys-devel/binutils: 2.23.2 sys-devel/gcc: 4.7.3-r1 sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4.2 sys-devel/make: 3.82-r4 sys-kernel/linux-headers: 3.9 (virtual/os-headers) sys-libs/glibc: 2.17 Repositories: gentoo ACCEPT_KEYWORDS="x86" ACCEPT_LICENSE="* -@EULA" CBUILD="i686-pc-linux-gnu" CFLAGS="-march=pentium3 -msse2 -O2 -pipe -mno-tls-direct-seg-refs" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /var/qmail/alias /var/qmail/control" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-march=pentium3 -msse2 -O2 -pipe -mno-tls-direct-seg-refs" DISTDIR="/usr/portage/distfiles" FCFLAGS="-march=i686 -O2 -pipe" FEATURES="assume-digests binpkg-logs buildpkg config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-march=i686 -O2 -pipe" GENTOO_MIRRORS="http://distro.ibiblio.org/pub/linux/distributions/gentoo/ http://gentoo.mirrors.pair.com/" LANG="en_US.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j2" PKGDIR="/usr/portage/packages/x86-opteron" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="" USE="acpi bash-completion berkdb bzip2 cracklib crypt custom-cflags hardened iconv logrotate mmx modules nls noauthcram nptl nptlonly openntpd pae pam pax_kernel pic posix qmail-spp readline session spf srs sse sse2 ssl symlink syslog sysvipc tcpd tls trashquota udev unicode urandom x86 zlib" ABI_X86="32" ELIBC="glibc" INPUT_DEVICES="keyboard" KERNEL="linux" LINGUAS="en en_US" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_3" USERLAND="GNU" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, SYNC, USE_PYTHON rootfs on / type rootfs (rw) 192.168.0.10:/vms/mail on / type nfs (rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,nolock,nordirplus,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.0.10,mountvers=3,mountproto=tcp,local_lock=all,addr=192.168.0.10)
I guess the problem goes away immediately when PORTAGE_TMPDIR/TMP is not set to a directory on an NFS mount.
I would assume so but these systems being nfs root always have tmp as a nfs mount, its part of root. Single partition export for the vm. But the lock issue does not happen all the time. This one was failing while I was able to update the same packages on another. There is little difference between the vms. I use a chroot base vm to update stuff, and push out binaries to the rest of the vms. I have experienced this for some time inconsistently. Just finally annoying me enough lately to say something. Its very odd as it ONLY happens when doing world updates. If I merge the exact same packages it does not happen. Seems to also only happen with binary merges, I do not experience that with compiling. I do not believe I can replicate either.
Running into this again today on two machines. I am setting up a new server, this one running KVM instead of Xen. Same hosts, copied over, and running into same problem on 2 out of 5 systems. Completely different host, same virtual guest, totally different NFS server. The other is a 32bit system, this is 64bit. I really believe portage is doing something to trigger this, as it only happens with world, and only with portage. I can wait a day or so at best to finish updating these machines if anyone would like further details on this. I can run tests, and such to see what is causing this. Its only with binary packages, and only when emerging world on certain guest virtual machines, that are NFS root mounted. Which if it was a general NFS error, the entire virtual machine being NFS root mounted would have issues. Instead its just portage, and what looks like to be very early on in unpacking or something. It takes seconds from start of emerge to failure.
Created attachment 371978 [details] failed emerge output
This is a python bug/issue http://bugs.python.org/issue13978, but its more a bug/issue in use than with python itself. In /usr/lib/portage/pym/portage/package/ebuild/prepare_build_dirs.py line 42, in prepare_build_dirs calls shutil.rmtree(clean_dir). Per the above bug it seems that a file is still open when a directory or files are trying to be removed. This is all portage behavior that needs to be address in code. I assume its due to an asychronous vs sychronous task. Thus a file is still open while its trying to be removed. Rather than it waiting for removal till no longer in use. I am poking around the code, but not familiar with python or portage coding. Not sure I can address/fix myself. But this should help point out what is causing this issue, so it can be addressed. I have run these systems with NFS root for many years. I have never seen any negative of tmpdir being on a nfs mount/root. Which FYI, tmp is a tmpfs filesystem nfs exported by the host. So all vms use the same tmpfs file system the host uses. It would not make sense to be eating up memory from the host and vms to have each vm have its own tmpfs. Thus easier to setup one, bind mount that, before export of vm root.
For reference https://www.google.com/search?q=python+rmtree+nfs+device+or+resource+busy
Commenting out this line in /usr/lib/portage/pym/_emerge/Binpkg.py eliminates the error, but then merge fails because the log build.log file does not exist. # This initializes PORTAGE_LOG_FILE. #portage.prepare_build_dirs(self.settings["ROOT"], self.settings, 1) Resulting in File "/usr/lib/portage/pym/portage/util/_async/PipeLogger.py", line 32, in _start encoding=_encodings['fs'], errors='strict'), mode='ab') FileNotFoundError: [Errno 2] No such file or directory: '/tmp/portage/dev-libs/libxml2-2.9.1-r1/temp/build.log' It takes a bit longer to produce that error vs the nfs one. So something to do with build.log file initialization when dealing with a binary package is causing this issue on nfs mounted tmpdir.
(In reply to William L. Thomson Jr. from comment #10) > Commenting out this line in /usr/lib/portage/pym/_emerge/Binpkg.py > eliminates the error, but then merge fails because the log build.log file > does not exist. The merge sorta failed, the package was installed, but failed after it was installed. I have a temporary fix by setting cleanup in portage.build_prepare_dirs to false, such as # This initializes PORTAGE_LOG_FILE. portage.prepare_build_dirs(self.settings["ROOT"], self.settings, 0) Presently portage is set to cleanup there, and not sure that makes sense for binary packages. # This initializes PORTAGE_LOG_FILE. portage.prepare_build_dirs(self.settings["ROOT"], self.settings, 1) That one change allows all my merges to proceed vs fail with nfs device or resource busy. I am not sure if this is a permanent proper solution or not. But does track down where the problem occurs for others to address in a permanent proper manner.
Created attachment 372006 [details, diff] patch to disable cleanup in Binpkg.py
While not a permanent solution, and might not work for you, but I used: FEATURES="keeptemp". since the temp was the one complaining. At the end of emerge you would need to delete */temp directories yourself.
This can likely be closed. I have not seen this in a long time. Though no longer running many systems on NFS root. If the problem does occur again. I believe it to by a python issue not portage code.
(In reply to William L. Thomson Jr. from comment #14) > This can likely be closed. I have not seen this in a long time. Though no > longer running many systems on NFS root. I think the potential for an EBUSY here is likely enough (regardless of NFS) that we should handle it, with a loud error message. We already handle EBUSY when unmerging package files. We can handle it here by falling back to rm -rf command. We definitely don't want to disable the cleanup like the attached patch does, so I'll mark that obsolete. > If the problem does occur again. I believe it to by a python issue not portage code. Yeah the issue that you initially reported may have been due to a python bug, but EBUSY is a more general problem that can easily be triggered by things other than python itself.
Seems James Le Cuirot/Chewi hit this when working with Icedtea on arm, via a contributors system they are letting James/Chewi use. This problem still exists.
