I've recently moved my main catalyst build directory to a NAS mountpoint over NFSv4.1. It seems there is an old "hack" called "sillyrename", where when a client deletes/unlinks a file, the NFS server will very temporarily rename the file to ".nfs<lots of hex digits>', then signal to the client the file is deleted. The NFS server will fully remove the file once it is sure that no other clients need it. Catalyst chokes at this point because the Python function it uses to delete the directory (shutil.rmtree) doesn't seem to be capable of dealing with this NFS obscurity: NOTICE:catalyst:Removing the chroot path ... 31 Jan 2020 23:50:16 EST: NOTICE : Emptying directory: /nas/catalyst/tmp/musl/stage1-mips3-20200119 NOTICE:catalyst:Emptying directory: /nas/catalyst/tmp/musl/stage1-mips3-20200119 31 Jan 2020 23:51:25 EST: ERROR : clear_dir failed Traceback (most recent call last): File "/usr/lib/python3.7/site-packages/catalyst/fileops.py", line 85, in clear_dir shutil.rmtree(target) File "/usr/lib/python3.7/shutil.py", line 498, in rmtree onerror(os.rmdir, path, sys.exc_info()) File "/usr/lib/python3.7/shutil.py", line 496, in rmtree os.rmdir(path) OSError: [Errno 93] Directory not empty: '/nas/catalyst/tmp/musl/stage1-mips3-20200119' [snip] Traceback (most recent call last): File "/usr/lib/python3.7/site-packages/catalyst/fileops.py", line 85, in clear_dir shutil.rmtree(target) File "/usr/lib/python3.7/shutil.py", line 494, in rmtree _rmtree_safe_fd(fd, path, onerror) File "/usr/lib/python3.7/shutil.py", line 452, in _rmtree_safe_fd onerror(os.unlink, fullname, sys.exc_info()) File "/usr/lib/python3.7/shutil.py", line 450, in _rmtree_safe_fd os.unlink(entry.name, dir_fd=topfd) OSError: [Errno 16] Device or resource busy: '.nfs000000000003b4f7000062a6' 31 Jan 2020 23:51:25 EST: ERROR : clean_stage1 failed, see previous log messages for details Portage has similar problems when its TMPDIR is hosted on NFSv4.1, and it tries to rmdir the ${T} directory, but Portage seems to just ignore the fact that the delete command fails. The only side-effect in this case is leftover files in $PORTAGE_TMPDIR. I don't know of a way to disable this "sillyrename" feature on the NFS server. It's running FreeBSD 12.1, and it seems that their upstream development has discussed disabling, or making a switch to disable it, available in FreeBSD 13.x, which is a ways off from release. This RedHat bug may shed some light on options, though the developers there note that shutil.rmtree was not designed to handle recursively deleting directories that still has open files: https://bugzilla.redhat.com/show_bug.cgi?id=1362667
*** This bug has been marked as a duplicate of bug 282928 ***
