Summary: | Portage should not fail if os.rename is across filesystem boundaries | ||
---|---|---|---|
Product: | Portage Development | Reporter: | Martin Väth <martin> |
Component: | Core | Assignee: | Portage team <dev-portage> |
Status: | RESOLVED FIXED | ||
Severity: | normal | Keywords: | InVCS |
Priority: | High | ||
Version: | unspecified | ||
Hardware: | All | ||
OS: | Linux | ||
URL: | http://forums.gentoo.org/viewtopic-t-465367-start-67.html | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Bug Depends on: | |||
Bug Blocks: | 181949, 187293 | ||
Attachments: |
substitute for os.rename
portage-2.1.3_rc5 patch to replace os.rename() with shutil.move() portage-2.1.3_rc5 patch to replace os.rename() with portage.movefile() in cases where the parent directory might change |
Description
Martin Väth
2007-06-23 09:09:25 UTC
Created attachment 122877 [details]
substitute for os.rename
The attached function might be used instead of os.rename from within portage:
It first attempts os.rename; if this fails because of filesystem boundaries, it copies the corresponding tree and removes the old one.
There's an shutil.move(src, dest) function that seem to be made for this type of thing, so hopefully we can just use that. Created attachment 122886 [details, diff]
portage-2.1.3_rc5 patch to replace os.rename() with shutil.move()
This is fixed in svn r6968.
(In reply to comment #2) > There's an shutil.move(src, dest) function that seem to be made > for this type of thing, so hopefully we can just use that. But as I said, shutil.move throws strange errors in my tests (python-2.4.4-r4), so I doubt that this solves the problem: With /var/db mounted by aufs+squashfs, a simple attempt to call shutil.move to rename a directory throws the error: shutil.move("/var/db/pkg/app-emacs", "/var/db/pkg/app-emacs-bak") File "/usr/lib/python2.4/shutil.py", line 189, in move raise Error, "Cannot move a directory '%s' into itself '%s'." % (src, dst) shutil.Error: Cannot move a directory '/var/db/pkg/app-emacs' into itself '/var/db/pkg/app-emacs-bak'. (In reply to comment #4) > shutil.move("/var/db/pkg/app-emacs", "/var/db/pkg/app-emacs-bak") > File "/usr/lib/python2.4/shutil.py", line 189, in move > raise Error, "Cannot move a directory '%s' into itself '%s'." % (src, dst) > shutil.Error: Cannot move a directory '/var/db/pkg/app-emacs' into itself > '/var/db/pkg/app-emacs-bak'. > That error is triggered by the flawed logic in this function: def destinsrc(src, dst): return abspath(dst).startswith(abspath(src)) That's unfortunate... (In reply to comment #5) > return abspath(dst).startswith(abspath(src)) Oh, I see... so it is a bug, but hopefully not relevant for portage. Created attachment 122922 [details, diff]
portage-2.1.3_rc5 patch to replace os.rename() with portage.movefile() in cases where the parent directory might change
That shutil.move() can cause problems so I've converted relevant cases to use portage.movefile() instead.
Thanks for the patch. (In reply to comment #7) > in cases where the parent directory might change The problem with unionfs/aufs is that there is usually a filesystem boundary within the *same* parent directory unless the file was freshly created/modified (which includes a "move" of the parent directory). One might consider the failure of rename() in such a case as a bug of unionfs/aufs, but it is really not possible to do an atomic rename() in such a case which is a POSIX requirement... If we're going (In reply to comment #8) > One might consider the > failure of rename() in such a case as a bug of unionfs/aufs, but it is really > not possible to do an atomic rename() in such a case which is a POSIX > requirement... I'd prefer to stick with rename(2) operations wherever possible. If the unionfs/aufs documentation explicitly states that a specific case will result in EXDEV, then we have an argument against using rename(2). The docs that I've seen seem to indicate that rename operations on files should succeed. http://aufs.sourceforge.net/aufs.html: When you issue rename(2) to the file on aufs, aufs may copyup it to the higher writable branch. If this behaviour is not what you want, then you should rename(2) it on the lower branch directly. Documentation/filesystems/unionfs/rename.txt: o: success E: error (either unionfs or vfs) X: EXDEV none = file does not exist file = file is a file dir = file is a empty directory child= file is a non-empty directory wh = file is a directory containing only whiteouts; this makes it logically empty none file dir child wh file o o E E E dir o E o E o child X E X E X wh o E o E o Since I can use only aufs in the moment, I didn't check the unionfs documentation and was therefore not aware that files might be treated differently than directories here. I tested now with aufs, and it is here also as the unionfs documentation describes: Renaming files or empty directories succeeds without any problems in all reasonable situations, only renaming a *nonempty* directory fails (if the directory was not in the writable branch before), i.e. the bug was originally triggered only because it was in the "child-child" situation (unionfs terminology). So it seems that your patch is the perfect solution... This has been released in 2.1.3_rc6. |