Sometimes it happens that an aborted package merge is never tried again. In this situation the temporary files inside /var/tmp/portage will stay there foreever. For this reason i would like to propose to clean this files per default using the bootmisc service under OpenRC or the tmpfiles unit under systemd. This would have the advantage, that a build.log for a failed merge would stay there for some time, but the folder will still not fill up with garbage over time. I.e., this is a proposal to set: clean_tmp_dirs="$clean_tmp_dirs /var/tmp/portage" inside /var/conf.d/bootmisc and install a file /usr/lib/tmpfiles.d/portage.conf with the content: D /var/tmp 0775 portage portage 7d or some other time span. Reproducible: Always
> D /var/tmp 0775 portage portage 7d I think you meant /var/tmp/portage; /var/tmp should definitely not be owned by the portage user! Anyway, vanilla systemd installs a tmpfiles snippet to clean all of /var/tmp on a 30 day cycle. We disable this on Gentoo by default. See bug 490676.
(In reply to Mike Gilbert from comment #1) > > D /var/tmp 0775 portage portage 7d > > I think you meant /var/tmp/portage; /var/tmp should definitely not be owned > by the portage user! yes, you are right. This was a mistake.
As floppym points out, we had to actually disable this because of Portage. Long story short, when tar extracts files from archive, it restores old mtimes. So if the tmpfiles run happens to be done while Portage is working, it can accidentally remove just-extracted sources.
Note that if you figured how to, say, make it figure out what to remove based on two top directory mtimes, that could work. I mean like globbing $PORTAGE_TMPDIR/*/* dirs, and removing the whole tree if directory is old.
What about executing the cleaning only during boot? i.e.: D! /var/tmp/portage 0775 portage portage 7d Still the 7d parameter might be violated by this option. ------------- The man page of tmpfiles.d says: If the exclamation mark is used, this line is only safe of execute during boot, and can break a running system. Lines without the exclamation mark are presumed to be safe to execute at any time, e.g. on package upgrades. systemd-tmpfiles will execute line with an exclamation mark only if option --boot is given. For example: # Make sure these are created by default so that nobody else can d /tmp/.X11-unix 1777 root root 10d # Unlink the X11 lock files r! /tmp/.X[0-9]*-lock The second line in contrast to the first one would break a running system, and will only be executed with --boot.
i have created an upstream bug for atomic directory deletion: https://github.com/systemd/systemd/issues/4933 I have therefore reopened this bug report, in order to track their reaction / willingness to implement such a feature.
(In reply to Michał Górny from comment #3) > As floppym points out, we had to actually disable this because of Portage. > Long story short, when tar extracts files from archive, it restores old > mtimes. So if the tmpfiles run happens to be done while Portage is working, > it can accidentally remove just-extracted sources. It would make more sense for it to use ctime instead of mtime.
(In reply to Zac Medico from comment #7) It actually checks all 3 timestamps (atime, ctime, mtime). If any of them is newer than the cutoff date, the file is not removed. https://github.com/systemd/systemd/blob/master/src/tmpfiles/tmpfiles.c#L547
(In reply to Till Schäfer from comment #5) > What about executing the cleaning only during boot? > > i.e.: > > D! /var/tmp/portage 0775 portage portage 7d I was thinking about this too. While it would certainly be safer, it could break resuming pretty recent builds. Like I unpacked a 7-day old chromium version, built half of it, then rebooted. Normally I'd resume the build after reboot but now systemd could potentially remove some old source files. (In reply to Zac Medico from comment #7) > (In reply to Michał Górny from comment #3) > > As floppym points out, we had to actually disable this because of Portage. > > Long story short, when tar extracts files from archive, it restores old > > mtimes. So if the tmpfiles run happens to be done while Portage is working, > > it can accidentally remove just-extracted sources. > > It would make more sense for it to use ctime instead of mtime. Can't tar restore ctime as well?
(In reply to Michał Górny from comment #9) > (In reply to Zac Medico from comment #7) > > (In reply to Michał Górny from comment #3) > > > As floppym points out, we had to actually disable this because of Portage. > > > Long story short, when tar extracts files from archive, it restores old > > > mtimes. So if the tmpfiles run happens to be done while Portage is working, > > > it can accidentally remove just-extracted sources. > > > > It would make more sense for it to use ctime instead of mtime. > > Can't tar restore ctime as well? The utime/utimes functions only allow setting of atime and mtime. At least on Linux, it's not possible set ctime. There's some discussion here: http://stackoverflow.com/questions/4537291/setting-creation-or-change-timestamps
(In reply to Michał Górny from comment #9) > I was thinking about this too. While it would certainly be safer, it could > break resuming pretty recent builds. Like I unpacked a 7-day old chromium > version, built half of it, then rebooted. Normally I'd resume the build > after reboot but now systemd could potentially remove some old source files. Is that actually "supported"? I don't think there is any way to do it using emerge(1), and you would have to use FEATURES="noauto" with ebuild(1).
(In reply to Mike Gilbert from comment #11) > (In reply to Michał Górny from comment #9) > > I was thinking about this too. While it would certainly be safer, it could > > break resuming pretty recent builds. Like I unpacked a 7-day old chromium > > version, built half of it, then rebooted. Normally I'd resume the build > > after reboot but now systemd could potentially remove some old source files. > > Is that actually "supported"? I don't think there is any way to do it using > emerge(1), and you would have to use FEATURES="noauto" with ebuild(1). ebuild(1) works just fine with no FEATURES changes. It notices phases were run, and runs remaining ones.
if ctime cannot be modified and is respected by tmpfiles I think the original issue (files can have timestamps in the past and get removed to early) is resolved, or am I wrong? The remaining cases from my point of view are: 1. (Reboot and) Continue merge: Can using tmpfiles lead to a situation where the files inside the package folder get partially removed and lead to failing merges afterwards? Or does portage always clean a directory when merging a package again? If the first case is true we should still rely on some "atomic" removal of the directories. If the sourcefiles get removed after 30 days and you need to re-do some work (e.g. unpacking) that is fine in my opinion and is better then having garbage filling up my hard disk. 2. Should be care about modification/access dates from the future? Typically this should be a result a erroneous tar file or are there some legal use cases of this situation? If we like to handle such a situation, we should also rely on atomic directory removal.
I think we need a customized script that checks the ctime of $PORTAGE_TMPDIR/portage/*/* directories and removes them atomically when they reach a certain age. For maintenance tasks like this, we typically create an emaint sub-command.
(In reply to Zac Medico from comment #14) > I think we need a customized script that checks the ctime of > $PORTAGE_TMPDIR/portage/*/* directories and removes them atomically when > they reach a certain age. For maintenance tasks like this, we typically > create an emaint sub-command. Yeah, I agree. The emaint plug-in system I created should make it quite easy to create a module that does this. You can use the various modules to see how to do it. Plus There is a wiki page stating the interface specs. The module can be released with it's own ebuild too. Especially for some initial releases, to get user feedback, improve the code. If it is something that enough people like we can merge it into the portage code base. It sounds like the emaint log module would be similar to what you want to do, so you can probably use it as a starting point to create your module: emaint tmp [--check] [--clean] [--age]...
