Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 16162 - "MD5 + timestamp equal or empty directory" unmerge check is insufficient
Summary: "MD5 + timestamp equal or empty directory" unmerge check is insufficient
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Enhancement/Feature Requests (show other bugs)
Hardware: All Linux
: High normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
: 30433 30522 36548 39809 50737 50898 54669 71219 99376 100742 123543 147326 153812 155953 155977 159165 169457 174369 183577 (view as bug list)
Depends on:
Blocks: 193766 45764 163416 194041
  Show dependency tree
 
Reported: 2003-02-21 16:19 UTC by Toby Dickenson
Modified: 2018-02-02 00:53 UTC (History)
31 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
Additional check before removing empty directory (portage-2.1.2-empty-directory.patch,2.93 KB, patch)
2007-02-01 21:41 UTC, Kaiting Chen
Details | Diff
Ebuild for the patch (portage-2.1.2-r5.ebuild,6.81 KB, text/plain)
2007-02-01 21:42 UTC, Kaiting Chen
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Toby Dickenson 2003-02-21 16:19:39 UTC
The bug in #12576 has reappeared, as of nethack-3.4.0-r6. Nethack can not save 
games because the save directory is not created. 
 
That bug was previously fixed with a dodir, but dodir creates an empty directory that is 
then removed by auto-cleaning.  
  
The "dodir .../save" needs to be a "keepdir .../save"
Comment 1 SpanKY gentoo-dev 2003-02-21 16:54:03 UTC

*** This bug has been marked as a duplicate of 14145 ***
Comment 2 Toby Dickenson 2003-02-21 17:21:33 UTC
I think it is a mistake to classify this the same as #14145. That bug is about creating a 
subdirectory of the users home directory. This is about creating 
/usr/share/games/nethack/save/. 
Comment 3 SpanKY gentoo-dev 2003-02-21 17:34:58 UTC
*shrug* nethack sucks heh 
Comment 4 SpanKY gentoo-dev 2003-02-26 22:11:33 UTC
ok, i added keepdir for the time being but this is something to be fixed in portage 
Comment 5 Martin Holzer (RETIRED) gentoo-dev 2003-02-27 17:18:13 UTC
this is cause of autoclean function
dodir has to be done with keepdir
Comment 6 SpanKY gentoo-dev 2003-02-28 01:03:48 UTC
actually it's not invalid, do not close the bug
Comment 7 SpanKY gentoo-dev 2003-10-05 20:40:02 UTC
*** Bug 30433 has been marked as a duplicate of this bug. ***
Comment 8 Marius Mauch (RETIRED) gentoo-dev 2003-11-24 23:14:22 UTC
what would you expect from portage here ?
Comment 9 SpanKY gentoo-dev 2003-11-25 07:08:51 UTC
i would expect it to leave the directories of course

keepdir is for keeping a directory after a package is gone, not for keeping a directory while upgrading a package
Comment 10 Marius Mauch (RETIRED) gentoo-dev 2003-11-25 11:46:20 UTC
That would require that for every empty dir we have to check if it is owned by an installed package, which would result in a noticable performance decrease. The other (hackish) option is to install a lockfile (not .keep) in the dir in src_install().
Comment 11 SpanKY gentoo-dev 2003-11-25 12:16:36 UTC
no, you would only have to check for the package you just cleaned

yes, the best solution would be all packages, but like you said, that would be a noticable hit

i dont think checking the current version of the package would be a big hit ...

for example, you just emerge foo-1.0-r1.  portage goes to clean out foo-1.0.  however, before punting any empty directories it consults the CONTENTS of foo-1.0-r1.  foo-1.0-r1 lists the aforementioned directory and thus does not clean it.
Comment 12 SpanKY gentoo-dev 2003-12-26 13:53:00 UTC
*** Bug 36548 has been marked as a duplicate of this bug. ***
Comment 13 Heinrich Wendel (RETIRED) gentoo-dev 2004-04-05 07:19:49 UTC
any status on this?
Comment 14 SpanKY gentoo-dev 2004-05-11 12:34:49 UTC
*** Bug 50737 has been marked as a duplicate of this bug. ***
Comment 15 Alan Schmitt 2004-05-11 12:51:54 UTC
FYI (if you don't follow what bug 50537 is), this bug breaks enlightenment after upgrading to the 16.7_pre1 version (a required symlink is not there, preventing enlightenment from starting).
Comment 16 SpanKY gentoo-dev 2004-05-12 21:24:38 UTC
*** Bug 50898 has been marked as a duplicate of this bug. ***
Comment 17 James Goruk 2004-05-12 22:43:59 UTC
Why not just check the date on the directories or symlinks ( in my case)
Just like Make does.

So if a link has been created since the merge began it is new and shouldn't be removed.

Wouldn't this fix it?

http://bugs.gentoo.org/show_bug.cgi?id=50898
Comment 18 SpanKY gentoo-dev 2004-06-11 17:19:18 UTC
*** Bug 30522 has been marked as a duplicate of this bug. ***
Comment 19 SpanKY gentoo-dev 2004-06-21 22:42:22 UTC
*** Bug 54669 has been marked as a duplicate of this bug. ***
Comment 20 Georgi Georgiev 2004-06-21 23:58:44 UTC
Hm... not exactly sure that bug #54669 and this bug are duplicates.

The problem of bug #53669 is that there are different packages that install identical[1] files. And this *is* a problem by itself. It is not much related to the fact that portage cannot keep track of directories by themselves, which is the problem of this bug. The solutions are completely different.

Bug #53669 is solved by making sure that no two packages install identical files.

This bug is solved by enabling portage to keep track of directories installed by packages. For example, keepdir was invented for a different purpose, but it can do the trick as a workaround.

A solution to *this* bug, is to make "dodir /foo" do a "touch ${D}/foo/.installed-${P}", and these ".installed-*" files are in turn recorded in CONTENTS along with their mtime. That should do the trick even when upgrading, because the mtime of that file will change with every installation.

A solution to both bugs would be to reverse the merge/unmerge process: i.e. make portage unmerge the "currently installed package" prior to merging the new one.

________
[1] identical:= same path, same mtime, same contents
Comment 21 Georgi Georgiev 2004-06-22 00:04:16 UTC
Ooops! All references to bug #53669 in the previous comment should read "bug #54669".
Comment 22 SpanKY gentoo-dev 2004-11-14 16:11:50 UTC
*** Bug 71219 has been marked as a duplicate of this bug. ***
Comment 23 Brian Harring (RETIRED) gentoo-dev 2004-11-14 19:12:16 UTC
*** Bug 71219 has been marked as a duplicate of this bug. ***
Comment 24 Brian Harring (RETIRED) gentoo-dev 2004-11-14 19:13:46 UTC
to all who keep arguing that their bugs aren't dupes of this, this bug is essentially "portage's md5/mtime solution sucks, requiring the .keepdir hack".

So... refcounts is the solution imo.  Either way, if it's about the current hackery used, then it's a dupe of this.
Comment 25 Cory Oldford 2005-06-21 19:42:02 UTC
Come on this has been ongoing forever. I hate the added steps of creating empty
directories that are required by LTSP. 
Comment 26 Francesco R. (RETIRED) gentoo-dev 2005-07-22 05:01:35 UTC
*** Bug 99891 has been marked as a duplicate of this bug. ***
Comment 27 Jason Stubbs (RETIRED) gentoo-dev 2005-10-07 08:55:49 UTC
*** Bug 39809 has been marked as a duplicate of this bug. ***
Comment 28 Jakub Moc (RETIRED) gentoo-dev 2006-05-25 12:12:50 UTC
*** Bug 123543 has been marked as a duplicate of this bug. ***
Comment 29 Nico Baggus 2006-05-28 10:44:17 UTC
(In reply to comment #28)
Ok, ALMOST identical, in my case not only directories were removed, but also other files because they came from some tar that was created externaly. TIME & MD5 are identical for some of those files ==> removed when upgraded.

Thesolution there would be to modify the date after untarring...

Comment 30 Jakub Moc (RETIRED) gentoo-dev 2006-09-12 08:38:04 UTC
*** Bug 147326 has been marked as a duplicate of this bug. ***
Comment 31 Zac Medico gentoo-dev 2006-09-12 09:53:30 UTC
*** Bug 134620 has been marked as a duplicate of this bug. ***
Comment 32 Matthew Kennedy (RETIRED) gentoo-dev 2006-10-19 20:36:44 UTC
dev-lisp/cmucl and dev-lisp/sbcl need to have the mtimes of their installed files preserved from the src_install step.  This is because each system will attempt to recompile parts of itself *after emerge* when it finds the installed source is not older than the corresponding compiled output.

Since the result of emerging is all files in the package have the same mtime, sbcl and cmucl will attempt to recompile parts of themselves which leads to permission denied problems because /usr/..., which is not user writable.

I think dev-lang/gst has a variation of this problem.

I have been using a hack to work around this.  I tar up the contents of ${D} during src_install and install the resulting tar with the rest of the package.  Then in pkg_postinst I untar into /.

Now I can almost hear the barfing from here, I know, it's not a pleasant hack and I'd really love to rid myself of it.  

What I'd like is to have the mtimes of installed files into ${D} be preserved for the merge step.
Comment 33 Zac Medico gentoo-dev 2006-10-19 21:44:07 UTC
(In reply to comment #32)
> What I'd like is to have the mtimes of installed files into ${D} be preserved
> for the merge step.

I suppose it would be safe to preserve the timestamps from ${D} if we use the contents from the just merged version for reference counting during the unmerge of the version that it replaced (same slot).  It seems like it makes sense to preserve the timestamps from ${D} like that.  Can anyone think of any drawbacks?
Comment 34 Georgi Georgiev 2006-10-19 21:56:45 UTC
(In reply to comment #33)
> I suppose it would be safe to preserve the timestamps from ${D} if we use the
> contents from the just merged version for reference counting during the unmerge
> of the version that it replaced (same slot).  It seems like it makes sense to
> preserve the timestamps from ${D} like that.  Can anyone think of any
> drawbacks?

Some time in the past, there was a problem with binary packages having a tendency to unmerge themselves on remerge (reinstalling the same version which installs files with the same mtimes as the old emerge). If this is no longer an issue, sure thing, preserve mtimes by default. If it is an issue though, maybe it's time for a new RESTRICT. 
Comment 35 Zac Medico gentoo-dev 2006-11-03 02:08:43 UTC
*** Bug 153812 has been marked as a duplicate of this bug. ***
Comment 36 Jakub Moc (RETIRED) gentoo-dev 2006-11-22 13:58:30 UTC
*** Bug 155977 has been marked as a duplicate of this bug. ***
Comment 37 Jakub Moc (RETIRED) gentoo-dev 2006-11-23 02:07:30 UTC
*** Bug 155953 has been marked as a duplicate of this bug. ***
Comment 38 Kaiting Chen 2007-02-01 21:29:50 UTC
Hey. It looks like this problem has been around for a while, and I coded up a portage patch the other day. It works great as of yet, but as soon as I do a little more testing I'll upload it. Basically, it works like this.

When portage unmerges a package and runs into an empty directory, before it removes it, it first checks to see if any packages WITH THE SAME NAME owns the directory. If any of the do, it keeps it there. There's no noticeable performance hit for this, and it's almost as good as checking to see if EVERY package owns the directory.

What this means is that when you upgrade a package or replace it (for example if you want to compile it with different use flags), empty directories will not be deleted without the ebuild having to specify anything. If you remove a slotted package and there's an empty directory that's owned by a different slotted version of the same package, it's kept as well. The empty directory is only removed if no other package of the same name owns it.
Comment 39 Kaiting Chen 2007-02-01 21:41:29 UTC
Created attachment 108894 [details, diff]
Additional check before removing empty directory

This is the patch I was promising
Comment 40 Kaiting Chen 2007-02-01 21:42:57 UTC
Created attachment 108896 [details]
Ebuild for the patch

Here's the ebuild the applies the patch
Comment 41 Kaiting Chen 2007-02-14 20:22:36 UTC
Um. Any progress on this?
Comment 42 Jakub Moc (RETIRED) gentoo-dev 2007-03-05 14:34:50 UTC
*** Bug 169457 has been marked as a duplicate of this bug. ***
Comment 43 Jakub Moc (RETIRED) gentoo-dev 2007-03-14 01:35:44 UTC
*** Bug 100742 has been marked as a duplicate of this bug. ***
Comment 44 Jakub Moc (RETIRED) gentoo-dev 2007-03-14 01:37:00 UTC
*** Bug 99376 has been marked as a duplicate of this bug. ***
Comment 45 Jakub Moc (RETIRED) gentoo-dev 2007-03-26 23:00:52 UTC
*** Bug 172341 has been marked as a duplicate of this bug. ***
Comment 46 Jakub Moc (RETIRED) gentoo-dev 2007-04-13 07:06:37 UTC
*** Bug 174369 has been marked as a duplicate of this bug. ***
Comment 47 Jakub Moc (RETIRED) gentoo-dev 2007-06-28 20:03:09 UTC
*** Bug 183577 has been marked as a duplicate of this bug. ***
Comment 48 Petr Bahula 2007-10-23 13:07:23 UTC
Halo, I was check my system for missing files, dirs etc.
I was found lot of packages, where had file or dir in /var/db/pkg/*/*/CONTENTS file but not in filesystem.
All these packages I was rebuild by "emerge -1 <package>".
All these packages are corret now but the dev-java/javatoolkit-0.2.0-r1 not.

I have portage 2.1.3.9

# ls /usr/share/javatoolkit/lib
ls: cannot access /usr/share/javatoolkit/lib: není souborem ani adresářem
# ls -a /usr/share/javatoolkit/   
.  ..  build-xml-source-target.xslt  pym
# emerge -1 javatoolkit
....
>>> Emerging (1 of 1) dev-java/javatoolkit-0.2.0-r1 to /
....
>>> Merging dev-java/javatoolkit-0.2.0-r1 to /
--- /usr/
....
--- /usr/share/javatoolkit/
--- /usr/share/javatoolkit/lib/
>>> /usr/share/javatoolkit/build-xml-source-target.xslt
--- /usr/share/javatoolkit/pym/
--- /usr/share/javatoolkit/pym/parser/
....
--- /usr/lib/javatoolkit/bin/
>>> Safely unmerging already-installed instance...
No package files given... Grabbing a set.
--- replaced obj /usr/share/man/man1/findclass.1.bz2
....
--- replaced obj /usr/share/javatoolkit/pym/__init__.py
--- replaced dir /usr/share/javatoolkit/pym
--- replaced dir /usr/share/javatoolkit/lib
--- replaced obj /usr/share/javatoolkit/build-xml-source-target.xslt
--- replaced dir /usr/share/javatoolkit
....
--- replaced dir /usr
 * Cleaning orphaned Python bytecode from /usr/share/javatoolkit ..
>>> Original instance of package unmerged safely.
 * Byte compiling python modules for python-2.4 .. ...                [ ok ]
>>> dev-java/javatoolkit-0.2.0-r1 merged.

>>> No packages selected for removal by clean
>>> Auto-cleaning packages...

>>> No outdated packages were found on your system.
 * GNU info directory index is up-to-date.
# ls /usr/share/javatoolkit/lib
ls: cannot access /usr/share/javatoolkit/lib: není souborem ani adresářem
# ls -a /usr/share/javatoolkit/   
.  ..  build-xml-source-target.xslt  pym
# grep /usr/share/javatoolkit/lib /var/db/pkg/dev-java/javatoolkit-0.2.0-r1/CONTENTS 
dir /usr/share/javatoolkit/lib

Is it the same bug?
Comment 49 Marius Mauch (RETIRED) gentoo-dev 2007-10-23 15:13:15 UTC
(In reply to comment #48)
> Is it the same bug?

Likely.
Comment 50 Zac Medico gentoo-dev 2007-10-23 15:45:43 UTC
(In reply to comment #48)
> --- replaced dir /usr/share/javatoolkit/lib
> --- replaced obj /usr/share/javatoolkit/build-xml-source-target.xslt
> --- replaced dir /usr/share/javatoolkit
> ....
> --- replaced dir /usr
>  * Cleaning orphaned Python bytecode from /usr/share/javatoolkit ..

I was able to reproduce that and I found that portage properly keeps the directory, but it is then removed during the pkg_postrm() phase. So if it's a bug, it's a problem with the ebuild or one of the eclasses it inherits. There is nothing that portage can do about it. If it's a problem then you need to file a bug with the ebuild maintainer (the java herd).
Comment 51 Petr Bahula 2007-10-23 17:40:16 UTC
(In reply to comment #50)
> (In reply to comment #48)

> I was able to reproduce that and I found that portage properly keeps the
> directory, but it is then removed during the pkg_postrm() phase. So if it's a
> bug, it's a problem with the ebuild or one of the eclasses it inherits. There
> is nothing that portage can do about it. If it's a problem then you need to
> file a bug with the ebuild maintainer (the java herd).
> 

OK, I check it and then send as new bug to relevant maintainer. Thanks.
Comment 52 Zac Medico gentoo-dev 2007-11-11 20:42:58 UTC
I think we can consider this bug fixed due to the follow enhancements that are available in >=portage-2.1.3.16:

1) The unmerge process now does proper reference counting within each package SLOT, so a new version's files can never be unmerged with an old version.

2) FEATURES=unmerge-orphans is enabled by default, preventing orphan files from being left behind due to changed checksums or timestamps.

3) The default behavior for file collisions (between different package SLOTs) is to generate an eerror log message via the elog framework. This ensures that any relevant collisions are recorded so that they can be dealt with when necessary.
Comment 53 Zac Medico gentoo-dev 2018-02-02 00:53:42 UTC
*** Bug 159165 has been marked as a duplicate of this bug. ***