Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bugzilla DB migration completed. Please report issues to Infra team via email via infra@gentoo.org or IRC
Bug 292566 - emerge @preserved-rebuild is not deleting some libraries
Summary: emerge @preserved-rebuild is not deleting some libraries
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: New packages (show other bugs)
Hardware: All Linux
: High normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
: 294425 297491 (view as bug list)
Depends on:
Blocks: preserve-libs
  Show dependency tree
 
Reported: 2009-11-09 19:30 UTC by William Hubbs
Modified: 2011-07-16 18:26 UTC (History)
7 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
emerge-info.txt (emerge-info.txt,3.14 KB, text/plain)
2009-11-09 19:30 UTC, William Hubbs
Details

Note You need to log in before you can comment on or make changes to this bug.
Description William Hubbs gentoo-dev 2009-11-09 19:30:28 UTC
All,

I get the followingafter I do an emerge:

!!! existing preserved libs:
>>> package: sys-libs/e2fsprogs-libs-1.41.9
 *  - /lib/libuuid.so
 *      used by /bin/mount (sys-apps/util-linux-2.16.1)
 *      used by /bin/umount (sys-apps/util-linux-2.16.1)
 *      used by /sbin/blkid (sys-apps/util-linux-2.16.1)
 *      used by 20 other files
 *  - /lib/libblkid.so
 *      used by /bin/mount (sys-apps/util-linux-2.16.1)
 *      used by /bin/umount (sys-apps/util-linux-2.16.1)
 *      used by /sbin/blkid (sys-apps/util-linux-2.16.1)
 *      used by 8 other files
Use emerge @preserved-rebuild to rebuild packages using these libraries

william@linux1 ~ $ emerge -p @preserved-rebuild

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild   R   ] sys-apps/util-linux-2.16.1 
[ebuild   R   ] dev-libs/apr-1.3.9 
[ebuild   R   ] sys-fs/e2fsprogs-1.41.9 
[ebuild   R   ] dev-libs/apr-util-1.3.9 
[ebuild   R   ] dev-util/subversion-1.6.5 

[If I emerge @preserved rebuild, these packages emerge successfully, but I
still get the message above.]

william@linux1 ~ $ portageq list_preserved_libs /
sys-libs/e2fsprogs-libs-1.41.9 /lib/libblkid.so /lib/libuuid.so
[note that this shows an extra library not mentioned in the output above.]

Any assistance getting passed this would be greatly appreciated.
Comment 1 William Hubbs gentoo-dev 2009-11-09 19:30:58 UTC
Created attachment 209764 [details]
emerge-info.txt

Here is my emerge --info.
Comment 2 Sebastian Luther (few) 2009-11-09 19:40:03 UTC
Isn't linking against an unversioned .so wrong? I'd say it's a bug in the packages that links against the preserved libs. A workaround is to manually delete the preserved libs and to run revdep-rebuild.
Comment 3 William Hubbs gentoo-dev 2009-11-09 22:02:43 UTC
That seemed to fix the issue.

By the way,those files in /lib were not actually libs,but symlinks.
Now in /lib there are no unversioned names for those libraries.  Is that ok?

Comment 4 Sebastian Luther (few) 2009-11-09 22:06:00 UTC
@flameeyes: could you give us some insight who is at fault here? (Hope you don't mind getting CCed on random bugs).
Comment 5 Diego Elio Pettenò (RETIRED) gentoo-dev 2009-11-10 07:11:07 UTC
Remind me about this next week please.
Comment 6 Erik 2009-11-18 12:34:12 UTC
I still have this problem:
!!! existing preserved libs:
>>> package: sys-libs/e2fsprogs-libs-1.41.9
 *  - /lib/libuuid.so
 *      used by /bin/mount (sys-apps/util-linux-2.16.1)
 *      used by /bin/umount (sys-apps/util-linux-2.16.1)
 *      used by /sbin/blkid (sys-apps/util-linux-2.16.1)
 *      used by 378 other files
 *  - /lib/libblkid.so
 *      used by /bin/mount (sys-apps/util-linux-2.16.1)
 *      used by /bin/umount (sys-apps/util-linux-2.16.1)
 *      used by /sbin/blkid (sys-apps/util-linux-2.16.1)
 *      used by 10 other files
Use emerge @preserved-rebuild to rebuild packages using these libraries


I can keep running that command until doomsday dawn, but it will always emerge the same 35 packages! Both /lib/libuuid.so and /lib/libblkid.so belong to sys-libs/e2fsprogs-libs-1.41.9, which is actually the newest version available! Some information:

# ls -l /lib/libuuid.so* /lib/libblkid.so*
lrwxrwxrwx 1 root root    13  2 jan  2009 /lib/libblkid.so -> libblkid.so.1
lrwxrwxrwx 1 root root    17 18 nov 00.35 /lib/libblkid.so.1 -> libblkid.so.1.1.0
-rwxr-xr-x 1 root root 83740 18 nov 00.35 /lib/libblkid.so.1.1.0
lrwxrwxrwx 1 root root    12  2 jan  2009 /lib/libuuid.so -> libuuid.so.1
lrwxrwxrwx 1 root root    16 18 nov 00.35 /lib/libuuid.so.1 -> libuuid.so.1.3.0
-rwxr-xr-x 1 root root 18036 18 nov 00.35 /lib/libuuid.so.1.3.0
# equery b /lib/libuuid.so* /lib/libblkid.so*
[ Searching for file(s) /lib/libuuid.so,/lib/libuuid.so.1,/lib/libuuid.so.1.3.0,/lib/libblkid.so,/lib/libblkid.so.1,/lib/libblkid.so.1.1.0 in *... ]
sys-apps/util-linux-2.16.1 (/lib/libuuid.so.1 -> libuuid.so.1.3.0)
sys-apps/util-linux-2.16.1 (/lib/libblkid.so.1.1.0)
sys-apps/util-linux-2.16.1 (/lib/libblkid.so.1 -> libblkid.so.1.1.0)
sys-apps/util-linux-2.16.1 (/lib/libuuid.so.1.3.0)
sys-libs/e2fsprogs-libs-1.41.9 (/lib/libblkid.so -> libblkid.so.1)
sys-libs/e2fsprogs-libs-1.41.9 (/lib/libuuid.so -> libuuid.so.1)
# emerge -s e2fsprogs-libs util-linux
Searching...
[ Results for search key : e2fsprogs-libs ]
[ Applications found : 1 ]

*  sys-libs/e2fsprogs-libs
      Latest version available: 1.41.9
      Latest version installed: 1.41.9
      Size of files: 484 kB
      Homepage:      http://e2fsprogs.sourceforge.net/
      Description:   e2fsprogs libraries (common error and subsystem)
      License:       GPL-2

Searching...
[ Results for search key : util-linux ]
[ Applications found : 1 ]

*  sys-apps/util-linux
      Latest version available: 2.16.1
      Latest version installed: 2.16.1
      Size of files: 3,464 kB
      Homepage:      http://www.kernel.org/pub/linux/utils/util-linux-ng/
      Description:   Various useful Linux utilities
      License:       GPL-2
Comment 7 Sebastian Luther (few) 2009-11-20 18:23:39 UTC
(In reply to comment #5)
> Remind me about this next week please.
> 

remind :)
Comment 8 Diego Elio Pettenò (RETIRED) gentoo-dev 2009-11-20 18:46:06 UTC
Okay somebody with the problem can please run this for me as root?

scanelf -n /bin/mount

What I'm afraid the problem is:

 - some (old?) libuuid or libblkid library used an unversioned soname (libuuid.so);
 - this lead to binaries having unversioned NEEDED lines in .dynamic;
 - and this finally lead Portage to try saving the .so links since they result needed.

Fixing this is … difficult; one solution would be to not preserve libraries that have unversioned sonames (like libuuid.so as it is here); unfortunately this doesn't solve the problem, since util-linux would require to be rebuilt most likely.

Probably would also warrant one more QA warning for unversioned SONAME values (we *already* have a warning for missing SONAME), and an audit of the tree…
Comment 9 Fabian Groffen gentoo-dev 2009-11-20 18:52:00 UTC
I thought about this as well, and concluded the same.  We should probably make Portage die with a QA check when a package wants to install a lib with unversioned soname, while a versioned lib at the same time exists.  This is relatively easy I think.  However, it doesn't solve the problem people have right now, and it's hard to get out of this loop.
Comment 10 Diego Elio Pettenò (RETIRED) gentoo-dev 2009-11-20 19:17:25 UTC
I'm afraid manual intervention is the only solution here :/
Comment 11 Erik 2009-11-20 20:24:07 UTC
# scanelf -n /bin/mount
 TYPE   NEEDED FILE
ET_EXEC libblkid.so.1,libuuid.so.1,libc.so.6 /bin/mount
Comment 12 Diego Elio Pettenò (RETIRED) gentoo-dev 2009-11-20 20:31:56 UTC
Okay now that seems a Portage mistake then… I'd say just remove any "preserved" symlink; ldconfig (env-update) will recreate them as needed.
Comment 13 Fabian Groffen gentoo-dev 2009-11-20 20:54:40 UTC
the difference I see here is this:

lrwxrwxrwx 1 root root    13 Jan 27  2009 /lib/libblkid.so -> libblkid.so.1
lrwxrwxrwx 1 root root    15 Jan 27  2009 /lib/libblkid.so.1 -> libblkid.so.1.0
-rwxr-xr-x 1 root root 47184 Jan 27  2009 /lib/libblkid.so.1.0

and e.g.

lrwxrwxrwx 1 root root    13 Feb 28  2008 /lib/libz.so -> libz.so.1.2.3
lrwxrwxrwx 1 root root    13 Feb 28  2008 /lib/libz.so.1 -> libz.so.1.2.3
-rwxr-xr-x 1 root root 84272 Feb 28  2008 /lib/libz.so.1.2.3

There is not a chain of symlinks for most libraries, they all point to the real version, where for libblkid there is a chain going from .so -> soname -> real lib.

Now this might be the cause of Portage thinking the actual needed entry is .so instead of .so.1, in some cases.  It seems to have recorded it correctly at least for me in the NEEDED.ELF.2 file:

X86_64;/bin/mount;;;libblkid.so.1,libuuid.so.1,libc.so.6
Comment 14 Diego Elio Pettenò (RETIRED) gentoo-dev 2009-11-20 21:01:10 UTC
Well my idea should work in all cases, since you only need to preserve a library *file* that has the given SONAME, not the symlinks: the libfoo.so symlink is unnecessary at runtime, and the $SONAME symlink is generated by ldconfig/env-update [1]. So the idea of not preserving symlinks should fix all this at once I guess.

[1] http://blog.flameeyes.eu/2009/10/27/a-shared-library-by-any-other-name
Comment 15 Fabian Groffen gentoo-dev 2009-11-20 21:05:51 UTC
The design of portage is to only "save" the soname and the real file, and the soname usually is a symlink to that file.

Do I understand you correctly that you basically suggest to "materialise" each soname (that thing that is referenced -- basically what Portage finds being pointed to) such that it only preserves what's actually being needed by other objects (in the correct case only the soname object)?

If that's what you suggest, then I think that might be a simplification of the current code as well, as it wouldn't have to check/take into account the symlink relation any more.
Comment 16 Fabian Groffen gentoo-dev 2009-11-20 21:10:08 UTC
I should add: Portage finds the "real" file by traversing symlinks starting from the soname object IIRC.
Comment 17 Diego Elio Pettenò (RETIRED) gentoo-dev 2009-11-20 21:10:23 UTC
What I'm saying is to check for the needed soname not by using the currently present symlinks but the data in NEEDED.ELF.2 (soname and abi pair); then just preserve *that* file, and leave the symlinks to be unmerged normally.
Comment 18 Fabian Groffen gentoo-dev 2009-11-20 21:20:38 UTC
I don't think I fully understand you.

Portage builds a map of consumers and providers.  It then matches all consumers on the providers, and if it sees that a provider goes away while there is still a consumer for it, it will preserve the provider.

Portage gets the consumers and providers from the NEEDED.ELF.2 files.  Consumers consume the `scanelf --needed` returned libs.  Providers are all the files found by scanelf, excluding symlinks, going by their soname returned by `scanelf --soname`.

For Portage to preserve such soname (a provider), it preserves the target in case of a symlink too, leaving the original symlink in tact.
Comment 19 Diego Elio Pettenò (RETIRED) gentoo-dev 2009-11-20 21:35:06 UTC
What I'm saying is to basically keep the same behaviour but *only preserving files* leaving the symlinks to their destiny… so if you got something like this:

libfoo.so.1 → libfoo.so.2.3.4 (SONAME libfoo.so.2)

which, for instance, might have been doing by a crazy user who knows nothing about sonames and ABIs, you don't get to preserve libfoo.so.1 (since the soname does not match).

If you got

libfoo.so.1 → libfoo.so.1.4.5 (SONAME libfoo.so.1)

then just preserve libfoo.so.1.4.5, leaving libfoo.so.1 to be deleted; it'll be left as an exercise to ldconfig to fix that up.
Comment 20 Fabian Groffen gentoo-dev 2009-11-21 07:39:29 UTC
(In reply to comment #19)
> What I'm saying is to basically keep the same behaviour but *only preserving
> files* leaving the symlinks to their destiny… so if you got something like
> this:
> 
> libfoo.so.1 → libfoo.so.2.3.4 (SONAME libfoo.so.2)
>
> which, for instance, might have been doing by a crazy user who knows nothing
> about sonames and ABIs, you don't get to preserve libfoo.so.1 (since the soname
> does not match).

This case is pretty much unrepairable IMO.  If it happens to accidentially work, then that's luck for crazy user, but Portage can't figure that out, ever.
I think it doesn't detect so at the moment either, because it only looks at the SONAME, not the filename.

> If you got
> 
> libfoo.so.1 → libfoo.so.1.4.5 (SONAME libfoo.so.1)
> 
> then just preserve libfoo.so.1.4.5, leaving libfoo.so.1 to be deleted; it'll be
> left as an exercise to ldconfig to fix that up.

This is not going to work.  Maybe this accidentially works on Linux (does ldconfig recreate symlinks for SONAME targets?), but it feels pretty wrong.  If an application is having libfoo.so.1 in its NEEDED entries, then that lib should exist.  Removing it will cause damage to all Prefix archs (including Linux) at least.  Next to that, it means Portage has to rely on a third party actually doing some necessary work.

What we /can/ do is move libfoo.so.1.4.5 to libfoo.so.1 and then only preserve that file.  This is what I initially suggested.

However, that may be just an improvement, it still doesn't explain why this situation occurs with the current code.
Comment 21 Diego Elio Pettenò (RETIRED) gentoo-dev 2009-11-24 15:07:21 UTC
*** Bug 294425 has been marked as a duplicate of this bug. ***
Comment 22 Ildar Sagdejev 2009-11-29 04:15:18 UTC
I've got the same problem.  What I see:

# emerge -1 sys-libs/e2fsprogs-libs

...

>>> Installing (1 of 1) sys-libs/e2fsprogs-libs-1.41.9
>>> needed    sym /lib64/libblkid.so
!!! File '/lib64/libblkid.so.1.1.0' will not be preserved due to missing contents entry

Any clues?
Comment 23 Berend Dekens 2009-12-16 14:08:47 UTC
I have the same situation and currently the counter is at 400 files that are supposedly using preserved versions.

Now before I start removing random libraries, is there a safe way to fix this?
Comment 24 Zac Medico gentoo-dev 2009-12-17 00:55:49 UTC
(In reply to comment #23)
> Now before I start removing random libraries, is there a safe way to fix this?

The safest thing is to move them to a temporary and if that breaks something you can just move it back (if the normal mv binary is broken, us busybox). After you've moved the libraries to a temporary location, run revdep-rebuild.
Comment 25 Trenton D. Adams 2009-12-19 17:14:21 UTC
*** Bug 297491 has been marked as a duplicate of this bug. ***
Comment 26 Berend Dekens 2009-12-21 23:00:29 UTC
I solved it like this:

- Open up a root terminal (and make sure you don't run out of power or reboot or whatever during this exercise)
- emerge --fetchonly util-linux e2fsprogs-libs
- emerge --unmerge e2fsprogs-libs
- emerge -av1 util-linux
- emerge -av1 e2fsprogs-libs
Comment 27 Trenton D. Adams 2009-12-22 01:36:38 UTC
The same thing happens with kalgebra/readline...

!!! existing preserved libs:
>>> package: sys-libs/readline-6.0_p3
 *  - /lib64/libreadline.so          
 *  - /lib64/libreadline.so.5        
 *  - /lib64/libreadline.so.5.2      
 *      used by /usr/bin/calgebra (kde-base/kalgebra-4.3.3)

I ran "emerge @preserved-rebuild", and I get the same output.

The following command resolves the issue, the same as "comment #23"
emerge -C =kde-base/kalgebra-4.3.3 && emerge -1 kde-base/kalgebra

Comment 28 Trenton D. Adams 2009-12-22 01:37:11 UTC
(In reply to comment #27)
> The following command resolves the issue, the same as "comment #23"
> emerge -C =kde-base/kalgebra-4.3.3 && emerge -1 kde-base/kalgebra
> 

err, um, "comment #26", sorry.
Comment 29 SpanKY gentoo-dev 2011-02-20 20:08:47 UTC
looks like this is still a problem with sys-apps/portage-2.2.0_alpha24:
>>> Installing (1 of 1) media-libs/libpng-1.5.1
 * Removing /usr/share/info
>>> needed    sym /usr/lib64/libpng14.so
>>> needed    sym /usr/lib64/libpng14.so.14
>>> needed    obj /usr/lib64/libpng14.so.14.5.0

it makes no sense to preserve libpng14.so as nothing needs it.  looking at all 'NEEDED*' files in /var/db/pkg/ shows that only libpng14.so.14 is needed which means only that should be preserved.

it happens to be a symlink, so the file it points to too is preserved (libpng14.so.14.5.0).  but links that point to a needed file should not be preserved.
Comment 30 Diego Elio Pettenò (RETIRED) gentoo-dev 2011-02-20 23:52:01 UTC
Portage is simply not deleting anything reporting a soname that is needed, without caring whether it's a symlink or not...

Maybe we can simply get it to remove anything reporting the soname that isn't a symlink, ldconfig will restore the symlink for us on Linux at least.
Comment 31 SpanKY gentoo-dev 2011-02-21 07:27:01 UTC
that would require ldconfig to be run, and it introduces a window where the SONAME doesnt exist.  simply fix the logic -- we're already tracking the SONAME in question (otherwise it wouldnt get flagged in the first place), so only preserve that and any links below it.
Comment 32 Zac Medico gentoo-dev 2011-07-16 18:26:58 UTC
This should be fixed since v2.2.0_alpha42:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=a30cc13e70baad6abf41224afadf4a91dd3eb828