| Summary: | emerge @preserved-rebuild is not deleting some libraries | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | William Hubbs <williamh> |
| Component: | New packages | Assignee: | Portage team <dev-portage> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | david.bouriaud, esigra, flameeyes, SebastianLuther, specious, trenton.d.adams, vitaliy.osypenko |
| Priority: | High | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Bug Depends on: | |||
| Bug Blocks: | 240323 | ||
| Attachments: | emerge-info.txt | ||
Created attachment 209764 [details]
emerge-info.txt
Here is my emerge --info.
Isn't linking against an unversioned .so wrong? I'd say it's a bug in the packages that links against the preserved libs. A workaround is to manually delete the preserved libs and to run revdep-rebuild. That seemed to fix the issue. By the way,those files in /lib were not actually libs,but symlinks. Now in /lib there are no unversioned names for those libraries. Is that ok? @flameeyes: could you give us some insight who is at fault here? (Hope you don't mind getting CCed on random bugs). Remind me about this next week please. I still have this problem: !!! existing preserved libs: >>> package: sys-libs/e2fsprogs-libs-1.41.9 * - /lib/libuuid.so * used by /bin/mount (sys-apps/util-linux-2.16.1) * used by /bin/umount (sys-apps/util-linux-2.16.1) * used by /sbin/blkid (sys-apps/util-linux-2.16.1) * used by 378 other files * - /lib/libblkid.so * used by /bin/mount (sys-apps/util-linux-2.16.1) * used by /bin/umount (sys-apps/util-linux-2.16.1) * used by /sbin/blkid (sys-apps/util-linux-2.16.1) * used by 10 other files Use emerge @preserved-rebuild to rebuild packages using these libraries I can keep running that command until doomsday dawn, but it will always emerge the same 35 packages! Both /lib/libuuid.so and /lib/libblkid.so belong to sys-libs/e2fsprogs-libs-1.41.9, which is actually the newest version available! Some information: # ls -l /lib/libuuid.so* /lib/libblkid.so* lrwxrwxrwx 1 root root 13 2 jan 2009 /lib/libblkid.so -> libblkid.so.1 lrwxrwxrwx 1 root root 17 18 nov 00.35 /lib/libblkid.so.1 -> libblkid.so.1.1.0 -rwxr-xr-x 1 root root 83740 18 nov 00.35 /lib/libblkid.so.1.1.0 lrwxrwxrwx 1 root root 12 2 jan 2009 /lib/libuuid.so -> libuuid.so.1 lrwxrwxrwx 1 root root 16 18 nov 00.35 /lib/libuuid.so.1 -> libuuid.so.1.3.0 -rwxr-xr-x 1 root root 18036 18 nov 00.35 /lib/libuuid.so.1.3.0 # equery b /lib/libuuid.so* /lib/libblkid.so* [ Searching for file(s) /lib/libuuid.so,/lib/libuuid.so.1,/lib/libuuid.so.1.3.0,/lib/libblkid.so,/lib/libblkid.so.1,/lib/libblkid.so.1.1.0 in *... ] sys-apps/util-linux-2.16.1 (/lib/libuuid.so.1 -> libuuid.so.1.3.0) sys-apps/util-linux-2.16.1 (/lib/libblkid.so.1.1.0) sys-apps/util-linux-2.16.1 (/lib/libblkid.so.1 -> libblkid.so.1.1.0) sys-apps/util-linux-2.16.1 (/lib/libuuid.so.1.3.0) sys-libs/e2fsprogs-libs-1.41.9 (/lib/libblkid.so -> libblkid.so.1) sys-libs/e2fsprogs-libs-1.41.9 (/lib/libuuid.so -> libuuid.so.1) # emerge -s e2fsprogs-libs util-linux Searching... [ Results for search key : e2fsprogs-libs ] [ Applications found : 1 ] * sys-libs/e2fsprogs-libs Latest version available: 1.41.9 Latest version installed: 1.41.9 Size of files: 484 kB Homepage: http://e2fsprogs.sourceforge.net/ Description: e2fsprogs libraries (common error and subsystem) License: GPL-2 Searching... [ Results for search key : util-linux ] [ Applications found : 1 ] * sys-apps/util-linux Latest version available: 2.16.1 Latest version installed: 2.16.1 Size of files: 3,464 kB Homepage: http://www.kernel.org/pub/linux/utils/util-linux-ng/ Description: Various useful Linux utilities License: GPL-2 (In reply to comment #5) > Remind me about this next week please. > remind :) Okay somebody with the problem can please run this for me as root? scanelf -n /bin/mount What I'm afraid the problem is: - some (old?) libuuid or libblkid library used an unversioned soname (libuuid.so); - this lead to binaries having unversioned NEEDED lines in .dynamic; - and this finally lead Portage to try saving the .so links since they result needed. Fixing this is … difficult; one solution would be to not preserve libraries that have unversioned sonames (like libuuid.so as it is here); unfortunately this doesn't solve the problem, since util-linux would require to be rebuilt most likely. Probably would also warrant one more QA warning for unversioned SONAME values (we *already* have a warning for missing SONAME), and an audit of the tree… I thought about this as well, and concluded the same. We should probably make Portage die with a QA check when a package wants to install a lib with unversioned soname, while a versioned lib at the same time exists. This is relatively easy I think. However, it doesn't solve the problem people have right now, and it's hard to get out of this loop. I'm afraid manual intervention is the only solution here :/ # scanelf -n /bin/mount TYPE NEEDED FILE ET_EXEC libblkid.so.1,libuuid.so.1,libc.so.6 /bin/mount Okay now that seems a Portage mistake then… I'd say just remove any "preserved" symlink; ldconfig (env-update) will recreate them as needed. the difference I see here is this: lrwxrwxrwx 1 root root 13 Jan 27 2009 /lib/libblkid.so -> libblkid.so.1 lrwxrwxrwx 1 root root 15 Jan 27 2009 /lib/libblkid.so.1 -> libblkid.so.1.0 -rwxr-xr-x 1 root root 47184 Jan 27 2009 /lib/libblkid.so.1.0 and e.g. lrwxrwxrwx 1 root root 13 Feb 28 2008 /lib/libz.so -> libz.so.1.2.3 lrwxrwxrwx 1 root root 13 Feb 28 2008 /lib/libz.so.1 -> libz.so.1.2.3 -rwxr-xr-x 1 root root 84272 Feb 28 2008 /lib/libz.so.1.2.3 There is not a chain of symlinks for most libraries, they all point to the real version, where for libblkid there is a chain going from .so -> soname -> real lib. Now this might be the cause of Portage thinking the actual needed entry is .so instead of .so.1, in some cases. It seems to have recorded it correctly at least for me in the NEEDED.ELF.2 file: X86_64;/bin/mount;;;libblkid.so.1,libuuid.so.1,libc.so.6 Well my idea should work in all cases, since you only need to preserve a library *file* that has the given SONAME, not the symlinks: the libfoo.so symlink is unnecessary at runtime, and the $SONAME symlink is generated by ldconfig/env-update [1]. So the idea of not preserving symlinks should fix all this at once I guess. [1] http://blog.flameeyes.eu/2009/10/27/a-shared-library-by-any-other-name The design of portage is to only "save" the soname and the real file, and the soname usually is a symlink to that file. Do I understand you correctly that you basically suggest to "materialise" each soname (that thing that is referenced -- basically what Portage finds being pointed to) such that it only preserves what's actually being needed by other objects (in the correct case only the soname object)? If that's what you suggest, then I think that might be a simplification of the current code as well, as it wouldn't have to check/take into account the symlink relation any more. I should add: Portage finds the "real" file by traversing symlinks starting from the soname object IIRC. What I'm saying is to check for the needed soname not by using the currently present symlinks but the data in NEEDED.ELF.2 (soname and abi pair); then just preserve *that* file, and leave the symlinks to be unmerged normally. I don't think I fully understand you. Portage builds a map of consumers and providers. It then matches all consumers on the providers, and if it sees that a provider goes away while there is still a consumer for it, it will preserve the provider. Portage gets the consumers and providers from the NEEDED.ELF.2 files. Consumers consume the `scanelf --needed` returned libs. Providers are all the files found by scanelf, excluding symlinks, going by their soname returned by `scanelf --soname`. For Portage to preserve such soname (a provider), it preserves the target in case of a symlink too, leaving the original symlink in tact. What I'm saying is to basically keep the same behaviour but *only preserving files* leaving the symlinks to their destiny… so if you got something like this: libfoo.so.1 → libfoo.so.2.3.4 (SONAME libfoo.so.2) which, for instance, might have been doing by a crazy user who knows nothing about sonames and ABIs, you don't get to preserve libfoo.so.1 (since the soname does not match). If you got libfoo.so.1 → libfoo.so.1.4.5 (SONAME libfoo.so.1) then just preserve libfoo.so.1.4.5, leaving libfoo.so.1 to be deleted; it'll be left as an exercise to ldconfig to fix that up. (In reply to comment #19) > What I'm saying is to basically keep the same behaviour but *only preserving > files* leaving the symlinks to their destiny… so if you got something like > this: > > libfoo.so.1 → libfoo.so.2.3.4 (SONAME libfoo.so.2) > > which, for instance, might have been doing by a crazy user who knows nothing > about sonames and ABIs, you don't get to preserve libfoo.so.1 (since the soname > does not match). This case is pretty much unrepairable IMO. If it happens to accidentially work, then that's luck for crazy user, but Portage can't figure that out, ever. I think it doesn't detect so at the moment either, because it only looks at the SONAME, not the filename. > If you got > > libfoo.so.1 → libfoo.so.1.4.5 (SONAME libfoo.so.1) > > then just preserve libfoo.so.1.4.5, leaving libfoo.so.1 to be deleted; it'll be > left as an exercise to ldconfig to fix that up. This is not going to work. Maybe this accidentially works on Linux (does ldconfig recreate symlinks for SONAME targets?), but it feels pretty wrong. If an application is having libfoo.so.1 in its NEEDED entries, then that lib should exist. Removing it will cause damage to all Prefix archs (including Linux) at least. Next to that, it means Portage has to rely on a third party actually doing some necessary work. What we /can/ do is move libfoo.so.1.4.5 to libfoo.so.1 and then only preserve that file. This is what I initially suggested. However, that may be just an improvement, it still doesn't explain why this situation occurs with the current code. *** Bug 294425 has been marked as a duplicate of this bug. *** I've got the same problem. What I see:
# emerge -1 sys-libs/e2fsprogs-libs
...
>>> Installing (1 of 1) sys-libs/e2fsprogs-libs-1.41.9
>>> needed sym /lib64/libblkid.so
!!! File '/lib64/libblkid.so.1.1.0' will not be preserved due to missing contents entry
Any clues?
I have the same situation and currently the counter is at 400 files that are supposedly using preserved versions. Now before I start removing random libraries, is there a safe way to fix this? (In reply to comment #23) > Now before I start removing random libraries, is there a safe way to fix this? The safest thing is to move them to a temporary and if that breaks something you can just move it back (if the normal mv binary is broken, us busybox). After you've moved the libraries to a temporary location, run revdep-rebuild. *** Bug 297491 has been marked as a duplicate of this bug. *** I solved it like this: - Open up a root terminal (and make sure you don't run out of power or reboot or whatever during this exercise) - emerge --fetchonly util-linux e2fsprogs-libs - emerge --unmerge e2fsprogs-libs - emerge -av1 util-linux - emerge -av1 e2fsprogs-libs The same thing happens with kalgebra/readline... !!! existing preserved libs: >>> package: sys-libs/readline-6.0_p3 * - /lib64/libreadline.so * - /lib64/libreadline.so.5 * - /lib64/libreadline.so.5.2 * used by /usr/bin/calgebra (kde-base/kalgebra-4.3.3) I ran "emerge @preserved-rebuild", and I get the same output. The following command resolves the issue, the same as "comment #23" emerge -C =kde-base/kalgebra-4.3.3 && emerge -1 kde-base/kalgebra (In reply to comment #27) > The following command resolves the issue, the same as "comment #23" > emerge -C =kde-base/kalgebra-4.3.3 && emerge -1 kde-base/kalgebra > err, um, "comment #26", sorry. looks like this is still a problem with sys-apps/portage-2.2.0_alpha24: >>> Installing (1 of 1) media-libs/libpng-1.5.1 * Removing /usr/share/info >>> needed sym /usr/lib64/libpng14.so >>> needed sym /usr/lib64/libpng14.so.14 >>> needed obj /usr/lib64/libpng14.so.14.5.0 it makes no sense to preserve libpng14.so as nothing needs it. looking at all 'NEEDED*' files in /var/db/pkg/ shows that only libpng14.so.14 is needed which means only that should be preserved. it happens to be a symlink, so the file it points to too is preserved (libpng14.so.14.5.0). but links that point to a needed file should not be preserved. Portage is simply not deleting anything reporting a soname that is needed, without caring whether it's a symlink or not... Maybe we can simply get it to remove anything reporting the soname that isn't a symlink, ldconfig will restore the symlink for us on Linux at least. that would require ldconfig to be run, and it introduces a window where the SONAME doesnt exist. simply fix the logic -- we're already tracking the SONAME in question (otherwise it wouldnt get flagged in the first place), so only preserve that and any links below it. This should be fixed since v2.2.0_alpha42: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=a30cc13e70baad6abf41224afadf4a91dd3eb828 |
All, I get the followingafter I do an emerge: !!! existing preserved libs: >>> package: sys-libs/e2fsprogs-libs-1.41.9 * - /lib/libuuid.so * used by /bin/mount (sys-apps/util-linux-2.16.1) * used by /bin/umount (sys-apps/util-linux-2.16.1) * used by /sbin/blkid (sys-apps/util-linux-2.16.1) * used by 20 other files * - /lib/libblkid.so * used by /bin/mount (sys-apps/util-linux-2.16.1) * used by /bin/umount (sys-apps/util-linux-2.16.1) * used by /sbin/blkid (sys-apps/util-linux-2.16.1) * used by 8 other files Use emerge @preserved-rebuild to rebuild packages using these libraries william@linux1 ~ $ emerge -p @preserved-rebuild These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild R ] sys-apps/util-linux-2.16.1 [ebuild R ] dev-libs/apr-1.3.9 [ebuild R ] sys-fs/e2fsprogs-1.41.9 [ebuild R ] dev-libs/apr-util-1.3.9 [ebuild R ] dev-util/subversion-1.6.5 [If I emerge @preserved rebuild, these packages emerge successfully, but I still get the message above.] william@linux1 ~ $ portageq list_preserved_libs / sys-libs/e2fsprogs-libs-1.41.9 /lib/libblkid.so /lib/libuuid.so [note that this shows an extra library not mentioned in the output above.] Any assistance getting passed this would be greatly appreciated.