When shared libraries change ABI, they change major version number. Let's see libfoo changing to version 2.0. The problem is that all executables linked to the old libfoo.so.1 will be broken if the old package is unmerged. (executables are linked to the file specified in SONAME of libfoo.so) This isn't serious in most case, as most libraries don't change ABI freqently, and critical programs (in first line those needed for compiling) are linked to a minimal set of libraries. But such a change is disastrous for libcurses (bash is linked to it, but statically due to bug #51901 ) and openssl includes an ugly hack to avoid the very same situation.
unless you have some kind of clean solution to propose, i dont have one the openssl stuff is ugly, but that's because i think aliz's has propogated it longer than he should of ... the way i do it now (and how i've done it with gdbm and readline for example), is in src_install() check to see if the old api library exists in $ROOT, and if so, copy it to $D ... then warn users that they have to do a revdep-rebuild in order to fix their system ... a few versions down the line and we drop the legacy support
Hm, let's see: 1. "ebuild qmerge" and "unmerge" should check every processed file if it is an ELF binary (magic patterns) 2. get the NEEDED entries in ELF binaries 3. make (or delete, respectively) a database entry indexed with the library name and value of the examined object 4. in case of "unmerge" remove the file only if noone is depending on it (0 values in the database) 5. repeat from step 2 for every NEEDED objects The drawback of this procedure is that portage needs to maintain an additional database. However unmerging libpam for example will not make the user's system unusable.
well then, that looks quite good :)
The NEEDED entries can be extracted from the dynamic section with "readelf -d -W a.out | grep '(NEEDED)'". "ldd" is recursive while finding the needed libraries.
Someone want to write something for it?
Basically BINCOMPAT. App A,B,C all rely on libfoo-1.0. Libfoo-1.0 is upgraded to 2.0 and breaks compatability. How does portage rebuild A,B,C. It's probably way out there.
This concept is not about rebuilding apps A, B and C, but the keep "unmerge libfoo-1.0" on the safe side. Let's assume, bash would be using shared libncurses. The current version is libncurses.so.5.4, but if libncurses.so.6.0 came out, after merge 6.0 and unmerge 5.4 we had a broken /bin/bash. This would prevent us to do anything, including remerge bash!
"the concept" is having bash not break at all when ncurses is rebuilt. Whether it's by storing the old lib around, which is a huge hack IMHO, or by recompiling packages against the new version (thus being able to drop the old lib = clean system). The first one...I dislike, it doesn't cover all scenarios; if glibc breaks binary compatability and you backed it up you have to hop on a liveCD to restore it. And at that point the backup was useless. Nothing stops you from backing up libfoo yourself, or keeping snapshots of your system.
Reopening for consideration.
Well, I like the idea outlined in comment #2. While rebuilding is the final solution to fix the problem it's still useful the keep the old lib version around, as you'll need it for the time the rebuild takes. Shouldn't be too difficult to implement as we already have the mapping in vdb, I'll take a shot at this.
genone, This is both a good idea and a really bad one. Before you commit anything please put it under heavy testing and ask for testing. My main concern would be when using the ROOT=/some/dir emerge -Ke world that things would start breaking like they do with openssl leaving orphan linking behind. Or that glsa updates would still end up using the old/wrong versions when things should really be upgraded.
(In reply to comment #11) > genone, > This is both a good idea and a really bad one. Before you commit anything > please > put it under heavy testing and ask for testing. Tried that in the past (with manifest2 for example), didn't work very well. Mid that it currently is hidden behind an undocumented FEATURE setting, and I don't plan to announce it until it has been tested and accepted by the dev community. Unfortunately people don't test stuff when it's not in the mainstream version, especially for stuff that doesn't give any direct results but affect long-term behavior. > My main concern would be when > using the ROOT=/some/dir emerge -Ke world that things would start breaking like > they do with openssl leaving orphan linking behind. Or that glsa updates would > still end up using the old/wrong versions when things should really be > upgraded. Binary packages should be mostly unaffected as the libs are injected between preinst and merge (just before collision-protect). So only packages created by quickpkg could possibly be affected by it, not sure if we have to find a way to avoid the preserved libs in those packages (should be quite easy once the registration of preserved libs is implemented) The main intention is to cover the time between the library update and until all dependent packages have been remerged, e.g. could add a message to emerge to instruct the user to rebuild the packages as long as any libs are preserved.
*** Bug 175548 has been marked as a duplicate of this bug. ***
I have seen an implementation that did not keep the old libraries in place but moved them to a separate directory like /lib.old or /usr/lib.old. It configured the linux linker to look into those directories - I think it used /etc/ld.so.conf. That way anything newly built linked against the new libraries but the old ones could be kept until everything was rebuilt. On other systems there are things like DYLD_FALLBACK_PATH.
(In reply to comment #14) > I have seen an implementation that did not keep the old libraries in place but > moved them to a separate directory like /lib.old or /usr/lib.old. It configured > the linux linker to look into those directories - I think it used > /etc/ld.so.conf. > > That way anything newly built linked against the new libraries but the old ones > could be kept until everything was rebuilt. Using a single path doesn't work well with multilib environments unfortunately. And eve with multiple directories we'd have to figure out the ABI of a lib to know where to store it which sucks. Also right now the implementation is generic, so it could be used for other means as well (e.g. python modules) if the required data would be available. That said the implementation is still incomplete as there is no API to query which libs have been preserved by this feature, which would be trivial to do when using a special dir like /usr/lib/old, but as said that doesn't work well with multilib.
(In reply to comment #15) > > Using a single path doesn't work well with multilib environments unfortunately. > And eve with multiple directories we'd have to figure out the ABI of a lib to > know where to store it which sucks. Also right now the implementation is > generic, so it could be used for other means as well (e.g. python modules) if > the required data would be available. > That said the implementation is still incomplete as there is no API to query > which libs have been preserved by this feature, which would be trivial to do > when using a special dir like /usr/lib/old, but as said that doesn't work well > with multilib. > Multilib? Like the platforms where both 32bit and 64bit libraries are installed? Or multiple versions of installed libraries? How does portage figure out where to put the library when it is installed? Whatever information it had back then is still present in the package that is being removed, isn't it? Anyway I do not think it can be a single directory. It is necessary to preserve the separation of /lib and /usr/lib for one. I am not sure why it would not work for python modules. In any case it would work only if the filename changed. Moving the file into a different directory would only provide the indication that it is old and applications that still use it should be updated. I think python can be configured to search multiple directories as well.
*** Bug 143467 has been marked as a duplicate of this bug. ***
(In reply to comment #16) > (In reply to comment #15) > > > > Using a single path doesn't work well with multilib environments unfortunately. > > And eve with multiple directories we'd have to figure out the ABI of a lib to > > know where to store it which sucks. Also right now the implementation is > > generic, so it could be used for other means as well (e.g. python modules) if > > the required data would be available. > > That said the implementation is still incomplete as there is no API to query > > which libs have been preserved by this feature, which would be trivial to do > > when using a special dir like /usr/lib/old, but as said that doesn't work well > > with multilib. > > > > Multilib? Like the platforms where both 32bit and 64bit libraries are > installed? Or multiple versions of installed libraries? The former. > How does portage figure > out where to put the library when it is installed? Whatever information it had > back then is still present in the package that is being removed, isn't it? It doesn't, the ebuild/profile determines that. There would be two options to figure it out: a) rely on path (lib32 vs lib64), which I'd consider unreliable, or b) use something like scanelf or readelf to determine the ABI, which is a specialized solution for ELF files, for other files we'd need another solution > I am not sure why it would not work for python modules. In any case it would > work only if the filename changed. Moving the file into a different directory > would only provide the indication that it is old and applications that still > use it should be updated. I think python can be configured to search multiple > directories as well. Sure it can, but why should we have to? The current solution is transparent to the rest of the system, a separate dir would require reconfiguration of several system components, and now that the registry is in place, doesn't provide much benefit. If you want to provide a patch for the separate-dir approach feel free to post it here though.
there really is no other option than using a solution like scanelf considering every object format has its own notion considering every system where this is a critical issue is based on ELF, relying on scanelf is not limiting in any way
I've just been pointed at this issue here after I came up with an idea fairly similar to comment 14. I think the whole thing has a lot of similarity to how we handle config files. In both cases some action by the admin is required, and in both cases you'd want to keep old files around to keep the system going. My idea would be to have some kind of LIB_PROTECT setting and whenever an ebuild would remove a matching file (maybe globbed to *.so) you'd move it from $DIR/$FILE to $DIR.old/$FILE. You can either patch the runtime linker or simply export environment variables to search these dirs as well. I don't fully comprehend why a path based approach isn't enough, but that's not my main concern. The important thing is that 1. Portage should protect all libs, the ebuilds don't have to care - just like ebuilds don't care about their config files 2. When there are old libs around, portage tells the user to run revdep-rebuild - just like portage tells him to run etc-update if some config file has an updated version 3. When revdep-rebuild checks linkage, it notes packages that use old libs - just the way etc-update notices that there are new config files around 4. When revdep-rebuild finds that an old lib is no longer needed, it gets removed - just like revdep-rebuild removes the updated config files after the user has made his choice This would solve two problems. One is the obvious problem that between the lib update and the completion of the revdep-rebuild, the system is in an unusable state. This keeps me from doing needed updates on a major multiuser system. The other is those annoying messages from thoughtful ebuilds that tell me that the old lib has been kept and that I should run revdep-rebuild on this lib. This is annoying for multiple reasons: 1. I have to notice and remember those messages (ELOG helps) 2. I have to remember to execute the commands once my primary emerge is done 3. I need to run revdep-rebuild for every library independendly 4. If I forget to delete the lib afterwards, it may be lying around forever 5. If I forget to run revdep-rebuild, there is nothing to remind me All those would be solved by the separate dir and revdep-rebuild approach. On IRC jakub was concerned that this would leave vulnerable libs around longer than needed if people simply ignore the request to run revdep-rebuild. On the other hand the fact that I know it will break things keeps me from updating vulnerable libs in the first place, which I consider more serious. If people do not do as they are told, we can't help it. That holds for config files, it should hold for old libs as well.
breaking people's systems in order to force them to run revdep-rebuild is not acceptable ... if they cant be bothered to rebuild their system, they get the pieces
(In reply to comment #20) > The important thing is that > 1. Portage should protect all libs, the ebuilds don't have to care > - just like ebuilds don't care about their config files > 2. When there are old libs around, portage tells the user to run revdep-rebuild > - just like portage tells him to run etc-update if some config file has an > updated version > 3. When revdep-rebuild checks linkage, it notes packages that use old libs > - just the way etc-update notices that there are new config files around > 4. When revdep-rebuild finds that an old lib is no longer needed, it gets > removed - just like revdep-rebuild removes the updated config files after > the user has made his choice 1 and 2 are already implemented in portage-2.2, 3 and 4 are the missing bits yet. @Michael/Paul: if you want to take a shot at implementing 3 and 4 in revdep-rebuild or the scanelf counterpart and need any information, let me know. @Zac: if we'd create a PackageSet based on the info from vartree.PreservedLibsRegistry, do you thing the (new) resolver would be able to get the order right, especially when "system" packages are affected?
This is supposed to be fixed in portage-2.2_pre5 or earlier.
