I had PYTHON_TARGETS="python2_7 python3_2" PYTHON_SINGLE_TARGET="python2_7" I emerge python 3.3 and changed that to PYTHON_TARGETS="python2_7 python3_3" PYTHON_SINGLE_TARGET="python2_7" (3_2 ==> 3_3) I unmerged python 3.2, re-emerged the @preserved's, and ran revdep-rebuild and python-updater. In less than 20 minutes, after re-emerging 12 small to medium-sized packages, I had a working, clean and consistent system: * All shared lib references to python 3.2 were gone. * All python references and modules of 3.2 were gone. * None of the packages installed on my system had any DEPEND's on python 3.2 in portage's package dependency tree. However, a emerge ... --deep --newuse system world re-emerged another 16 packages just because their PYTHON_TARGETS has changed, among them libreoffice, boost, wireshark, ... taking more than 5 hours in total! The installed versions of those packages had absolutely *zero* dependencies on python 3.2, and they also didn't have any DEPEND's on it in portage, so re-emerging them was totally unnecessary. Needing python to build or install a package (but not to run it), or having python just potentially used by some optional part of a package (which perhaps aren't build anyway because of some USE flag settings), is absolutely no reason to make the package rebuild-depend on the python version in PYTHON_TARGETS. So please take PYTHON_TARGETS out of the --newuse check (or move it to a separate --newpython), the current behaviour doesn't make sense and is extremely annoying. We have revdep-rebuild and python-updater, that's enough to handle python updates.
Forgot to mention: The main reason for the unnecessary rebuilds is not that the packages needed python just for build or install, or just for optional components, but the main reason is that my primary python is 2.7, so most of the packages depend on 2.7 *only* and aren't affected by the change 3.2 ==> 3.3. So as long as their "primary" python doesn't change, they should not be rebuilt when PYTHON_TARGETS is changed.
python-updater will be gone once we migrate everything to PYTHON_TARGETS -- it will be completely unnecessary at that point. I agree that --newuse causes some unnecessary rebuilds. I usually recommend --changed-use instead of --newuse, but I don't think that would help here. For python-r1 based ebuilds, we need --newuse to consider python_targets, so some special exception would not be helpful here. Maybe we could remove the python_targets use flags from python-single-r1 ebuilds and base the deps entirely on python_single_target. That would probably require adjusting the dependencies on all packages currently using the eclass.
We sure can do it but I'm not sure if it's worth the effort. It will effectively cause every python-single-r1 to rely on whether it's dep is python-r1 or python-single-r1. That is, converting a dep to support multiple impls will require adjusting all rev-deps...
Yeah, I'm putting this in the "not worth it" bucket as well. Given the rate of python releases, this should really only be an issue once every couple of years.
(In reply to Mike Gilbert from comment #2) > Maybe we could remove the python_targets use flags from python-single-r1 > ebuilds and base the deps entirely on python_single_target. That would > probably require adjusting the dependencies on all packages currently using > the eclass. We have 2 cases of unnecessary rebuilds: 1.) Change of python_targets, but not python_single-target, i.e. the the installed ebuild uses a version of python not affected by the change. 2.) Ebuilds not having any runtime dependencies on the python version at all, either because python is needed only in some optional parts not getting built, or because python is needed only at build or install time, or because they install only python source code, no libs. Your suggestion only helps for 1.) (which is the majority of cases). But why isn't it possible to check for rebuilds by looking at actual runtime dependencies, as it was done by python-updater, and as it is still done for shared-lib dependencies by revdep-rebuild and @preserved ? This looks more natural to me, and it would reduce rebuilds to the necessary minimum. For shared-lib dependencies, it would be absolutely mad to rebuild based on some static version information or portage package dependencies, independent of actual linking dependencies, so why is python basing its rebuilds on static information? For example, libreoffice, which is the most annoying rebuild (taking several hours), as far as I know neither links against python nor installs any precompiled python libs, so why rebuild it?
app-office/libreoffice has deps which state that it has runtime dep on Python (Python's in COMMON_DEPEND). If that's not the case, that's a bug in the ebuild which needs to be fixed. As for deps with USE=-python, that's indeed a problematic case but I don't think we have a way of handling this. Maybe we would need an EAPI extension, either something new or some special form of REQUIRED_USE to suggest whether rebuilds are required or not. In the more general form, this could be implemented as some kind of runtime-switchable USE flags (IUSE_RUNTIME) with conditionals.
(In reply to Klaus Kusche from comment #5) > But why isn't it possible to check for rebuilds by looking at actual runtime > dependencies, as it was done by python-updater, and as it is still done > for shared-lib dependencies by revdep-rebuild and @preserved ? > python-updater is a workaround that was implemented because we were not exporting enough metadata to the package manager for it to know when a rebuild is (possibly) needed. From my perspective, portage should handle this, and the user should not need to manually rebuild things using an external tool. We (the Gentoo developers) are also working on eliminating revdep-rebuild as well by implementing slot-operator dependencies. This will also eliminate the need to call emerge @preserved-rebuild manually. The tradeoff in both cases is that sometimes packages get rebuild unnecessarily. I'm ok with that.
(In reply to Michał Górny from comment #6) > app-office/libreoffice has deps which state that it has runtime dep on > Python (Python's in COMMON_DEPEND). If that's not the case, that's a bug in > the ebuild which needs to be fixed. Sorry, I was wrong on that one. I assumed that libreoffice only installs serveral python source files and calls python as a command (so it would need *some* python installed, but would not be bound to a specific version). However, I just checked carefully, it also links against libpython (indirectly, so I missed it first), hence it needs to be rebuilt. However, this would be handled by revdep-rebuild or @preserved anyway, so I think there is no need to double-handle it based on python_target. Any python rebuild mechanism should perhaps only care for the cases not detected by revdep-rebuild or @preserved, e.g. for packages installing python modules into /usr/lib/python*/site-packages . At least, please change the emerge output in each python ebuild to clearly state how to upgrade correctly (it currently still only mentions eselect + python-updater, not PYTHON_TARGETS + emerge --newuse). I think it should be, in that order: * Change PYTHON_TARGETS and/or PYTHON_SINGLE_TARGET * eselect python (still used / needed ?) * emerge --newuse * emerge --depclean (old version should be removed, otherwise something's wrong!) * revdep-rebuild or emerge @preserved (only to make sure it worked) > As for deps with USE=-python, that's indeed a problematic case but I don't > think we have a way of handling this. Maybe we would need an EAPI extension, > either something new or some special form of REQUIRED_USE to suggest whether > rebuilds are required or not. USE=-python is only the most obvious form. There could be arbitrary USE flags controlling optional python-using code.
(In reply to Mike Gilbert from comment #7) > (In reply to Klaus Kusche from comment #5) > > But why isn't it possible to check for rebuilds by looking at actual runtime > > dependencies, as it was done by python-updater, and as it is still done > > for shared-lib dependencies by revdep-rebuild and @preserved ? > > > > python-updater is a workaround that was implemented because we were not > exporting enough metadata to the package manager for it to know when a > rebuild is (possibly) needed. From my perspective, portage should handle > this, and the user should not need to manually rebuild things using an > external tool. > > We (the Gentoo developers) are also working on eliminating revdep-rebuild as > well by implementing slot-operator dependencies. This will also eliminate > the need to call emerge @preserved-rebuild manually. > > The tradeoff in both cases is that sometimes packages get rebuild > unnecessarily. I'm ok with that. Basically, I appreciate this idea, as long as the number of additional rebuilds is kept minimal. However, please * keep it as a *separate*, *manual* step, which can be invoked later or replaced by manual emerges, or sliced into several separate emerges, * make sure that one can still check what gets rebuilt *why* don't keep rebuild dependencies hidden behind the scenes (I was already annoyed a lot by the little red "r" in emerge, not telling me at all *why*): - for newuse, I want to know the flags causing rebuilds - for lib rebuilds, I want to know the dependency! (at least, --verbose should display rebuild dependencies!) Reasons: - Sometimes, I don't have the time to wait for hours of rebuilds (I'm on a notebook, and I have to turn it off and take it with me), or not a network connection good enough for huge downloads, or not the CPU power to spare while giving a course, or whatever, but want to have some emerge update immediately (e.g. for security). For example, libpng including rebuilds would keep my notebook busy for about one day, and my notebook never ever runs that long continuously - I *have* to split that into separate parts. - Some rebuilds (e.g. libreoffice) need manual preparations (make space on my ramdisk, use --jobs=1 instead of 4, ...). - Sometimes, at world updates I even mask specific packages (like poppler, libpng, libjpeg, ...) or downgrade them again afterwards, or just reinstall the missing .so files from a backup, to delay big rebuilds to the next weekend. - Sometimes, I even decide that I can live with a broken package (e.g. libreoffice) for some days until I find time for a rebuild. - I'm patching some packages during emerge, so I want to be able to emerge them separately, and *not* having them rebuilt automatically together with all other packages. - If a rebuild fails, the updated library causing the rebuild is most likely the culprit. It would be nice to know *which* library that was...
(In reply to Klaus Kusche from comment #9) You are asking for things that are way outside the scope if this single bug report. The gentoo-dev mailing list would be a better forum to voice your concerns.
(In reply to Klaus Kusche from comment #9) > Reasons: > - Sometimes, I don't have the time to wait for hours of rebuilds > (I'm on a notebook, and I have to turn it off and take it with me), > or not a network connection good enough for huge downloads, > or not the CPU power to spare while giving a course, or whatever, > but want to have some emerge update immediately (e.g. for security). > For example, libpng including rebuilds would keep my notebook busy > for about one day, and my notebook never ever runs that long > continuously - I *have* to split that into separate parts. > - Some rebuilds (e.g. libreoffice) need manual preparations > (make space on my ramdisk, use --jobs=1 instead of 4, ...). FYI: you can FEATURES=buildpkg http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&chap=3#doc_chap4 and build the packages on big iron and then just install from built binary packages. > - I'm patching some packages during emerge, > so I want to be able to emerge them separately, > and *not* having them rebuilt automatically together with all other FYI: command ebuild might help you with that.
I'm sorry but there's no way we can fix this right now. I had some hopes wrt GLEP 73 but it was removed from EAPI 7 and Portage team did not implement it so far.
