Given the --dynamic-deps=n default in portage-2.3.20, it would be friendly to have an option to automatically enable --changed-deps=y if it does not conflict with other options, and report a list of packages with changed deps (for purposes of filing bugs). My goal is to protect users from having a frustrating experience, by default. I imagine something like the --with-bdeps-auto option will work nicely, so people will get the --changed-deps=y behavior if they don't specify conflicting options such as --usepkgonly, and they can specify --changed-deps-auto=n in EMERGE_DEFAULT_OPTS if they don't want this behavior.
Wouldn't this expose users to a fair number of unnecessary rebuilds and false positive bugreports whenever developers decide that the change doesn't require propagation?
> report a list of packages with changed deps This is not a solution imho, as - it'll still be quite a bit more work for maintainers to do revision bumps even for subtle dependency adjustments - it'll not update the vdb, so users still needlessly have to rebuild affected ebuilds at some point in time How about the following: - store a hash of the ebuilds dependency list in the tree, as well as in vdb - after sync, perform this: - in case updated ebuild revisions match installed counterparts and the hashes of their dependencies do not match, test if an updated dependency would imply some change to be taken care of (a possible downgrade, an additional dependency), instead of a plain rebuild of just the ebuild in question - in the former case, a following world update cares for required rebuilds - if the latter is the case, the local integrity of the vdb should be unaffected, so simply copy the changed ebuild to vdb and log the information about this silent update
Practically all dependency changes have consequences, otherwise we wouldn't change them, therefore they are all worthy of propagation. Even DEPEND changes are worthy of propagation, since portage uses these by default to keep build time dependencies (and their dependencies) up to date (see bug 598444).
Patch posted for review: https://archives.gentoo.org/gentoo-portage-dev/message/46a6db48eb1af0a879036842516948cf https://github.com/gentoo/portage/pull/247
> Practically all dependency changes have consequences, otherwise we wouldn't change them, therefore they are all worthy of propagation. Regarding the consequences this is not quite true, since it's not just the dependency change you have to take into account, but the state of the vdb as well; Examples: - a dependency >=foo/bar-x.y gets changed to >=foo/bar-x.y+1, but in vdb is already pkg/foo/bar-x.y+N recorded - a dependency foo/bar gets changed to foo/bar:x, but in vdb is a pkg/foo/bar matching slot x recorded Such changes can and do happen all the time, because of issues known way later than creation or even stabilization of ebuilds. But there is no point to rebuild in these cases, just the vdb has to be updated regarding the ebuilds dependency requirements. > Even DEPEND changes are worthy of propagation, since portage uses these by default to keep build time dependencies (and their dependencies) up to date (see bug 598444). What I proposed is entirely orthogonal to bug 598444 and not against propagation of depend changes at all. Quite the opposite.
(In reply to Carlo from comment #5) > Such changes can and do happen all the time, because of issues known way > later than creation or even stabilization of ebuilds. But there is no point > to rebuild in these cases, just the vdb has to be updated regarding the > ebuilds dependency requirements. That's feasible and I'm not opposed to it. It's essentially a persistent version of the emerge --dynamic-deps option. However, the procedure needs to be standardized as a PMS extension in order to create a level playing field for package managers. That makes it beyond the scope of what I want to accomplish with this bug report. My goal for this bug report is to provide a safety net for users, such that they'll be automatically notified that the --changed-deps option is available when needed to solve problems involving stale dependencies. This goal is achievable without extending PMS, since all package managers are capable of triggering package rebuilds (for example they can all do the equivalent of `emerge -e @world` which gives similar results but rebuilds more packages than necessary).
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=5dbddff5525c6ba6d20b0ab33eae6de9ab3081eb commit 5dbddff5525c6ba6d20b0ab33eae6de9ab3081eb Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-01-28 14:03:44 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-01-30 04:24:03 +0000 emerge: add --changed-deps-report option (bug 645780) The --dynamic-deps=n default causes confusion for users that are accustomed to dynamic deps, therefore add a --changed-deps-report option that is enabled by default for deep updates (if --usepkgonly is not enabled). The report is entirely suppressed in the following cases in which the packages with changed dependencies are entirely harmless to the user: * --changed-deps or --dynamic-deps is enabled * none of the packages with changed deps are in the graph These cases suppress noise for the unaffected user, even though some of the changed dependencies might be worthy of revision bumps. The --quiet option suppresses the NOTE section of the report, but the HINT section is still displayed since it might help users resolve problems that are solved by --changed-deps. Example output is as follows: !!! Detected ebuild dependency change(s) without revision bump: net-misc/openssh-7.5_p1-r3::gentoo sys-fs/udisks-2.7.5::gentoo NOTE: Refer to the following page for more information about dependency change(s) without revision bump: https://wiki.gentoo.org/wiki/Project:Portage/Changed_Deps In order to suppress reports about dependency changes, add --changed-deps-report=n to the EMERGE_DEFAULT_OPTS variable in '/etc/portage/make.conf'. HINT: In order to avoid problems involving changed dependencies, use the --changed-deps option to automatically trigger rebuilds when changed dependencies are detected. Refer to the emerge man page for more information about this option. Bug: https://bugs.gentoo.org/645780 man/emerge.1 | 11 ++++ pym/_emerge/create_depgraph_params.py | 7 +++ pym/_emerge/depgraph.py | 99 +++++++++++++++++++++++++++++++++-- pym/_emerge/main.py | 13 +++++ 4 files changed, 127 insertions(+), 3 deletions(-)}
> That's feasible and I'm not opposed to it. It's essentially a persistent version of the emerge --dynamic-deps option. However, the procedure needs to be standardized as a PMS extension in order to create a level playing field for package managers. That makes it beyond the scope of what I want to accomplish with this bug report. The PMS is second here. The problem is the policy change. If maintainers are really willing to perform the daunting workload to do revision bumps for all and everything, then it's pointless to continue this. And please do not underestimate the extra maintenance work, this policy change implies. I'd even go so far, that this policy change is the perfect idea to loose even more users and maintainers than in the last years. The real issue isn't the speed of Portage or the existing oddities regarding dependency calculation, but the lack of quality and currentness of the gentoo main repository.
(In reply to Carlo from comment #8) > The PMS is second here. The problem is the policy change. If maintainers are > really willing to perform the daunting workload to do revision bumps for all > and everything, then it's pointless to continue this. And please do not > underestimate the extra maintenance work, this policy change implies. I'd > even go so far, that this policy change is the perfect idea to loose even > more users and maintainers than in the last years. I don't know how much things are going to change. It's possible we will find coping mechanisms in order to avoid having to do a lot of rebuilds. For example, we might use virtuals to isolate dependencies that are likely to change, and propagate dependency changes via virtual revision bumps which are more lightweight.
I have a lot of respect for your work on Portage, Zac, but "virtuals to isolate dependencies" or what else won't lower the maintenance workload and you'd probably end with inflationary virtual/ohweforgottoslotlibfoo-r3246; Sounds like a bad joke and misuse of the concept of virtuals to me. I'm sorry, but this seems terrible to me.
I saw the new behaviour for the first time yesterday as a result of my usual 'emerge -uvND world' update procedure and, having read through both the various descriptions in the emerge man page and the thread of the comments in this bug, I am left with a number of questions/reservations: Initially, I got a list of some 40 ebuilds as a result of the changed dependencies ebuilds report so I duly rebuilt the majority of these (excluding two instances of webkit-gtk and also llvm because of the compile-time implications). Afterwards I wondered just what it as that I had achieved: so far as I could tell I had been experiencing neither build- nor run-time problems on account of the now-rebuilt software and there were certainly no preserved-libs requirements outstanding. So, in effect, what are the criteria which determine 'when changed dependencies are detected'? Given that the combined build time for two instances of webkit-gtk and llvm on my system amounts to some 14 hours I wouldn't want to have to do these if the underlying changes in the dependency ebuilds were entirely trivial, e.g. a change of application home page. (It might be useful if the changed dependency ebuilds were enumerated beneath the corresponding entry in the report.) Also, I read that although there are guidelines for devs as to whether an ebuild is -r[n] bumped or not there is also discretion - but presumably all changed dependency ebuilds will be flagged. I do wonder if what's being implemented here isn't analagous to the preserved-rebuild aspect of portage and therefore whether the merits of an @changed-dependency-ebuild set mightn't be considered as an alternative approach? Also, how visible this ought to be by default to the non-dev users? Thanks.
(In reply to Carlo from comment #10) > Sounds like a bad joke and misuse of the concept of virtuals to me. I'm talking about using virtuals as a layer of dependency indirection, which is well within the bound of their intended use. I'm not suggesting that virtuals are the solution to every one of these --changed-deps issues, but they might be useful for some of them. (In reply to Adrian.Bassett from comment #11) > Initially, I got a list of some 40 ebuilds as a result of the changed > dependencies ebuilds report so I duly rebuilt the majority of these > (excluding two instances of webkit-gtk and also llvm because of the > compile-time implications). Afterwards I wondered just what it as that I > had achieved: so far as I could tell I had been experiencing neither build- > nor run-time problems on account of the now-rebuilt software and there were > certainly no preserved-libs requirements outstanding. The problems involve explicit emerge dependency calculation failures or more subtle dependency calculation issues that can go unnoticed. There are at least 4 classes of problems that I've identified and documented on this page: https://wiki.gentoo.org/wiki/Project:Portage/Changed_Deps Bug 645550 tracks issues that have been reported since sys-apps/portage-2.3.20 uses --dynamic-deps=n by default. > So, in effect, what are the criteria which determine 'when changed > dependencies are detected'? It means that *DEPEND variable have changed, either directly in the ebuild, or in eclasses that the ebuild inherits. > Given that the combined build time for two > instances of webkit-gtk and llvm on my system amounts to some 14 hours I > wouldn't want to have to do these if the underlying changes in the > dependency ebuilds were entirely trivial, e.g. a change of application home > page. > (It might be useful if the changed dependency ebuilds were enumerated > beneath the corresponding entry in the report.) Yes, that would a very nice feature to have. > Also, I read that although > there are guidelines for devs as to whether an ebuild is -r[n] bumped or not > there is also discretion - but presumably all changed dependency ebuilds > will be flagged. Yes, all changed dependencies are flagged because there are many types of problems that stale dependencies can cause, and they are not necessarily easy to detect. For example, they can trigger missed package updates, which can easily go unnoticed. > I do wonder if what's being implemented here isn't analagous to the > preserved-rebuild aspect of portage and therefore whether the merits of an > @changed-dependency-ebuild set mightn't be considered as an alternative > approach? We've already got the --changed-deps option for that, which is hinted by emerge as follows: HINT: In order to avoid problems involving changed dependencies, use the --changed-deps option to automatically trigger rebuilds when changed dependencies are detected. Refer to the emerge man page for more information about this option. > Also, how visible this ought to be by default to the non-dev users? If you want to ignore the changed dependencies then you can set EMERGE_DEFAULT_OPTS="--changed-deps-report=n" in /etc/portage/make.conf (this is mentioned in the emerge output). The purpose of the report is to help users cope with problems caused by changed dependencies, since they've been sheltered from these problems in the past.
(In reply to Zac Medico from comment #12) > (In reply to Adrian.Bassett from comment #11) Thanks for the detailed clarifications.
(In reply to Zac Medico from comment #12) > > So, in effect, what are the criteria which determine 'when changed > > dependencies are detected'? > > It means that *DEPEND variable have changed, either directly in the ebuild, > or in eclasses that the ebuild inherits. So, for example, media-sound/flacon makes an appearance in the report as a dependency ebuild which has changed without a revision bump: ... media-sound/flacon-4.0.0::gentoo ... Looking at https://gitweb.gentoo.org/repo/gentoo.git/commit/media-sound/flacon?id=f619079436e1d6f667e092a5fd60de08600365c2 it's obvious what the change is: + dev-qt/qtconcurrent:5 but, given that I already have this package:slot on my system since Dec 8 at the latest: $ ls -d /var/db/pkg/*/qtconcurrent* drwxr-xr-x 2 root root 4096 Dec 8 22:51 /var/db/pkg/dev-qt/qtconcurrent-5.9.3 $ cat /var/db/pkg/dev-qt/qtconcurrent-5.9.3/SLOT 5/5.9 then the presence of media-sound/flacon in the report is a false positive whose re-building will have no substantive impact on the system? Is it feasible/planned to further refine portage to eliminate such entries in the report? Thanks.
(In reply to Adrian.Bassett from comment #14) > Is it feasible/planned to further refine portage to eliminate such entries > in the report? We could spend a lot of time doing that kind of thing, and the time would be better spent on a tool that merged the dependencies so that a rebuild is not necessary.
Nice to have this option now, but it isn't really useful for me: The list with reported ebuilds contains 293 packages. Creating one bug for each package will take 5-10 hours (1-2 minutes for each bug) - nothing I can and will do :/ I expect an auto-report feature isn't something that can be done. I already attached a list I created manually to another bug (all affected from the same commit) and was told to create one single report for each package. In this case wouldn't it be possible to simply list every affected package in one bug and ask devs to regularly have a look? If you have a look at bugs.gentoo.org/645550 you can see that since the introduction of this feature no further blocks were reported - probably because it's just too time consuming :/
(In reply to Franz Fellner from comment #16) > Nice to have this option now, but it isn't really useful for me: The list > with reported ebuilds contains 293 packages. Creating one bug for each > package will take 5-10 hours (1-2 minutes for each bug) - nothing I can and > will do :/ I expect an auto-report feature isn't something that can be done. > I already attached a list I created manually to another bug (all affected > from the same commit) and was told to create one single report for each > package. The thing is, there's a lot of pushback from people who don't have the spare cpu cycles to rebuild all this stuff. I think the only thing that's going to make everyone happy will be a tool to apply the dependency update in-place without a rebuild. > In this case wouldn't it be possible to simply list every affected package > in one bug and ask devs to regularly have a look? If you have a look at > bugs.gentoo.org/645550 you can see that since the introduction of this > feature no further blocks were reported - probably because it's just too > time consuming :/ It's not just that, it's the strong aversion to rebuilds. I suspect a lot of people have put --dynamic-deps=y in EMERGE_DEFAULT_OPTS by now.
Zac, I'm going to ask you to revert this report to default to off. It's not helpful if Portage follows useful output with a spew of useless messages that sometimes can kill short scrollback buffer. If any user actually cared to start reporting bugs about those packages, he'd get shot by bug-wranglers for spamming the Bugzilla. If any user made one big bug report, he'd be shot for making a useless bug that can't be reasonably assigned. Even if a report for this actually hit the correct maintainer, half of the time it will be closed as INVALID/WORKSFORME or some other indication that the change was intentional and harmless. Then there will be another report. Then another. In the end, there will be a lot of angry developers and hopefully some of them will not end up misdirecting their anger at users. That said, you may suggest to run the report *if* Portage detects something wrong with the depgraph. Presuming anyone has a clue how to detect that in a way that doesn't trigger all the false positives.
Yes, I want to disable it by default. Here's the patch: https://archives.gentoo.org/gentoo-portage-dev/message/88b85783f11fa271835b1449d953804c https://github.com/gentoo/portage/pull/250
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=59d58d4c3074321d5ae7eb6c1feb2816b5f27775 commit 59d58d4c3074321d5ae7eb6c1feb2816b5f27775 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-02-03 02:39:35 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-02-03 03:09:43 +0000 emerge: disable --changed-deps-report by default (bug 645780) This option is too noisy to enable by default, since it reports hundreds of packages for most users. Suggested-by: Michał Górny <mgorny@gentoo.org> Bug: https://bugs.gentoo.org/645780 man/emerge.1 | 7 +------ pym/_emerge/create_depgraph_params.py | 6 ++---- 2 files changed, 3 insertions(+), 10 deletions(-)}
> The thing is, there's a lot of pushback from people who don't have the spare cpu cycles to rebuild all this stuff. This might be true, but it's also too simplistic. From my POV the fallout - build failures to investigate / compatibility with entirely private overlays - is in general higher than necessary. While Gentoo is a versatile tool, most of the time you need to get work done, instead loosing time on maintenance. And I'm quite sure some of these rebuilds will point out even more issues to be investigated. It's more about "man cycles" than cpu cycles.
The --changed-deps-report option is no longer enabled by default, because it's too noisy. Also, the --dynamic-deps=n default has been reverted (bug 646458). If we are ever make --dynamic-deps=n default again, we need automated monitoring of all dependency changes that will eventually reach users. My plan is to add an egencache option to generate a QA report for changed dependencies, see bug 646622.
