If die is triggered in pkg_pretend() for one of many packages to be merged, portage --keep-going does not attempt to build the other packages. eg. emerge --keep-going -uDvN libreoffice baz bar Running pre-merge checks for app-office/libreoffice-3.5.0.3 * Checking for at least 512 mebibytes RAM ... * Checking for at least 6 gibibytes disk space at "/var/tmp/portage/app-office/libreoffice-3.5.0.3/temp" ... [ !! ] * There is NOT at least 6 gibibytes disk space at "/var/tmp/portage/app-office/libreoffice-3.5.0.3/temp" .... $ # emerge has returned without trying to merge baz and bar! More importantly, this impedes an emerge -uDNv --keep-going world. (Obviously, --skipfirst also doesn't cause the non-failing packages to be built, as libreoffice may be the first package to fail pre-merge tests, but isn't the first in the list to be merged. Figuring out appropriate behaviour might be subject to debate) Reproducible: Always
Created attachment 302179 [details] emerge --info
I should think this was a design choice, since --keep-going is intended to keep going when ebuild phases _after_ pkg_pretend() fail.
Created attachment 302181 [details] command-line & stdout
Another aspect of this that (perhaps) should be considered is that the system state may have changed between the pre-merge checks and the package triggering them being built; eg. emerge massive-binary-package libreoffice > pre-merge : libreoffice checks disk space > merge : massive-binary-package uses up disk space > merge : libreoffice starts a build, assuming that disk space is available (fails)
I think that the --exclude functionality should be called in these cases; so: $ emerge -uDavN --keep-going world # with libreoffice and others in world <libreoffice pre-merge fails> -> The equivalent to `emerge -uDvN --keep-going --exclude libreoffice world` is called
*** Bug 465264 has been marked as a duplicate of this bug. ***
*** Bug 518694 has been marked as a duplicate of this bug. ***
I'm sorry but this is the desired behavior and we're not supposed to change it without broader discussion. The goal here is to fail as soon as possible. Since pkg_pretend() is executed immediately after confirming the dependency tree, sysadmin can reasonably wait for it to finish or fail, and can fix the reasons for failures before starting the real update. The requested behavior of --keep-going would go against that particular goal. If emerge were to drop packages because of pkg_pretend() failures, the sysadmin would likely have to actually wait for partial update to finish before fixing his system. It is also possible that the premature partial update would result in necessity of extra rebuilds afterwards. That said, emerge tries to do its best for you. In particular: 1. all pkg_pretend() phases are run even if one of them fails. This way, you already know of all the issues and likely can fix them in one configuration update. 2. emerge --resume --exclude ... can be used to restart the merge omitting unwanted packages. (In reply to James Broadhead from comment #4) > Another aspect of this that (perhaps) should be considered is that the > system state may have changed between the pre-merge checks and the package > triggering them being built; > > eg. emerge massive-binary-package libreoffice > > pre-merge : libreoffice checks disk space > > merge : massive-binary-package uses up disk space > > merge : libreoffice starts a build, assuming that disk space is available (fails) That's why pkg_pretend() checks come with equivalent pkg_setup() checks. The latter are performed directly before merging the package. Since they are done during the non-interactive part of emerge, --keep-going is applied normally to those checks.
That said, I don't think we'd have a problem accepting a patch that would add an extra option to enable the desired behavior.
(In reply to Michał Górny from comment #8) > I'm sorry but this is the desired behavior and we're not supposed to change > it without broader discussion. Really? Some of the behaviour already exists if a package fails to compile. This is really no different. Be it failure to compile or failure to pass pkg__pretend, one package and/or its dependencies should not prevent other non-dependent packages from merging. Otherwise why bother to keep going? > The goal here is to fail as soon as possible. Since pkg_pretend() is > executed immediately after confirming the dependency tree, sysadmin can > reasonably wait for it to finish or fail, and can fix the reasons for > failures before starting the real update. Why should emerging say hundreds of package fail because there is not enough say temp space for Firefox? Its not like this happens in seconds and can take minutes. > The requested behavior of --keep-going would go against that particular > goal. If emerge were to drop packages because of pkg_pretend() failures, the > sysadmin would likely have to actually wait for partial update to finish > before fixing his system. It is also possible that the premature partial > update would result in necessity of extra rebuilds afterwards. That is no different than when a package fails to compile. The if a package failes to compile that has dependencies, its dependencies are removed, and it continues on with other packages. So how is this any different? In fact failing during compile would go against what you describe as the sysadmin has to wait till the compile fails to take action. > That said, emerge tries to do its best for you. In particular: > > 1. all pkg_pretend() phases are run even if one of them fails. This way, you > already know of all the issues and likely can fix them in one configuration > update. A package no others depend on failing pkg_pretend should not block others. That package, and/or its depenencies should be removed. Just like if the package failed to compile. The same thing should take place either way. Instead your describing inconsistent beviour. Keep going does one thing with pkg_pretend, and antoher if a compile fails. It should do the same thing in both cases, consistent behaviour of keep going. Your saying it should fail right away due to 1 package failing pkg_pretend. But then again if that same package fails to compile, the merge will continue on less any dependencies of that package. That makes absolutely no sense. Please re-read your argument with that in mind. I think you will come to the same conclusion. > 2. emerge --resume --exclude ... can be used to restart the merge omitting > unwanted packages. Or the sysadmin can address why the pacakge failed pkg_pretend. Which I believe is what you were sayings was the intended behaviour. Though again it contradicts behavior of a pacakge failing to compile. Be it a pkg_pretend failing, or package failing to compile, --keep-going should have consistent behavior and keep going in either case. Right now it only does in the latter, and your saying it was designed that way for the former. Despite it not making sense... One will lose way more time with a package failing to compile, than with a package failing pkg_pretend. Either way it should continue on and merge what it can, regardless of why it failed. Right now it does with compiling, but not with pkg_pretend. If designed this way, then --keep-going was designed to act differently at difference stages of the merge process, and that makes no sense what so ever. It should do the same thing regardless of what is failing.
This case is very similar to bug 295229, and I think that adding an option as mentioned in comment #9 is a good idea.
*** Bug 545972 has been marked as a duplicate of this bug. ***
(In reply to William L. Thomson Jr. from comment #10) I concur in full extent.
I have started work on this here: https://github.com/gentoo/portage/compare/master...jamesbroadhead:keep_going present status: incomplete, non-functional
*** Bug 581088 has been marked as a duplicate of this bug. ***
For the time being I am getting around this issue by setting I_KNOW_WHAT_I_AM_DOING in the env. You can do this globally in make.conf or package specific in env/package.env per https://wiki.gentoo.org/wiki//etc/portage/package.env and https://wiki.gentoo.org/wiki/Knowledge_Base:Overriding_environment_variables_per_package If you set this globally it seems to return to normal behavior. That is pre-checks run, they can fail, merge continues. If the package does fail to build, portage will remove the package and deps if keep-going is set and used. Problem resolved... For now I am doing per package, but might move to global.
*** Bug 605088 has been marked as a duplicate of this bug. ***
*** Bug 607874 has been marked as a duplicate of this bug. ***
The fix for bug 710432 makes the pkg_pretend phases run inside of the --keep-going retry loop, which enables --keep-going to recover from pkg_pretend failures. https://archives.gentoo.org/gentoo-portage-dev/message/b480a9bc7b4a1d17415256eb31bd405e https://github.com/gentoo/portage/pull/625
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=c7fa3f1eb1ce1ebc0d1219dacba555e1a29d5f22 commit c7fa3f1eb1ce1ebc0d1219dacba555e1a29d5f22 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-09-20 00:32:57 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-09-21 05:17:24 +0000 emerge: enable parallel-fetch during pkg_pretend (bug 710432) Execute pkg_pretend phases in a coroutine while parallel-fetch is running concurrently. When it's time to execute the pkg_pretend phase for a remote binary package, use a Scheduler _get_prefetcher method to get a running prefetcher if available, and otherwise start a new fetcher. Since pkg_pretend phases now run inside of the --keep-going retry loop, --keep-going is now able to recover from pkg_pretend failures, which fixes bug 404157. Bug: https://bugs.gentoo.org/404157 Bug: https://bugs.gentoo.org/710432 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/Scheduler.py | 142 +++++++++++++++++++++++++++++++++-------------- 1 file changed, 99 insertions(+), 43 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=862266358f01efa90c12d182d1f85a3014b80439 commit 862266358f01efa90c12d182d1f85a3014b80439 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-09-21 05:35:06 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-09-21 05:59:31 +0000 sys-apps/portage: Bump to version 3.0.8 #404157 emerge: --keep-going support pkg_pretend failures #644246 Add mercurial sync support #710432 emerge: Enable parallel-fetch during pkg_pretend #743115 emerge: Fix slot conflict backtracking to group similar missed updates into a single backtracking try #743631 emerge: Fix slot conflict backtracking to prefer choices that satisfy all dependencies Bug: https://bugs.gentoo.org/739718 Bug: https://bugs.gentoo.org/404157 Bug: https://bugs.gentoo.org/644246 Bug: https://bugs.gentoo.org/710432 Bug: https://bugs.gentoo.org/743115 Bug: https://bugs.gentoo.org/743631 Package-Manager: Portage-3.0.8, Repoman-3.0.1 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.8.ebuild | 267 ++++++++++++++++++++++++++++++++++ 2 files changed, 268 insertions(+)
