Bug 476350 – emerge --keep-going should be able to do everything not blocked.

• Product: Portage Development
• Status: CONFIRMED ---

[Description Robert White 2013-07-10 06:06:18 UTC]

When a package is blocked and the user has supplied --keep-going, and one or more packages not related to the blockage have been selected, the un blocked package shoudl still be built.

Example: packages A, B, C, D and E are due for update.

B needs C
C is blocked for some reason (so neither B nor C are safe to install/update).

A, D and E should still be processed if --keep-going has been specified.

If C is blocked because it's update conflicts with E, then A and D should still be processed.

Basically if something comes up blocked a list should be made of all the packages, then every package in the the blocking set is removed. Then the dependent packages who's install-or-update are dependent on the blocked update are removed, etc. A package that can be built using the existing install is not removed. Lather-rinse-repeat. If that itterative process finishes with a non-empty list then that is a list of packages that are "innocent" of the conflicts in the system and they can be safely updated.

Real World Example:

I am running ~amd64. app-emulation/emul-linux-x86-medialibs and app-emulation/emul-linux-x86-soundlibs are blocked right now with the the ABI_X86="64 32" make flag. This is bleeding edge and likely to snag.

None of that blockage should be holding back xterm, dhcp, dosfstools or ntp (etc, etc, etc).

Reproducible: Always

[Comment 1 Thomas Capricelli 2018-11-20 02:54:07 UTC]

It's getting worse and worse every week, probably because more and more packages fail now in the main tree. The result is that we cant update system at all, because there's always one package failing early and emerge wont keep going after this. This is getting critical.

[Comment 2 Robert White 2020-10-26 14:08:17 UTC]

So I've noticed that if I have --jobs set to 3 or more and I don't use --keep-going I can often just repeatedly issue "emerge --resume" _without_ --skipfirst and end up with many more successfully built packages.

The implication of that is that lots of build stalls are missing dependencies or other forms of phantom state.

(And honestly if I knew enough Python and the portage code base This would have just been a patch submission.)

So imagine if the build order were not fully pre-calculated as is apparently done now, but was instead managed as three lists:

list 1: Ready Builds :: Any ebuild task that has no unbuilt dependencies is on this list.

list 2: Dependent Builds :: Any ebuild task that needs orne or more other ebuilds to be done before this task can start is on this list.

list 3: Failed Builds :: If any ebuild fails the task is moved from list1 to this list.

And imagine that each structure representing a selected package atom (hereafter just referred to as an ebuild) has a private dependency list of ebuilds that should be built before it is built itself and a counter that starts life at zero.

Execution Model:

1) The dependency tree is calculated just as it is now. Each ebuild data structure is given its own short list of the other ebuilds that it is dependent on, if any.

2) The dependency list is transcribed into lists one and two; the ones with no dependencies go on list 1; the rest go on list two.

3) The item count of each list is noted.

4) The builder dispatcher runs down list 1 from first to last, keeping --jobs=X ebuilds active.

5a) If an ebuild successfully merges the item is removed from list 1 and then the builder dispatcher goes down list 2 removing the successful ebuild from the dependency list any items it finds and increments that build's counter. If that leaves that item's dependency list empty the item is removed from list 2 and appended to list one.

5b) If an ebuild fails for any reason it is removed from list one and appended to list three.

When list 1 is exhausted the contents of list three followed by list two forms the resume list, but... If --keep-going is specified...

6) If the size of any list has changed from the memorized sizes the contents of list 3 is moved to list one and the controller process loops back to step three.

So basically at this point no further progress is possible, unless...

7) If (a proposed) --evolve option is used the controller task goes down list 3.

7a) If there is one or more ebuild version available that is greater than the installed version, and who's dependencies are all met by installed builds then the ebuild is revised to attempt the latest of those possible alternatives.

7a1) If the attempted alternative would not satisfy an ebuild from list 2, that ebuild is moved to the end of list 3 immediately

7a2) The controller invokes a normal pass on list 1 immediately.

7b) If there are no possible alternatives the ebuild is removed from list 3.

7b1) The controller scans list 2 and removes the ebuild from the dependency lists of any ebuilds it finds there. If that leaves the ebuild's dependency list empty; and the ebuild's counter is non-zero or the ebuild's version is greater than the installed version or there is no currently installed version...

7b1i) If the ebuild is still valid for the existing installed packages then move it to list 1 and process list 1 immediately.

7b1ii) Otherwise move the ebuild to the end of list 3.

---

Once the evolver reaches the end of list 3 the system is as fully upgraded as practically possibe (with respect to the packages selected by the command line).

This logic can also map directly over dependency loops that can be solved with variant USE flag variations. So if you have an A<-B<-C<-A loop that coulld probably be resolved if A was built with USE=-pam or USE=-doc or whatever, the Then the starting list could have e.g. A-doc with a dependency list that doesn't include C, a single ebuild for B that only depends on A-doc, and also A+doc ebuild that does depend on C.