Bug 247106 – --jobs with --resume re-ordering making it hard to hit with --skipfirst

• Product: Portage Development
• Status: CONFIRMED ---

[Description Steen Eugen "Miravlix" Poulsen 2008-11-16 18:10:27 UTC]

When you try to continue with --resume, it changes the sort order, this can make it really hard to hit the correct package with --skipfirst, I often have to do 3-4+ --resume until it finally gives me the package i want to skip as number one.

Two emerge --resume -p always result in the same sort order, it's only if you have a active emerge that saves it resume info that you can't trust the new session to start with the last file of the previous session.

Having a packet that breaks you need to skip and you end up doing 3-4+ emerge --resume installing a few other packets until the "sort rules" finally result in the broken packet ends up as number one is quite cumbersome way to work.

It would really be a big help if emerge --resume would guarantee that the very last package of the previous run is the first package of the resume run.



Reproducible: Always

Steps to Reproduce:
1.emerge -e @world
2.glibc breaks
3.emerge --resume --skipfirst

Actual Results:  
I think that on most systems will result in something else than glibc gets skipfirsted.


Expected Results:  
emerge skips glibc as it was the last package in the saved resume previous run.

[Comment 1 Zac Medico 2008-11-16 18:57:19 UTC]

(In reply to comment #0)
> When you try to continue with --resume, it changes the sort order, this can
> make it really hard to hit the correct package with --skipfirst, I often have
> to do 3-4+ --resume until it finally gives me the package i want to skip as
> number one.

I'm not sure what you mean by this because --skipfirst always removes the first package in the list from the old merge list. It then recalculates the new merge list from the remaining packages.

Perhaps you should use the --keep-going option instead of --resume?

Another alternative to --resume is to mask the unwanted package in /etc/portage/package.mask and then recalculate dependencies from scratch.

[Comment 2 Zac Medico 2008-11-16 19:30:04 UTC]

There are 2 other approaches I should also mention:

1) You can use --nodep --resume to ensure that the order of the resume list does not change. However, this disables blocker handling, so it's not good for cases in which the merge list uninstallations triggered by blockers (relatively rare).

2) You can use the AgeSet to create a dynamic package set. For example, this command will create a set named 'anonymous' that will reinstall everything installed more that 1 day ago:

  emerge @anonymous\{class=portage.sets.dbapi.AgeSet,age=1\}

The AgeSet is documented in the set configuration documentation that's installed with USE=doc.

[Comment 3 Steen Eugen "Miravlix" Poulsen 2008-11-16 20:35:03 UTC]

It was a while ago I started changed my behavior from "emerge --resume --skipfirst" to looking at the list before skipping, so I can't know for sure if I fooled myself back then into thinking --skipfirst was broken, it definitely wasn't 2.2_rc14 I was testing with. So, I've marked the bug as resolved later and will come back after I've tested the issue with the current 2.2 portage.

> Perhaps you should use the --keep-going option instead of --resume?

No this will remove package that depends on the package that gets skipped over ruining the emerge -e system (I end up only recompiling a subset of world, instead of just having to deal with a handful of issue packages)
 
> Another alternative to --resume is to mask the unwanted package in
> /etc/portage/package.mask and then recalculate dependencies from scratch.

The package is not unwanted, it does simply not compile without special actions haveing to be handled by me.

Take qemu it requires that I switch to a gcc 3 compiler.

vmware-workstation has this Interactive flag, so it fails when called from my emerge front ends.

libtool1 and libtool2 broken packages that require that I switch to one or the other before they will install.

etc.


Anyway, marked it as later as it's going to take a while to re-run tests to come up with a rc14 test that verifies it or invalidates it.

[Comment 4 Zac Medico 2008-11-16 21:03:24 UTC]

(In reply to comment #3)
> > Perhaps you should use the --keep-going option instead of --resume?
> 
> No this will remove package that depends on the package that gets skipped over
> ruining the emerge -e system (I end up only recompiling a subset of world,
> instead of just having to deal with a handful of issue packages)

We might consider adding an option to change how --keep-going behaves when mixed with --empty-tree.

[Comment 5 Steen Eugen "Miravlix" Poulsen 2008-12-12 12:35:26 UTC]

* ERROR: x11-libs/libview-0.6.2 failed.
 * Call stack:
 *               ebuild.sh, line   49:  Called src_compile
 *             environment, line 2601:  Called die
 * The specific snippet of code:
 *       emake || die "Compilation failed."
 *  The die message:
 *   Compilation failed.
 *
 * If you need support, post the topmost build error, and the call stack if relevant.
 * A complete build log is located at '/var/tmp/portage/x11-libs/libview-0.6.2/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/x11-libs/libview-0.6.2/temp/environment'.
 * This ebuild is from an overlay: '/srv/shared/portage/layman/vmware/'
 *

 * GNU info directory index is up-to-date.

!!! existing preserved libs:
>>> package: sys-devel/libtool-2.2.6a
 *  - /usr/lib/libltdl.so.3
 *  - /usr/lib/libltdl.so.3.1.6
 *      used by /usr/bin/canberra-gtk-play (media-libs/libcanberra-0.10)
 *      used by /usr/bin/epiphany (www-client/epiphany-2.24.2.1-r10)
 *      used by /usr/bin/gnome-sound-properties (gnome-base/gnome-control-center-2.24.0.1)
 *      used by /usr/lib/libcanberra.so.0.1.3 (media-libs/libcanberra-0.10)
Use emerge @preserved-rebuild to rebuild packages using these libraries
liferaft ~ # merge --resume --skipfirst

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild     U ] x11-libs/libview-0.6.2 [0.6.1] USE="-debug" 0 kB [0=>1]
[ebuild     U ] media-plugins/gst-plugins-ffmpeg-0.10.6 [0.10.5] 0 kB [0]
[ebuild     U ] net-analyzer/gnome-netstatus-2.12.2 [2.12.1] USE="-debug" 0 kB [0]
[ebuild     U ] gnome-base/gnome-2.24.1 [2.24.0] USE="-accessibility -cdr -cups -dvdr -esd -ldap -mono" 0 kB [2=>0]

Total: 4 packages (4 upgrades), Size of downloads: 0 kB
Portage tree and overlays:
 [0] /usr/portage
 [1] /srv/shared/portage/layman/vmware
 [2] /srv/shared/portage/layman/gnome

Would you like to resume merging these packages? [Yes/No] 


Well, that took me forever to get another case of this regression.

Portage fails compiling libview. I restart it with emerge --resume --skipfirst and as you see it still has libview as first package and I prolly lost something else.

[Comment 6 Steen Eugen "Miravlix" Poulsen 2008-12-12 12:37:58 UTC]

liferaft ~ # merge --resume

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild     U ] media-plugins/gst-plugins-mad-0.10.10 [0.10.8] 0 kB [0]
[ebuild     U ] x11-libs/libview-0.6.2 [0.6.1] USE="-debug" 0 kB [0=>1]
[ebuild     U ] media-plugins/gst-plugins-ffmpeg-0.10.6 [0.10.5] 0 kB [0]
[ebuild     U ] net-analyzer/gnome-netstatus-2.12.2 [2.12.1] USE="-debug" 0 kB [0]
[ebuild     U ] gnome-base/gnome-2.24.1 [2.24.0] USE="-accessibility -cdr -cups -dvdr -esd -ldap -mono" 0 kB [2=>0]

Total: 5 packages (5 upgrades), Size of downloads: 0 kB
Portage tree and overlays:
 [0] /usr/portage
 [1] /srv/shared/portage/layman/vmware
 [2] /srv/shared/portage/layman/gnome

Would you like to resume merging these packages? [Yes/No] 

And the --resume without skipfirst as you can see it resorted the list and killed the wrong package.

[Comment 7 Zac Medico 2008-12-12 19:25:04 UTC]

One possible explanation for this behavior is that you are building in parallel with --jobs, which causes the merge list to be processed in a somewhat unpredictable order. Were you using --jobs when you observed this behavior?

[Comment 8 Steen Eugen "Miravlix" Poulsen 2008-12-12 20:37:58 UTC]

(In reply to comment #7)
> unpredictable order. Were you using --jobs when you observed this behavior?

Yes.

Multiple failed packages isn't going to make this any prettier, and there is no --skipmany anyway.

[Comment 9 Zac Medico 2008-12-12 22:25:11 UTC]

It seems that --skipfirst is fundamentally broken in a case like this. Perhaps we should focus on making --keep-going behave differently when used together with --emptytree.

[Comment 10 Alex Efros 2011-03-13 23:34:41 UTC]

(In reply to comment #7)
> Were you using --jobs when you observed this behavior?

No. I never used --jobs, but it's usual for me to see this behaviour while `emerge -e world`.