Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 640328 - portage: emerge resume list should be kept after resuming fail
Summary: portage: emerge resume list should be kept after resuming fail
Status: UNCONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-12-08 20:44 UTC by Nikita Zlobin
Modified: 2022-11-16 07:04 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Nikita Zlobin 2017-12-08 20:44:55 UTC
I have latest stable portage version.

When you resume failed long emerge sequence, resuming sometimes fails, talling about invalid resume list or that some packages are masked due to unsatisfied dependencies, etc....

The end of all these reports is one:

Please restart/continue the operation
manually, or use --skipfirst to skip the first package in the list and
any other packages that may be masked or have missing dependencies.

...what points, that it is possible to run it with --skipfirst or --exclude options... or just edit mtimedb file to get all it working. However, second attempt to resume tells "nothing to resume". And indeed, all section in mtimedb is cleared.

Last time i lost resume list, when tried to update one important lib during all resume sequence (it still would be rebuilt by sequence... i don't remember already, what is that lib, but remember, that nearly 800 from total ≈3000 remained).

And right now i did "haskell-update -cu -- -av". And though i did "emerge -avuDN world" before that, some haskell packages were pulled for update. This time i did nothing like parallel emerge in another term. However, after emerge failed (one of setup process was simply killed, may due to concurrency with palemoon's js), i tried to resume (as i do after such unfortunal process fails), but some packages where masked due to dependencies (this was also reason, why essential option "--keep-going" did not work). And again, list is cleared, while i had emerged 23/62 packages.

One time, before first mentioned fail occured i was enough lucky to copy list from mtimedb into text editor for some purpose (don't remember, what i did), so after few massive search&replaces i could resume from manually recreated list, pasted to mtimedb.
Comment 1 Zac Medico gentoo-dev 2017-12-12 10:31:15 UTC
IIRC the reason we're aggressive about deleting resume lists is that before --keep-going people used to do loops like this:

    while true; do emerge --resume && break; done

Why do you use --resume?

As described in bug 420459, comment #3, instead of using --resume, I typically just run `emerge -uDN @world` again, after masking broken packages if necessary. That pulls in all the packages that were skipped, so I don't really need the list of skipped packages for anything.
Comment 2 Nikita Zlobin 2017-12-13 09:54:36 UTC
Don't you mean, that all '--resume' feature is unneeded, may be orphane, old school, deprecated... (what ever words are better)?

When world update is ran initially, it involves also packages, that need to be rebuilt on update of some packages, which require it by specifying '=' in slot/subslot field, in DEPEND variable.

In case, when dependency is rebuilt, but progress fails somewhere in the middle. rebuilding its dependants, repeated run of initial world update command will not rebuild those packages, as no reason to rebuild.

Usually when it fails due to config or compile error, i search for ability to fix it by manual running these necessary commands in builddir by proper user (after duing # sudo -u portage bash), then runing
# ebuild <pkg> merge

hopely to get it done its job even though it still may need --skipfirst with --resume, to not repeatedly fail on same ebuild (or even if some ebuild was to old to build with latest gcc or other tools, but i unmasked its later revision, which is fixed for that, but keyworded, one example rss-glx).

Also it is important to keep build order. When doing rebuild of entire tree by --emptytree, and list is scrambled... the only remaining way is to start from begining. Moreover keeping in head, that it is usual case, when it fails due to some internal compiler error, so simple 'emerge --resume' is what exactly is needed in this case.

And about mentioned reason for agressive mtimedb removing: it is rather unexpected behavior, because printed text doesn't imply such behavior. Backing up that file in begining of emerge process doesn't seem to be best solution, because during emerge this list is going to change as far as emerge makes progress. Though it is better, because it will save time, necessary to build it again... or backups may be updated and versioned.
Comment 3 Zac Medico gentoo-dev 2017-12-13 22:45:35 UTC
(In reply to Nikita Zlobin from comment #2)
> Don't you mean, that all '--resume' feature is unneeded, may be orphane, old
> school, deprecated... (what ever words are better)?

Maybe so.

> Also it is important to keep build order. When doing rebuild of entire tree
> by --emptytree, and list is scrambled... the only remaining way is to start
> from begining. Moreover keeping in head, that it is usual case, when it
> fails due to some internal compiler error, so simple 'emerge --resume' is
> what exactly is needed in this case.

For this case I think we really should add an option to specify a timestamp that is used together with --emptytree to rebuild only packages built before a certain time.

> And about mentioned reason for agressive mtimedb removing: it is rather
> unexpected behavior, because printed text doesn't imply such behavior.
> Backing up that file in begining of emerge process doesn't seem to be best
> solution, because during emerge this list is going to change as far as
> emerge makes progress. Though it is better, because it will save time,
> necessary to build it again... or backups may be updated and versioned.

We need to specify how it should behave for masked packages and packages with unsatisfied dependencies. Maybe we need options to control this behavior? Would you be upset if it simply dropped some packages from the resume list due to them being masked or having unsatisfied dependencies?
Comment 4 Nikita Zlobin 2022-09-09 05:14:28 UTC
Some good news.
I discovered (already wrote in multiple existing issues), that --nodeps --resume combo prevents dependency check on resume, which is real reason for "Invalid resume list" error. This also has chance to fix unpredictable list reordering on resume.
Comment 5 Nikita Zlobin 2022-11-16 07:04:34 UTC
Looks like it's fixed, though rather under calculate, so may be better check twice. E.g.:
- start installing something with some "to be installed" dependencies, 
- remove some of new deps from mtimedb
- try to restart