Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 383439

Summary: emerge parallelization
Product: Gentoo Linux Reporter: Andreas Steinmetz <ast>
Component: New packagesAssignee: Portage team <dev-portage>
Severity: enhancement CC: gentoo
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 184128    

Description Andreas Steinmetz 2011-09-18 02:48:36 UTC
Now this is a suspicion as I don't have hard facts.

When you do an emerge -jN where e.g. N=8 you quite often get the feeling that possible parallelization is not done. It quite often seems that packages that seemingly don't have anything in common don't get emerged in parallel while the -jN option would allow that, thus causing unnecessary delays. An example that just happened to me while watching a system update:

During an emerge update with -j8 the following happens:

app-text/ghostscript_gpl is emerged as a single job, nothing else is running.

Just before net/mail qpopper and dev-lang/php were emerged.

Now, ghostscript for sure can't depend on qpopper or php. Still it was emerged as a single job thus causing unnecessary delays (the darn auto-tools, forced make -j1, things like that).

Another example:
After an interrupted emerge (the zlib _Z_OF versus OF problem) when chromium was emerged but failed (stated problem) and virtuoso-server was long since aborted due to the same problem, after a emergency fix to zconf.h and a new start with emerge -j8 the following happened:

Only 7 jobs out of 8 were started, this including virtuoso-server.
The chromium job was not started so one slot being empty. I am quite certain that chromium doesn't depend on virtuoso-server, otherwise the build wouldn't have been started in the previous emerge run in the first place. Still a possible job slot is being left empty.

Due to the fact that the weekly amount of Gentoo package updates is not neglegible, multicore systems are today state of the art and that emerge doesn't really seem to properly parallelize I would ask to really keep a close eye on this problem and enhance emerge to allow proper parallelization.

Reproducible: Always
Comment 1 Zac Medico gentoo-dev 2011-09-18 22:26:32 UTC
Behavior like you describe can be triggered by deep dependencies, which emerge identifies by traversing dependencies recursively. That's part of the design, and bug 279623 is about adding a mode for more aggressive parallelization.

There are lots of things that depend on ghostscript, so it wouldn't surprise me if it was a deep dependency that triggered the behavior. The output of `emerge -pv --depclean` will show a full dependency tree, which we can use to confirm this.
Comment 2 Zac Medico gentoo-dev 2011-12-12 07:48:29 UTC

*** This bug has been marked as a duplicate of bug 279623 ***
Comment 3 Zac Medico gentoo-dev 2011-12-15 09:21:42 UTC
Another reason this type of behavior is that system packages (or their dependencies) may be waiting for other builds (like ghostscript) to complete before they are merged, in order to avoid issues like the one reported in bug 256616. I have updates running right now, and I noticed this happening during a app-emulation/wine build because there was a sys-power/upower install being held back by the app-emulation/win build.
Comment 4 Zac Medico gentoo-dev 2012-02-15 17:23:44 UTC
*** Bug 403895 has been marked as a duplicate of this bug. ***