Right now, I have a quad-core. Some time in the next few months, I plan to build a machine that can run 24 threads (dual hex-core with SMT). When emerging one or more Gentoo packages, I would like the system to be able to use all available CPU resources to minimize build time. If there's only one package building, I'd like --jobs for make to be 24. If there are LOTS of packages to be built, and all of them are currently in ./configure, then I'd like --jobs for emerge to be 24 (or more because many are I/O bound). Of course, I can't do both, because even that system would likely choke on the maximum of 576 gcc processes that would allow. There is the --load-average option, but that turns out to be basically useless. Load average has too much hysteresis. Starting out, load will be zero, and emerge and make will start launching lots and lots of processes, until load average finally catches up. By then, the system is way overloaded, and then emerge and make won't launch anything, even for some time past the point when the system is not overloaded. In short, all --load-average does is cause an oscillation of floods and droughts of work to do. You would think that this option would be useful on shared systems to prevent emerge from interfering with other work being done, but the oscillation happens there too. It appears that Gentoo really has no mechanism to intelligently match the work generated to the available CPU resources. A quick and dirty solution (with relatively harmless race conditions) would be for emerge and make to pay attention to the instantaneous CPU load. Smarter would be to have make and emerge use a facility similar to Apple's Grand Central Dispatch. Either way, as multicore systems proliferate (have you read about Intel's 80-core prototype they wrote about a few years back?), Gentoo should adapt to use them intelligently. Reproducible: Always
There is already a way, although not very elegant. It involves using distcc to distribute jobs to localhost.
Ah, yes. I've heard of that. Unfortunately, that method has a lot of unnecessary complexity. Why can't emerge and make use a similar mechanism that is entirely local?
It looks like, even with distcc, both configure and preprocessing are not distributed.
I've been looking into this, and I cannot find any web page that mentions how distcc handles the case when multiple independent builds are happening simultaneously. distcc seems to rely on the --jobs option for make. From what I can tell, if I use -jN for make, and multiple makes were running, then you'll get up to multiples of N compiles going on. distcc doesn't seem to have any mechanism of its own to limit the number of concurrent compiles -- if it did, there would be a separate option to supply that info, and it doesn't appear to have one. The only option is the one in MAKEOPTS, which is handed to make, not distcc.
Ok, it looks like distccd has a --jobs option, but my understanding is that jobs sent to localhost don't go through distccd.
You might be interested in bug 279623.
Just set MAKEOPTS to #cpu*2 and try to run emerge --jobs 24, but you will run into dependency issues. If you don't split up the package bundling of the compile jobs, you won't resolve much of these.
I used to do something like that (on my quad core, -j5 for both make and emerge, for a max of 25 concurrent processes), then I would get mystery build failures for KDE packages. I reported this bug, and I discussed it with people on IRC, and we figured out that I was running out of memory, both 8GB of main and 64GB swap, and processes were getting killed off. Turns out, KDE compiles can use a huge amount of memory. This is why I'm suggesting something more intelligent be done.
Here's a blog post about libdispatch on linux: http://blog.technologeek.org/2010/07/17/330 And a link to the debian package: http://packages.debian.org/sid/libdispatch0
