It would be nice if portage provided a jobserver to each ebuild phase via the MAKEFLAGS environment variable, and ensured that any slots consumed by a given ebuild phase are automatically reclaimed after the phase completes. Slots can be allocated using emerge --jobs and --load-average arguments.
I second this request.
This also will help with bugs like #737098. Thank you!
- current emerge may run parallel ebuilds
- each ebuild may use GNU make
- top-level GNU make of each ebuild may create jobserver
- children GNU make(s) may use created jobserver by extended MAKEFLAGS
- there can be multiple GNU make jobserver in parallel
- these jobservers can overuse given cpu resources
- let emerge create a master GNU make jobserver
- each parallel ebuild top-level GNU make should ask as client job than
- no bash-ish jobserver example for easy emerge or prescript integration?
- no simple/plain GNU make jobserver command?
- sharing pipes between parallel ebuilds possible? sandbox?
GNU make procedure:
$ cat Makefile
$ make -j63
(In reply to and from comment #3)
> - there can be multiple GNU make jobserver in parallel
> - these jobservers can overuse given cpu resources
Note that you can mitigate the over-provision issue already without a central portage-provided jobserver by using make's --load-average option. Multiple parallel emerges will then implicitly communicate over the load overage and restrict their task creation based on it.
For example, on a 8 core system, you could set:
MAKEOPTS="--jobs 8 --load-average 9"
The work in make 4.4 (not yet released) will make this much easier as we don't have to worry about fd inheritance at all: https://github.com/ninja-build/ninja/issues/1139#issuecomment-1223785608.
Of course, we still need to implement a basic jobserver, but marxin's example is a solid basis to start on.
There's now a PR for this: https://github.com/gentoo/portage/pull/913 from syu!