It would be nice if portage provided a jobserver to each ebuild phase via the MAKEFLAGS environment variable, and ensured that any slots consumed by a given ebuild phase are automatically reclaimed after the phase completes. Slots can be allocated using emerge --jobs and --load-average arguments. References: https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html http://make.mad-scientist.net/papers/jobserver-implementation/ https://github.com/rust-lang/cargo/pull/4110 https://github.com/alexcrichton/jobserver-rs
I second this request.
This also will help with bugs like #737098. Thank you!
+1 clarification: - current emerge may run parallel ebuilds - each ebuild may use GNU make - top-level GNU make of each ebuild may create jobserver - children GNU make(s) may use created jobserver by extended MAKEFLAGS problem: - there can be multiple GNU make jobserver in parallel - these jobservers can overuse given cpu resources goal: - let emerge create a master GNU make jobserver - each parallel ebuild top-level GNU make should ask as client job than unanswered: - no bash-ish jobserver example for easy emerge or prescript integration? - no simple/plain GNU make jobserver command? - sharing pipes between parallel ebuilds possible? sandbox? GNU make procedure: $ cat Makefile all: echo $$MAKEFLAGS $ make -j63 echo $MAKEFLAGS -j63 --jobserver-auth=3,4
(In reply to and from comment #3) > problem: > - there can be multiple GNU make jobserver in parallel > - these jobservers can overuse given cpu resources Note that you can mitigate the over-provision issue already without a central portage-provided jobserver by using make's --load-average option. Multiple parallel emerges will then implicitly communicate over the load overage and restrict their task creation based on it. For example, on a 8 core system, you could set: MAKEOPTS="--jobs 8 --load-average 9"
The work in make 4.4 (not yet released) will make this much easier as we don't have to worry about fd inheritance at all: https://github.com/ninja-build/ninja/issues/1139#issuecomment-1223785608. Of course, we still need to implement a basic jobserver, but marxin's example is a solid basis to start on.
There's now a PR for this: https://github.com/gentoo/portage/pull/913 from syu!
https://lore.kernel.org/all/20240404111613.2574424-1-martin@geanix.com/
