Bug 692576 – sys-apps/portage: GNU make jobserver integration

• Product: Portage Development
• Status: CONFIRMED ---

[Description Zac Medico 2019-08-19 21:38:46 UTC]

It would be nice if portage provided a jobserver to each ebuild phase via the MAKEFLAGS environment variable, and ensured that any slots consumed by a given ebuild phase are automatically reclaimed after the phase completes. Slots can be allocated using emerge --jobs and --load-average arguments.

References:

https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html
http://make.mad-scientist.net/papers/jobserver-implementation/
https://github.com/rust-lang/cargo/pull/4110
https://github.com/alexcrichton/jobserver-rs

[Comment 1 Daniel Santos 2020-02-11 05:35:23 UTC]

I second this request.

[Comment 2 Cănărău Constantin 2022-02-10 17:29:56 UTC]

This also will help with bugs like #737098. Thank you!

[Comment 3 and 2022-06-14 15:04:05 UTC]

+1

clarification:
- current emerge may run parallel ebuilds
- each ebuild may use GNU make
- top-level GNU make of each ebuild may create jobserver
- children GNU make(s) may use created jobserver by extended MAKEFLAGS

problem:
- there can be multiple GNU make jobserver in parallel
- these jobservers can overuse given cpu resources

goal:
- let emerge create a master GNU make jobserver
- each parallel ebuild top-level GNU make should ask as client job than

unanswered:
- no bash-ish jobserver example for easy emerge or prescript integration?
- no simple/plain GNU make jobserver command?
- sharing pipes between parallel ebuilds possible? sandbox?

GNU make procedure:
$ cat Makefile  
all:
        echo $$MAKEFLAGS

$ make -j63
echo $MAKEFLAGS
-j63 --jobserver-auth=3,4

[Comment 4 Florian Schmaus 2022-06-14 16:11:33 UTC]

(In reply to and from comment #3)
> problem:
> - there can be multiple GNU make jobserver in parallel
> - these jobservers can overuse given cpu resources

Note that you can mitigate the over-provision issue already without a central portage-provided jobserver by using make's --load-average option. Multiple parallel emerges will then implicitly communicate over the load overage and restrict their task creation based on it.

For example, on a 8 core system, you could set:

    MAKEOPTS="--jobs 8 --load-average 9"

[Comment 5 Sam James 2022-08-24 22:12:15 UTC]

The work in make 4.4 (not yet released) will make this much easier as we don't have to worry about fd inheritance at all: https://github.com/ninja-build/ninja/issues/1139#issuecomment-1223785608.

Of course, we still need to implement a basic jobserver, but marxin's example is a solid basis to start on.

[Comment 6 Sam James 2022-09-29 01:33:13 UTC]

There's now a PR for this: https://github.com/gentoo/portage/pull/913 from syu!

[Comment 7 Sam James 2024-04-08 15:56:49 UTC]

https://lore.kernel.org/all/[e-mail]/