When building large packages make.conf provides one with the option to modify the options passed on to `make` and thereby limit the resources allotted to building processes. One option in particular is -j (--jobs), which limits the number of simultaneous jobs to a pre-defined value. The multiprocessing.eclass provides ebuild-writers with the option to directly get this value via $(makeopts_jobs). For large builds, however, the number of simultaneous processes is often not the limiting factor - IO is. Therefor make provides the option -l (or --load-average) as another constraint: no new build-processes are started as long as the current load is higher than the (floating-point) value provided. This would be especially valuable for big packages like boost or chromium (both of which already use makeopts_jobs(). Please note that emerge provides similarly named options; these only apply to parallel merges, not to the build-processes, handled by make or other build-systems (like ninja in chromium's case). Reproducible: Always
Created attachment 362706 [details] multiprocessing.eclass added the function makeopts_loadavg(), based on makeopts_jobs(). $ diff -Naur /usr/portage/eclass/multiprocessing.eclass multiprocessing.eclass --- /usr/portage/eclass/multiprocessing.eclass 2013-10-12 23:31:16.000000000 +0200 +++ multiprocessing.eclass 2013-11-06 22:30:03.487129897 +0100 @@ -55,6 +55,25 @@ echo ${jobs:-1} } +# @FUNCTION: makeopts_loadavg +# @USAGE: [${MAKEOPTS}] +# @DESCRIPTION: +# Searches the arguments (defaults to ${MAKEOPTS}) and extracts the value set +# for load-average. For make and ninja based builds this will mean new jobs are +# not only limited by the jobs-value, but also by the current load - which might +# get excessive due to I/O and not just due to CPU load. +# Be aware that the returned number might be a floating-point number. Test +# whether your software supports that. +makeopts_loadavg() { + [[ $# -eq 0 ]] && set -- ${MAKEOPTS} + # This assumes the first .* will be more greedy than the second .* + # since POSIX doesn't specify a non-greedy match (i.e. ".*?"). + local lavg=$(echo " $* " | sed -r -n \ + -e 's:.*[[:space:]](-l|--load-average[=[:space:]])[[:space:]]*([0-9]+|[0-9]+\.[0-9]+)[^0-9.]*:\2:p' \ + -e 's:.*[[:space:]](-l|--load-average)[[:space:]].*:999:p') + echo ${lavg:-1} +} + # @FUNCTION: multijob_init # @USAGE: [${MAKEOPTS}] # @DESCRIPTION:
Created attachment 362932 [details, diff] multiprocessing.eclass.patch Attached patch from comment such that it can be downloaded.
it should be noted, that --load-average on MAKEOPTS finds some impressive bugs. MAKEOPTS="-j300 -l5" should mean "you can start up to 300 jobs as long as load is below 5" but what it actually seems to mean is "break up the work into 300 segments and then run a segment when you have load below 5". That means that it will often stall in a work chain much longer than expected and it finds parallalization issues when things are not properly deped in the Makefile. I'm not saying to do this or not to, I'm just saying it finds a lot of bugs that I know I can't fix.
Comment on attachment 362706 [details] multiprocessing.eclass we like to see patches, not whole files
Comment on attachment 362932 [details, diff] multiprocessing.eclass.patch seems like a straightforward implementation
http://sources.gentoo.org/eclass/multiprocessing.eclass?r1=1.3&r2=1.4
