Bug 822123 – emerge lacks option to limit -j by memory usage

• Product: Portage Development
• Status: UNCONFIRMED ---

[Description korte 2021-11-06 15:34:29 UTC]

It is possible to limit -j JOBS by load. It would be nice, if the number of jobs could be limited by memory.

For every job 2GB could be calculated, if no other defaults or details are available. At least for the big packages there should be a size available. For the small packages there could also be a smaller size, which could lead to higher system utilization while emerging. Since the package size can vary with every version a var in the ebuild could be used or one system wide file with atoms with a memory size information.


https://wiki.gentoo.org/wiki/Portage_TMPDIR_on_tmpfs has a list with big packages.

Reproducible: Always

[Comment 1 Tee KOBAYASHI 2021-11-07 02:42:04 UTC]

Maybe we want something like --memsuspend in GNU parallel (https://www.gnu.org/software/parallel/man.html)?

[Comment 2 korte 2021-11-07 09:08:00 UTC]

(In reply to Tee KOBAYASHI from comment #1)
> Maybe we want something like --memsuspend in GNU parallel
> (https://www.gnu.org/software/parallel/man.html)?

If it is easier to implement, that could be helpful. While one or more jobs are stopped, no new jobs should be started. Imho the job(s) with the least RAM usage should be stopped first. I have no clue how to calculate the swapping load. Is the sum of real memory usage of the running emerge processes plus the memory of the running system of relevance? Is there a recent swapping load number in the system available?
Swapping could be less, when using memsuspend since some stopped jobs stay in the swap. The only prerequisite would be to have enough RAM+swap available, which should be imho possible.

RAM+swap could be:

X=make -j x
Y=emerge -j y
default_size=2GB (of one compile process, e.g.)
BIGGEST=13GB # RAM usage of the biggest package that will be installed, e.g.firefox?

RAM+swap = X*Y*default_size+BIGGEST # having at least 1*default_size for safety reasons, since BIGGEST already includes the default_size and big package may have a lower -j1 in package.env .

This could avoid hitting the memory limiting part of smallest(CPUs,RAM/2GB) when using emerge -j JOBS and JOBS>1.

On my Pi400 there could be several packages compiled during the time a big package with MAKEOPTS="-j1" is using less then 2GB.

[Comment 3 korte 2022-05-24 05:57:16 UTC]

As workaroud I use:
MAKEOPTS="-j2" emerge --jobs=2 --exclude "`grep one-make-thread.conf /etc/portage/package.env | grep -v "^#" | cut -f 1 | tr '\n' ' '`" -DvuaN @world
MAKEOPTS="-j1" emerge --jobs=1 -DvuaN @world

Would be nice to get some feedback in irc, if this works for others, too.