Bug 941551

Summary:	Intelligently reduce number of make jobs for ebuilds which require large amounts of RAM
Product:	Portage Development	Reporter:	Patrick McMunn <doctorwhoguy>
Component:	Conceptual/Abstract Ideas	Assignee:	Portage team <dev-portage>
Status:	RESOLVED DUPLICATE
Severity:	normal	CC:	doctorwhoguy
Priority:	Normal
Version:	unspecified
Hardware:	All
OS:	Linux
Whiteboard:
Package list:		Runtime testing required:	---

Description Patrick McMunn 2024-10-15 03:18:38 UTC

I have a system with 16 GB RAM. My CPU has 4 hyper threaded cores. My make.conf has -j9 as the default number of make jobs which is fine for almost all packages I build on my system. But there are a small number of packages which I have to use package.env for to limit the number of make jobs because my system will use so much RAM that it begins using swap and thrashing the hard drive when compiling those packages thus bringing my system to a completely unusable state. Examples include media-gfx/openvdb, media-libs/embree, media-libs/oidn, and net-libs/webkit-gtk.

It would be a beneficial feature to add to portage to automatically detect such packages, or at least it would be a good feature to add to some eclass so that ebuild maintainers can implement a feature, for dealing with packages which use unusually large amounts of RAM per make job and then scaling back the number of make jobs used for a particular ebuild if the system does not have enough RAM to meet the requirements of the make.conf value for MAKEOPTS.

There is already a mechanism in place for checking that a minimum amount of hard drive space is available. Some mechanism for checking the available RAM and then checking that a minimum amount of RAM per make job multiplied by the MAKEOPTS value is available for certain ebuilds would be nice. Then such ebuilds could potentially edit the MAKEOPTS value for that specific ebuild to be a multiple that would yield a total RAM required value under the system's available RAM.

Reproducible: Always

Comment 1 Sam James archtester

2024-10-15 03:20:09 UTC

I think what you really want is ninja to get jobserver support and/or Portage to provide its own which adjusts the number of jobs based on RAM as well as load average.

*** This bug has been marked as a duplicate of bug 692576 ***

Comment 2 Patrick McMunn 2024-10-19 01:43:59 UTC

(In reply to Sam James from comment #1)
> I think what you really want is ninja to get jobserver support and/or
> Portage to provide its own which adjusts the number of jobs based on RAM as
> well as load average.
> 
> *** This bug has been marked as a duplicate of bug 692576 ***

Well, not all build systems use ninja, but yeah, I want the number of make jobs to be automatically adjusted based on available RAM so that the system won't slow to a crawl by thrashing the hard drive due to swap. I didn't realize that this was already a goal. So thank you for pointing me to a preexisting bug report. This is something I'd be willing to donate for if I knew my donation would specifically go to supporting someone working on this issue. It'd probably just be $5/month because I can't afford much, but I'd do that.