Bug 934382 – start merging the finished builds when the tmpfs goes below a certain amount of free space

• Product: Portage Development
• Status: IN_PROGRESS ---

[Description Andreas K. Hüttel 2024-06-15 23:15:28 UTC]

In particular when using --jobs, emerge now tends to collect some build results before it starts merging them. This fills up a tmpfs where the builds take place, and may lead to avoidable failures. 

Feature request: monitor the free space and the free inodes at the place where builds take place (PORTAGE_TMPDIR), and if it falls below configurable values, merge already finished builds instead of starting up even more builds.

Cheers!

[Comment 1 Zac Medico 2024-06-16 02:24:14 UTC]

This should be pretty easy to do, since it will be similar to --load-average in implementation. Like --load-average, it will monitor the value of a system property and prevent new jobs from starting when the value goes beyond a user defined threshold.

[Comment 2 Eli Schwartz 2024-06-16 03:51:38 UTC]

For background, in bug 663324 portage was enhanced with FEATURES=merge-wait. The goal is that we shouldn't merge one package while another is in the process of being compiled, as that can introduce unpredictable breakage. The downside is that actually merging a package may involve winning a race condition depending on the precise parallelization.

Regularly draining the pending mergeable builds is sensible.

As discussed on IRC, there's an interesting discussion about whether to do this based on possibly-fragile free space / inode heuristics or whether to use # of packages. But either way, something should be implemented.

[Comment 3 Zac Medico 2024-06-16 18:42:20 UTC]

(In reply to Eli Schwartz from comment #2)
> As discussed on IRC, there's an interesting discussion about whether to do
> this based on possibly-fragile free space / inode heuristics or whether to
> use # of packages. But either way, something should be implemented.

Wwe can add a separate option for each, since it only adds an additional user threshold to monitor. For example we could name the options --jobs-tmpdir-space-threshold and --jobs-merge-wait-threshold.

[Comment 4 Zac Medico 2024-06-19 02:04:07 UTC]

(In reply to Eli Schwartz from comment #2)
> Regularly draining the pending mergeable builds is sensible.

Implemented --jobs-merge-wait-threshold in https://github.com/gentoo/portage/pull/1349.

[Comment 5 Zac Medico 2024-06-19 18:13:08 UTC]

Opened https://github.com/gentoo/portage/pull/1351 to supersede https://github.com/gentoo/portage/pull/1345 and distinguish between both used blocks and used files (inodes).