Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 934382 - start merging the finished builds when the tmpfs goes below a certain amount of free space
Summary: start merging the finished builds when the tmpfs goes below a certain amount ...
Status: IN_PROGRESS
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: PullRequest
Depends on:
Blocks: 184128
  Show dependency tree
 
Reported: 2024-06-15 23:15 UTC by Andreas K. Hüttel
Modified: 2024-06-19 18:13 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Andreas K. Hüttel archtester gentoo-dev 2024-06-15 23:15:28 UTC
In particular when using --jobs, emerge now tends to collect some build results before it starts merging them. This fills up a tmpfs where the builds take place, and may lead to avoidable failures. 

Feature request: monitor the free space and the free inodes at the place where builds take place (PORTAGE_TMPDIR), and if it falls below configurable values, merge already finished builds instead of starting up even more builds.

Cheers!
Comment 1 Zac Medico gentoo-dev 2024-06-16 02:24:14 UTC
This should be pretty easy to do, since it will be similar to --load-average in implementation. Like --load-average, it will monitor the value of a system property and prevent new jobs from starting when the value goes beyond a user defined threshold.
Comment 2 Eli Schwartz 2024-06-16 03:51:38 UTC
For background, in bug 663324 portage was enhanced with FEATURES=merge-wait. The goal is that we shouldn't merge one package while another is in the process of being compiled, as that can introduce unpredictable breakage. The downside is that actually merging a package may involve winning a race condition depending on the precise parallelization.

Regularly draining the pending mergeable builds is sensible.

As discussed on IRC, there's an interesting discussion about whether to do this based on possibly-fragile free space / inode heuristics or whether to use # of packages. But either way, something should be implemented.
Comment 3 Zac Medico gentoo-dev 2024-06-16 18:42:20 UTC
(In reply to Eli Schwartz from comment #2)
> As discussed on IRC, there's an interesting discussion about whether to do
> this based on possibly-fragile free space / inode heuristics or whether to
> use # of packages. But either way, something should be implemented.

Wwe can add a separate option for each, since it only adds an additional user threshold to monitor. For example we could name the options --jobs-tmpdir-space-threshold and --jobs-merge-wait-threshold.
Comment 4 Zac Medico gentoo-dev 2024-06-19 02:04:07 UTC
(In reply to Eli Schwartz from comment #2)
> Regularly draining the pending mergeable builds is sensible.

Implemented --jobs-merge-wait-threshold in https://github.com/gentoo/portage/pull/1349.
Comment 5 Zac Medico gentoo-dev 2024-06-19 18:13:08 UTC
Opened https://github.com/gentoo/portage/pull/1351 to supersede https://github.com/gentoo/portage/pull/1345 and distinguish between both used blocks and used files (inodes).