Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 459936 - multiprocessing.eclass: please support nesting of subprocesses
Summary: multiprocessing.eclass: please support nesting of subprocesses
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Eclasses (show other bugs)
Hardware: All Linux
: Normal enhancement (vote)
Assignee: Gentoo's Team for Core System packages
Depends on:
Reported: 2013-03-02 07:07 UTC by Michał Górny
Modified: 2021-03-25 16:41 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Note You need to log in before you can comment on or make changes to this bug.
Description Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2013-03-02 07:07:03 UTC
Please support nesting of subprocesses in multiprocessing.eclass. That is, calling multijob_* from within a multijob subprocess to start a new process tree.

As I see it, to keep this simple all the 'child' multijobs would have use a separate pipe to communicate with their children. However, all of them would also need to have a way of maintaining subprocess count on the given depth.

The latter could probably be done via the use of 'top' multijob pipe to request 'allocations' from master server.

I would see the process tree as something like:

 - A
   - A-1
   - A-2
 - B
   - B-1

assuming that jobcount on level 1 was '2', and on level 2 was '3', with 'B-2' being started whenever one of level 2 processes ends.

I can try to write the patches in a few days if you like this concept.
Comment 1 SpanKY gentoo-dev 2013-04-28 04:51:39 UTC
subsets of multijobs should not (cannot really) be responsible for managing the process count as they're all running in different env contexts.

the first multijob_init to run will have to create a master process that exists just to maintain mj_num_jobs in the same environment.  when the last multijob_finish runs, it will reap that process.  you can keep track of this by using an internal var set to $BASHPID.

once that's isolated, i don't think any of the other code needs changing.  the fact that they're in subshells provides all the isolation we need.

i see that we currently leak mj_control_fd.  we should probably have both multijob_{init,finish} close that.