Bug 459936 – multiprocessing.eclass: please support nesting of subprocesses

• Product: Gentoo Linux
• Status: CONFIRMED ---

[Description Michał Górny 2013-03-02 07:07:03 UTC]

Please support nesting of subprocesses in multiprocessing.eclass. That is, calling multijob_* from within a multijob subprocess to start a new process tree.

As I see it, to keep this simple all the 'child' multijobs would have use a separate pipe to communicate with their children. However, all of them would also need to have a way of maintaining subprocess count on the given depth.

The latter could probably be done via the use of 'top' multijob pipe to request 'allocations' from master server.

I would see the process tree as something like:

 - A
   - A-1
   - A-2
 - B
   - B-1

assuming that jobcount on level 1 was '2', and on level 2 was '3', with 'B-2' being started whenever one of level 2 processes ends.

I can try to write the patches in a few days if you like this concept.

[Comment 1 SpanKY 2013-04-28 04:51:39 UTC]

subsets of multijobs should not (cannot really) be responsible for managing the process count as they're all running in different env contexts.

the first multijob_init to run will have to create a master process that exists just to maintain mj_num_jobs in the same environment.  when the last multijob_finish runs, it will reap that process.  you can keep track of this by using an internal var set to $BASHPID.

once that's isolated, i don't think any of the other code needs changing.  the fact that they're in subshells provides all the isolation we need.

i see that we currently leak mj_control_fd.  we should probably have both multijob_{init,finish} close that.