Please support nesting of subprocesses in multiprocessing.eclass. That is, calling multijob_* from within a multijob subprocess to start a new process tree. As I see it, to keep this simple all the 'child' multijobs would have use a separate pipe to communicate with their children. However, all of them would also need to have a way of maintaining subprocess count on the given depth. The latter could probably be done via the use of 'top' multijob pipe to request 'allocations' from master server. I would see the process tree as something like: - A - A-1 - A-2 - B - B-1 assuming that jobcount on level 1 was '2', and on level 2 was '3', with 'B-2' being started whenever one of level 2 processes ends. I can try to write the patches in a few days if you like this concept.
subsets of multijobs should not (cannot really) be responsible for managing the process count as they're all running in different env contexts. the first multijob_init to run will have to create a master process that exists just to maintain mj_num_jobs in the same environment. when the last multijob_finish runs, it will reap that process. you can keep track of this by using an internal var set to $BASHPID. once that's isolated, i don't think any of the other code needs changing. the fact that they're in subshells provides all the isolation we need. i see that we currently leak mj_control_fd. we should probably have both multijob_{init,finish} close that.
