Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 608900

Summary: sys-apps/portage: emerge does does not spawn parallel builds when appropriate in some cases
Product: Portage Development Reporter: Zac Medico <zmedico>
Component: Core - Interface (emerge)Assignee: Portage team <dev-portage>
Status: RESOLVED INVALID    
Severity: normal    
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: All   
See Also: https://bugs.gentoo.org/show_bug.cgi?id=279623
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 184128    

Description Zac Medico gentoo-dev 2017-02-10 20:47:47 UTC
I notices that a large build with portage-2.3.3 was not spawning parallel builds when it seemed like it should, so I sent it a SIGUSR1 signal and poked around. I set a breakpoint before the break statement in the Scheduler._dependent_on_scheduled_merges method, and I found that it was failing to choose packages due to a dependency on ('ebuild', '/', 'sys-devel/gettext-0.19.7', 'merge', 'gentoo'), even though the package was neither in self._pkg_queue nor self._completed_tasks:

(Pdb) bt
  /usr/lib/python-exec/python3.4/emerge(50)<module>()
-> retval = emerge_main()
  /usr/lib64/python3.4/site-packages/_emerge/main.py(1224)emerge_main()
-> return run_action(emerge_config)
  /usr/lib64/python3.4/site-packages/_emerge/actions.py(3261)run_action()
-> retval = action_build(emerge_config, spinner=spinner)
  /usr/lib64/python3.4/site-packages/_emerge/actions.py(524)action_build()
-> retval = mergetask.merge()
  /usr/lib64/python3.4/site-packages/_emerge/Scheduler.py(1042)merge()
-> rval = self._merge()
  /usr/lib64/python3.4/site-packages/_emerge/Scheduler.py(1444)_merge()
-> self._main_loop()
  /usr/lib64/python3.4/site-packages/_emerge/Scheduler.py(1415)_main_loop()
-> self._event_loop.iteration()
  /usr/lib64/python3.4/site-packages/portage/util/_eventloop/EventLoop.py(241)iteration()
-> if self._run_timeouts():
  /usr/lib64/python3.4/site-packages/portage/util/_eventloop/EventLoop.py(496)_run_timeouts()
-> if not x.function(*x.args):
  /usr/lib64/python3.4/site-packages/_emerge/PollScheduler.py(127)_schedule()
-> self._schedule_tasks()
  /usr/lib64/python3.4/site-packages/_emerge/Scheduler.py(1612)_schedule_tasks()
-> if self._schedule_tasks_imp():
  /usr/lib64/python3.4/site-packages/_emerge/Scheduler.py(1711)_schedule_tasks_imp()
-> pkg = self._choose_pkg()
  /usr/lib64/python3.4/site-packages/_emerge/Scheduler.py(1504)_choose_pkg()
-> if not self._dependent_on_scheduled_merges(pkg, later):
> /usr/lib64/python3.4/site-packages/_emerge/Scheduler.py(1552)_dependent_on_scheduled_merges()
-> dependent = True
(Pdb) p node
<Package ('ebuild', '/', 'sys-devel/gettext-0.19.7', 'merge', 'gentoo')>
(Pdb) p node in self._pkg_queue
False
(Pdb) p node in self._completed_tasks
False
(Pdb) p node in self._digraph
True

So, this one 'merge' node somehow got into the digraph, but not into the merge list. This prevents all of its parent nodes from being spawned as parallel builds.
Comment 1 Zac Medico gentoo-dev 2017-02-10 22:12:22 UTC
It turns out that the behavior that I observed was actually the designed behavior, because the sys-devel/gettext-0.19.7 package had been built but is was in the merge queue due to being a dependency of @system, as discussed in the FAQ (see bug 256616 and bug 259954):

https://wiki.gentoo.org/wiki/Project:Portage/FAQ#When_packages_are_built_in_parallel_with_the_--jobs_option.2C_why_aren.27t_some_packages_installed_immediately_after_they_have_finished_building.3F

So, gettext-0.19.7 was in an in-between state. It was no longer in self._pkg_queue and it was waiting to be merged, and it would be added to self._completed_tasks after it got merged.