Bug 653946 – sys-apps/portage: ManifestScheduler._iter_tasks() event loop recursion

• Status: RESOLVED FIXED
• Product: Portage Development

[Description Zac Medico 2018-04-24 06:27:40 UTC]

There's a portdbapi.getFetchMap call that triggers event loop recursion:

> Traceback (most recent call last):
>   File "bin/egencache", line 1119, in <module>
>     sys.exit(egencache_main(sys.argv[1:]))
>   File "bin/egencache", line 1095, in egencache_main
>     signum = run_main_scheduler(scheduler)
>   File "pym/portage/util/_async/run_main_scheduler.py", line 27, in run_main_scheduler
>     scheduler.wait()
>   File "pym/_emerge/AsynchronousTask.py", line 77, in wait
>     self._wait()
>   File "pym/portage/util/_async/AsyncScheduler.py", line 92, in _wait
>     self._event_loop.iteration()
>   File "pym/portage/util/_eventloop/EventLoop.py", line 281, in iteration
>     return self._iteration(*args)
>   File "pym/portage/util/_eventloop/EventLoop.py", line 346, in _iteration
>     if self._run_timeouts():
>   File "pym/portage/util/_eventloop/EventLoop.py", line 597, in _run_timeouts
>     if self._run_idle_callbacks():
>   File "pym/portage/util/_eventloop/EventLoop.py", line 556, in _run_idle_callbacks
>     if x._callback(*x._args):
>   File "pym/_emerge/AsynchronousTask.py", line 97, in _async_wait_cb
>     self.wait()
>   File "pym/_emerge/AsynchronousTask.py", line 80, in wait
>     self._wait_hook()
>   File "pym/_emerge/AsynchronousTask.py", line 198, in _wait_hook
>     self._exit_listener_stack.pop()(self)
>   File "pym/portage/package/ebuild/_parallel_manifest/ManifestTask.py", line 42, in _manifest_proc_exit
>     self.wait()
>   File "pym/_emerge/AsynchronousTask.py", line 80, in wait
>     self._wait_hook()
>   File "pym/_emerge/AsynchronousTask.py", line 198, in _wait_hook
>     self._exit_listener_stack.pop()(self)
>   File "pym/portage/package/ebuild/_parallel_manifest/ManifestScheduler.py", line 93, in _task_exit
>     AsyncScheduler._task_exit(self, task)
>   File "pym/portage/util/_async/AsyncScheduler.py", line 71, in _task_exit
>     self._schedule()
>   File "pym/_emerge/PollScheduler.py", line 154, in _schedule
>     self._schedule_tasks()
>   File "pym/portage/util/_async/AsyncScheduler.py", line 55, in _schedule_tasks
>     task = self._next_task()
>   File "pym/portage/package/ebuild/_parallel_manifest/ManifestScheduler.py", line 30, in _next_task
>     return next(self._task_iter)
>   File "pym/portage/package/ebuild/_parallel_manifest/ManifestScheduler.py", line 70, in _iter_tasks
>     list(portdb.getFetchMap(cpv, mytree=mytree))
>   File "pym/portage/dbapi/porttree.py", line 734, in getFetchMap
>     mytree=mytree, loop=loop))
>   File "pym/portage/util/_eventloop/EventLoop.py", line 827, in run_until_complete
>     self.iteration()

[Comment 1 Zac Medico 2018-04-24 08:13:07 UTC]

I've posted a patch for review that fixes it to use async_fetch_map:

https://archives.gentoo.org/gentoo-portage-dev/message/d1725e1b736a504ff70dc34030546352
https://github.com/gentoo/portage/pull/313

[Comment 2 Larry the Git Cow 2018-04-27 23:42:24 UTC]

The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=3e77f0199cb401acf974089fb6aa378fd45d0e90

commit 3e77f0199cb401acf974089fb6aa378fd45d0e90
Author:     Zac Medico <[e-mail]>
AuthorDate: 2018-04-24 06:54:05 +0000
Commit:     Zac Medico <[e-mail]>
CommitDate: 2018-04-27 22:56:02 +0000

    ManifestScheduler: async fetchlist_dict (bug 653946)
    
    In order to avoid event loop recursion, pass fetchlist_dict to
    ManifestTask as a Future.
    
    Bug: https://bugs.gentoo.org/653946

 pym/portage/dbapi/porttree.py                      | 70 ++++++++++++++++++++++
 .../ebuild/_parallel_manifest/ManifestScheduler.py | 25 ++++----
 .../ebuild/_parallel_manifest/ManifestTask.py      | 24 +++++++-
 pym/portage/tests/dbapi/test_portdb_cache.py       |  3 +-
 4 files changed, 105 insertions(+), 17 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=be61882996099322bb3a1e82e71f475b4141ad40

commit be61882996099322bb3a1e82e71f475b4141ad40
Author:     Zac Medico <[e-mail]>
AuthorDate: 2018-04-24 23:28:08 +0000
Commit:     Zac Medico <[e-mail]>
CommitDate: 2018-04-27 21:33:00 +0000

    Add iter_gather function (bug 653946)
    
    This is similar to asyncio.gather, but takes an iterator of
    futures as input, and includes support for max_jobs and max_load
    parameters. For bug 653946, this will be used to asynchronously
    gather the results of the portdbapi.async_fetch_map calls that
    are required to generate a Manifest, while using the max_jobs
    parameter to limit the number of concurrent async_aux_get calls.
    
    Bug: https://bugs.gentoo.org/653946

 pym/portage/util/futures/iter_completed.py | 73 ++++++++++++++++++++++++++++++
 1 file changed, 73 insertions(+)}

[Comment 3 Zac Medico 2018-07-02 18:52:21 UTC]

Fixed in portage-2.3.40-r1.

[Comment 4 Zac Medico 2018-07-02 18:54:39 UTC]

Fixed in portage-2.3.40-r1.