Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 654038

Summary: sys-apps/portage: FetchIterator.__iter__ event loop recursion
Product: Portage Development Reporter: Zac Medico <zmedico>
Component: CoreAssignee: Portage team <dev-portage>
Status: RESOLVED FIXED    
Severity: normal Keywords: InVCS
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: All   
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on: 648790    
Bug Blocks: 591760, 651804, 653856    

Description Zac Medico gentoo-dev 2018-04-25 05:07:22 UTC
The FetchIterator.__iter__ method needs to yield one asynchronous task per ebuild, in order to make this aux_get call asynchronous:

> Traceback (most recent call last):
>   File "bin/emirrordist", line 21, in <module>
>     sys.exit(emirrordist_main(sys.argv[1:]))
>   File "pym/portage/_emirrordist/main.py", line 438, in emirrordist_main
>     signum = run_main_scheduler(MirrorDistTask(config))
>   File "pym/portage/util/_async/run_main_scheduler.py", line 27, in run_main_scheduler
>     scheduler.wait()
>   File "pym/_emerge/AsynchronousTask.py", line 54, in wait
>     self._wait()
>   File "pym/portage/_emirrordist/MirrorDistTask.py", line 237, in _wait
>     CompositeTask._wait(self)
>   File "pym/_emerge/CompositeTask.py", line 90, in _wait
>     self.scheduler.iteration()
>   File "pym/portage/util/_eventloop/EventLoop.py", line 281, in iteration
>     return self._iteration(*args)
>   File "pym/portage/util/_eventloop/EventLoop.py", line 302, in _iteration
>     if self._run_timeouts():
>   File "pym/portage/util/_eventloop/EventLoop.py", line 597, in _run_timeouts
>     if self._run_idle_callbacks():
>   File "pym/portage/util/_eventloop/EventLoop.py", line 556, in _run_idle_callbacks
>     if x._callback(*x._args):
>   File "pym/_emerge/AsynchronousTask.py", line 74, in _async_wait_cb
>     self.wait()
>   File "pym/_emerge/AsynchronousTask.py", line 57, in wait
>     self._wait_hook()
>   File "pym/_emerge/AsynchronousTask.py", line 175, in _wait_hook
>     self._exit_listener_stack.pop()(self)
>   File "pym/portage/util/_async/AsyncScheduler.py", line 64, in _task_exit
>     self._schedule()
>   File "pym/_emerge/PollScheduler.py", line 154, in _schedule
>     self._schedule_tasks()
>   File "pym/portage/util/_async/AsyncScheduler.py", line 48, in _schedule_tasks
>     task = self._next_task()
>   File "pym/portage/util/_async/TaskScheduler.py", line 20, in _next_task
>     return next(self._task_iter)
>   File "pym/portage/_emirrordist/FetchIterator.py", line 72, in __iter__
>     mytree=tree)
>   File "pym/portage/dbapi/porttree.py", line 587, in aux_get
>     myrepo=myrepo, loop=loop))
>   File "pym/portage/util/_eventloop/EventLoop.py", line 827, in run_until_complete
>     self.iteration()
Comment 1 Zac Medico gentoo-dev 2018-04-25 09:29:31 UTC
I've posted a patch for review that will fix this bug:

https://archives.gentoo.org/gentoo-portage-dev/message/7dc8c8f70e80bb5628541e245c2823db
https://github.com/gentoo/portage/pull/315
Comment 2 Larry the Git Cow gentoo-dev 2018-04-28 14:01:57 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=7c652d1b967f9c4c6a7fbd9fc5e46a0e57438a16

commit 7c652d1b967f9c4c6a7fbd9fc5e46a0e57438a16
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-04-25 06:42:10 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-04-28 13:56:42 +0000

    FetchIterator: fix event loop recursion (bug 654038)
    
    Since construction of FetchTask instances requires results from
    aux_get calls that would trigger event loop recursion when executed
    synchronously, add an _async_fetch_tasks function to construct
    FetchTask instances asynchronously and return a Future. Use an
    _EbuildFetchTasks class to wait for the FetchTask instances to
    become available, and then execute them.
    
    Bug: https://bugs.gentoo.org/654038

 pym/portage/_emirrordist/FetchIterator.py | 324 ++++++++++++++++++++----------
 1 file changed, 218 insertions(+), 106 deletions(-)}
Comment 3 Zac Medico gentoo-dev 2018-07-02 18:53:13 UTC
Fixed in portage-2.3.40-r1.