The problem is that doebuild calls _spawn_phase which uses asyncio._safe_loop(), and then the coroutine code in EbuildPhase tries to use the global event loop instead of the local loop from asyncio._safe_loop().
Created attachment 655180 [details] example script that hangs
There are multiple coroutines that need to be fixed, and this fixes the first one: > diff --git a/lib/_emerge/EbuildPhase.py b/lib/_emerge/EbuildPhase.py > index 4bc2749bd..07209f976 100644 > --- a/lib/_emerge/EbuildPhase.py > +++ b/lib/_emerge/EbuildPhase.py > @@ -70,11 +70,11 @@ class EbuildPhase(CompositeTask): > _locked_phases = ("setup", "preinst", "postinst", "prerm", "postrm") > > def _start(self): > - future = asyncio.ensure_future(self._async_start(), loop=self.scheduler) > + future = asyncio.ensure_future(self._async_start(loop=self.scheduler), loop=self.scheduler) > self._start_task(AsyncTaskFuture(future=future), self._async_start_exit) > > @coroutine > - def _async_start(self): > + def _async_start(self, loop=None): > > need_builddir = self.phase not in EbuildProcess._phases_without_builddir
This patch makes it easy to find the missing loop arguments: > diff --git a/lib/portage/util/futures/compat_coroutine.py b/lib/portage/util/futures/compat_coroutine.py > index 79bd0da68..741fbe296 100644 > --- a/lib/portage/util/futures/compat_coroutine.py > +++ b/lib/portage/util/futures/compat_coroutine.py > @@ -67,6 +67,8 @@ def _generator_future(generator_func, *args, **kwargs): > keyword argument named 'loop' is given, then it is used instead of > the default event loop. > """ > + if kwargs.get('loop') is None: > + raise AssertionError("Missing required argument 'loop'") > loop = asyncio._wrap_loop(kwargs.get('loop')) > result = loop.create_future() > _GeneratorTask(generator_func(*args, **kwargs), result, loop=loop)
What on earth commit does that reference, Zac?! And if its 2012, is it still relevant?!? :)
(In reply to Michael 'veremitz' Everitt from comment #4) > What on earth commit does that reference, Zac?! And if its 2012, is it still > relevant?!? :) For completeness, here is a reply Zac posted to my via IRC [#gentoo-portage] - [Tue 6:32 pm]<zmedico> veremitz: bug 737698 is triggered by the local event loop support that solves https://bugs.sabayon.org/show_bug.cgi?id=3305 (since local event loops can run in non-main threads)
The error shown in https://bugs.sabayon.org/show_bug.cgi?id=3305 is still relevant, but today with the asyncio event loop it would manifest as RuntimeError('This event loop is already running') if two threads try to run the main loop at once.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=dc7919541712d846574e6b7d672a3bed0ca7ef1a commit dc7919541712d846574e6b7d672a3bed0ca7ef1a Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-08-18 06:31:54 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-08-19 04:01:46 +0000 coroutine: use explicit loop parameter (bug 737698) In order to support local event loops within API functions like doebuild, use an explicit loop parameter when calling a coroutine. Internal code will now raise an AssertionError if the loop parameter is omitted for a coroutine, but API consumers may omit it. Bug: https://bugs.gentoo.org/737698 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/Binpkg.py | 8 ++-- lib/_emerge/EbuildPhase.py | 16 ++++---- lib/_emerge/Scheduler.py | 4 +- lib/_emerge/SequentialTaskQueue.py | 4 +- lib/_emerge/SpawnProcess.py | 5 ++- lib/portage/dbapi/bintree.py | 12 +++--- lib/portage/dbapi/vartree.py | 8 ++-- .../repository/storage/hardlink_quarantine.py | 26 ++++++------- lib/portage/repository/storage/hardlink_rcu.py | 34 ++++++++-------- lib/portage/repository/storage/inplace.py | 10 ++--- lib/portage/repository/storage/interface.py | 10 ++--- lib/portage/sync/syncbase.py | 2 +- lib/portage/tests/dbapi/test_auxdb.py | 9 +++-- lib/portage/tests/emerge/test_simple.py | 6 +-- lib/portage/tests/process/test_AsyncFunction.py | 4 +- lib/portage/tests/process/test_PipeLogger.py | 2 +- .../util/futures/asyncio/test_child_watcher.py | 4 +- .../tests/util/futures/test_compat_coroutine.py | 45 ++++++++++++---------- lib/portage/tests/util/test_socks5.py | 2 +- lib/portage/util/_async/BuildLogger.py | 4 +- lib/portage/util/_async/ForkProcess.py | 6 +-- lib/portage/util/_async/PipeLogger.py | 4 +- lib/portage/util/_async/SchedulerInterface.py | 4 +- lib/portage/util/futures/_asyncio/process.py | 16 ++++---- lib/portage/util/futures/_sync_decorator.py | 3 +- lib/portage/util/futures/compat_coroutine.py | 7 +++- lib/portage/util/socks5.py | 4 +- repoman/lib/repoman/modules/scan/depend/profile.py | 4 +- 28 files changed, 138 insertions(+), 125 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=ed032cc07b8877e32f3c5d4a9f6e95096d6419ea commit ed032cc07b8877e32f3c5d4a9f6e95096d6419ea Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-08-19 05:40:22 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-08-19 05:43:17 +0000 app-portage/repoman: Bump to version 3.0.1 #737698 Update repoman for compatibility with portage-3.0.4 Bug: https://bugs.gentoo.org/737698 Package-Manager: Portage-3.0.4, Repoman-3.0.1 Signed-off-by: Zac Medico <zmedico@gentoo.org> app-portage/repoman/Manifest | 1 + app-portage/repoman/repoman-3.0.1.ebuild | 64 ++++++++++++++++++++++++++++++++ 2 files changed, 65 insertions(+) https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=8304a26f4c4d5ef446cac60caa41df5529700929 commit 8304a26f4c4d5ef446cac60caa41df5529700929 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-08-19 05:13:39 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-08-19 05:43:16 +0000 sys-apps/portage: Bump to version 3.0.4 #737698 fix hang for API consumers, involving support for local event loops which may run in threads Bug: https://bugs.gentoo.org/733180 Bug: https://bugs.gentoo.org/737698 Package-Manager: Portage-3.0.3, Repoman-3.0.0 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.4.ebuild | 264 ++++++++++++++++++++++++++++++++++ 2 files changed, 265 insertions(+)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=8d962cb5cc97a5092ff45446c0f8da55b27d2434 commit 8d962cb5cc97a5092ff45446c0f8da55b27d2434 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2021-01-18 10:09:05 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2021-01-18 11:18:43 +0000 coroutine: do not require loop parameter The loop parameter is not needed since global_event_loop now returns the running event loop for the current thread. Bug: https://bugs.gentoo.org/737698 Bug: https://bugs.gentoo.org/763339 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/util/futures/compat_coroutine.py | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)