The global_event_loop function currently returns one event loop instance per pid, but we really want it to behave like asycio.get_event_loop() which returns the running loop for current thread if one exists, and even bypasses the event loop policy: > def get_event_loop(): > """Return an asyncio event loop. > > When called from a coroutine or a callback (e.g. scheduled with call_soon > or similar API), this function will always return the running event loop. > > If there is no running event loop set, the function will return > the result of `get_event_loop_policy().get_event_loop()` call. > """ > # NOTE: this function is implemented in C (see _asynciomodule.c) > current_loop = _get_running_loop() > if current_loop is not None: > return current_loop > return get_event_loop_policy().get_event_loop()
Patch posted for review: https://archives.gentoo.org/gentoo-portage-dev/message/58f929a270e88477637f3d787d25c436 https://github.com/gentoo/portage/pull/654
For the case of "loop running in non-main thread" of API consumer, this change makes portage compatible with PEP 492 coroutines with async and await syntax.
With the current patch, unit tests consistently hang with python3.6 and python3.7, but not for python3.8 and later. If I disable this set_wakeup_fd code from bug 655656 then it suppresses the problem: https://gitweb.gentoo.org/proj/portage.git/commit/?id=adee194534f0b3d9762efd1e8e8713c316b93f5a I think the problem must be related to thread usage in the child watcher (also see bug 764905). However, I've tried using a child watcher wrapper to deliver the callbacks via call_soon_threadsafe, and that didn't solve the problem
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=386178481eb86ac603cd90ef1bb6ac6b68e51c50 commit 386178481eb86ac603cd90ef1bb6ac6b68e51c50 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2021-01-04 09:14:36 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2021-01-11 09:36:48 +0000 global_event_loop: return running loop for current thread Like asyncio.get_event_loop(), return the running loop for the current thread if there is one, and otherwise construct a new one if needed. This allows the _safe_loop function to become synonymous with the global_event_loop function. For the case of "loop running in non-main thread" of API consumer, this change makes portage compatible with PEP 492 coroutines with async and await syntax. Portage internals can safely begin using async / await syntax instead of compat_coroutine. Bug: https://bugs.gentoo.org/763339 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/util/_eventloop/global_event_loop.py | 28 +++------------------- lib/portage/util/futures/_asyncio/__init__.py | 30 +++++++++++++++++------- 2 files changed, 24 insertions(+), 34 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=2d9df956e086c0766ee352320ecc346c27d23df3 commit 2d9df956e086c0766ee352320ecc346c27d23df3 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2021-01-11 09:45:19 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2021-01-11 09:47:00 +0000 sys-apps/portage: Bump to version 3.0.13 #763339 always allow event loops to run in threads #764764 fix virtual/dist-kernel slot operator rebuilds #764905 wrap asyncio child watcher for thread safety Bug: https://bugs.gentoo.org/764923 Bug: https://bugs.gentoo.org/763339 Bug: https://bugs.gentoo.org/764764 Bug: https://bugs.gentoo.org/764905 Package-Manager: Portage-3.0.13, Repoman-3.0.2 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.13.ebuild | 268 +++++++++++++++++++++++++++++++++ 2 files changed, 269 insertions(+)
The problem with python3.6 and python3.7 was just that a strong reference was needed for the main loop.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=8d962cb5cc97a5092ff45446c0f8da55b27d2434 commit 8d962cb5cc97a5092ff45446c0f8da55b27d2434 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2021-01-18 10:09:05 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2021-01-18 11:18:43 +0000 coroutine: do not require loop parameter The loop parameter is not needed since global_event_loop now returns the running event loop for the current thread. Bug: https://bugs.gentoo.org/737698 Bug: https://bugs.gentoo.org/763339 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/util/futures/compat_coroutine.py | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
