Currently portage uses its internal EventLoop implementation in API consumer threads, but it can use the default asyncio event loop implementation instead.
Patch posted for review: https://archives.gentoo.org/gentoo-portage-dev/message/9235d897f0b49f1b25569bb57f0865c6 https://github.com/gentoo/portage/pull/645
New patch: https://archives.gentoo.org/gentoo-portage-dev/message/e66737e9b2205f774a55d97ba04e130e https://github.com/gentoo/portage/pull/646
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=cecd2f8a259cf2991f2324c9a14e26170ba0ddcf commit cecd2f8a259cf2991f2324c9a14e26170ba0ddcf Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-12-06 09:25:17 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-12-07 02:32:27 +0000 Use default asyncio event loop implementation in API consumer threads Make the _safe_loop function return an AsyncioEventLoop instance, so that the default asyncio event loop implementation will be used in API consumer threads. This is possible because the underlying asyncio.get_event_loop() function returns a separate event loop for each thread. The AsyncioEventLoop _run_until_complete method will now appropriately handle a ValueError from signal.set_wakeup_fd(-1) if it is not called in the main thread. For external API consumers calling from a non-main thread, an asyncio loop must be registered for the current thread, or else an error will be raised like this: RuntimeError: There is no current event loop in thread 'Thread-1'. In order to avoid this RuntimeError, the external API consumer is responsible for setting an event loop and managing its lifecycle. For example, this code will set an event loop for the current thread: asyncio.set_event_loop(asyncio.new_event_loop()) In order to avoid a ResourceWarning, the caller should also close the corresponding loop before the current thread terminates. Bug: https://bugs.gentoo.org/758755 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/util/_eventloop/asyncio_event_loop.py | 6 +++++- lib/portage/util/futures/_asyncio/__init__.py | 26 +++++++++++++++++------ 2 files changed, 24 insertions(+), 8 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=dcbcac809213537afaa6b4f9822146a2e984f773 commit dcbcac809213537afaa6b4f9822146a2e984f773 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-12-07 06:05:04 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-12-07 07:48:10 +0000 _safe_loop: instantiate asyncio loop for API consumer thread In order to maintain compatibility with an API consumer thread which has not instantiated an asyncio loop for the current thread prior to calling the portage API, instantiate a loop on its behalf. Since a ResourceWarning will be triggered if the loop has not been closed before the process exits, add the loop to a WeakValueDictionary, and close it if it still exists during exit for the current pid. Fixes: cecd2f8a259c ("Use default asyncio event loop implementation in API consumer threads") Bug: https://bugs.gentoo.org/758755 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/util/futures/_asyncio/__init__.py | 49 ++++++++++++++++++++++----- 1 file changed, 40 insertions(+), 9 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6a1a30033dd0e79e4678a5a6a8ce8d3ceddf265c commit 6a1a30033dd0e79e4678a5a6a8ce8d3ceddf265c Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-12-07 08:13:48 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-12-07 08:48:09 +0000 sys-apps/portage: Bump to version 3.0.12 #758740 Use default asyncio event loop in child processes #758755 Use default asyncio event loop in API consumer threads Bug: https://bugs.gentoo.org/758740 Bug: https://bugs.gentoo.org/758755 Bug: https://bugs.gentoo.org/756028 Package-Manager: Portage-3.0.12, Repoman-3.0.2 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.12.ebuild | 267 +++++++++++++++++++++++++++++++++ 2 files changed, 268 insertions(+)
If an asyncio loop is not running in the main thread, then API consumer threads will not work in practice with less than python3.8, since https://bugs.python.org/issue35621 makes MultiLoopChildWatcher necessary in order to avoid errors like this: > RuntimeError: Cannot add child handler, the child watcher does not have a loop attached.
