All portage internals have been migrated to use asyncio compatible APIs, so the next step is to use asyncio's default event loop for version of python that have asyncio. Due to lack of support for forks in asyncio's default event loop policy (see https://bugs.python.org/issue22087), it would make sense for portage to set a customized event loop policy in any forks that is spawns, which can easily be done after the fork in the ForkProcess._spawn method.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=82a3cda6f1ffbbbd75c962d610c7eb4a04a865d9 commit 82a3cda6f1ffbbbd75c962d610c7eb4a04a865d9 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-05-05 21:00:51 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-05-05 21:00:51 +0000 Minimize _asyncio_wrapper usage (bug 654390) Since migration to asyncio compatible APIs is now complete, minimize _asyncio_wrapper usage. Also remove _asyncio_wrapper from SchedulerInterface, since there are no more consumers. Bug: https://bugs.gentoo.org/654390 pym/_emerge/AbstractPollTask.py | 2 +- pym/portage/_emirrordist/FetchIterator.py | 1 - pym/portage/dbapi/porttree.py | 3 --- .../tests/util/futures/asyncio/test_subprocess_exec.py | 1 - pym/portage/tests/util/futures/test_iter_completed.py | 4 ++-- pym/portage/tests/util/futures/test_retry.py | 16 ++++++++-------- pym/portage/util/_async/SchedulerInterface.py | 1 - pym/portage/util/futures/__init__.py | 5 +---- pym/portage/util/futures/_asyncio/tasks.py | 1 - pym/portage/util/futures/executor/fork.py | 3 +-- pym/portage/util/futures/futures.py | 2 +- pym/portage/util/futures/iter_completed.py | 3 --- 12 files changed, 14 insertions(+), 28 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=5a5ed99cb5a6e8913df2e9ca29b4b4d5c179c20f commit 5a5ed99cb5a6e8913df2e9ca29b4b4d5c179c20f Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-05-05 23:04:10 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-05-06 00:35:44 +0000 RetryTestCase: support ThreadPoolExecutor (bug 654390) In order to support the default asyncio event loop's ThreadPoolExecutor, use a threading.Event instance to support cancellation of tasks. Bug: https://bugs.gentoo.org/654390 pym/portage/tests/util/futures/test_retry.py | 96 +++++++++++++++++++++------- 1 file changed, 74 insertions(+), 22 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=85ac23b7c0c58cef72d22281d66d086521c01e3e commit 85ac23b7c0c58cef72d22281d66d086521c01e3e Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-05-06 11:05:03 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-05-06 11:41:45 +0000 asyncio: add _wrap_loop helper (bug 654390) In order to deal with asyncio event loop compatibility issues, add a _wrap_loop helper. For example, since python3.4 does not have the AbstractEventLoop.create_future() method, this helper function can be used to add a wrapper that implements the create_future method for python3.4. Bug: https://bugs.gentoo.org/654390 pym/portage/dbapi/porttree.py | 12 ++++++------ .../tests/util/futures/asyncio/test_child_watcher.py | 2 +- .../util/futures/asyncio/test_event_loop_in_fork.py | 8 ++++---- .../tests/util/futures/asyncio/test_pipe_closed.py | 4 ++-- .../util/futures/asyncio/test_run_until_complete.py | 2 +- .../util/futures/asyncio/test_subprocess_exec.py | 4 ++-- pym/portage/util/futures/_asyncio/__init__.py | 19 ++++++++++++++++++- pym/portage/util/futures/_asyncio/tasks.py | 7 +++++-- pym/portage/util/futures/executor/fork.py | 4 ++-- pym/portage/util/futures/iter_completed.py | 7 +++---- pym/portage/util/futures/retry.py | 3 +-- 11 files changed, 45 insertions(+), 27 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=2d500ce2bc96995752dfc2fb475a7abe907e38b6 commit 2d500ce2bc96995752dfc2fb475a7abe907e38b6 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-05-06 21:28:25 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-05-06 22:51:23 +0000 asyncio: explicitly close event loops (bug 654390) The default asyncio event loop triggers a resource warning if it is not explicitly closed, therefore close it when appropriate. Bug: https://bugs.gentoo.org/654390 bin/ebuild | 4 ++++ bin/ebuild-ipc.py | 5 ++++- bin/egencache | 5 ++++- bin/emaint | 3 +++ bin/emerge | 5 +++++ bin/emirrordist | 6 +++++- bin/portageq | 6 +++++- bin/quickpkg | 3 +++ pym/portage/tests/runTests.py | 6 +++++- pym/portage/tests/util/futures/asyncio/test_child_watcher.py | 5 +++++ pym/portage/tests/util/futures/asyncio/test_event_loop_in_fork.py | 6 ++++++ pym/portage/tests/util/futures/asyncio/test_pipe_closed.py | 7 +++++++ pym/portage/tests/util/futures/asyncio/test_run_until_complete.py | 5 +++++ pym/portage/tests/util/futures/asyncio/test_subprocess_exec.py | 7 ++++++- pym/portage/tests/util/futures/test_retry.py | 5 ++++- pym/portage/util/_eventloop/global_event_loop.py | 5 ++++- 16 files changed, 75 insertions(+), 8 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=fefc6f1feb57503ce4826de1e405c12ef761c0ce commit fefc6f1feb57503ce4826de1e405c12ef761c0ce Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-05-06 23:41:15 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-05-07 00:19:54 +0000 RetryTestCase: test ThreadPoolExecutor and ForkExecutor (bug 654390) ThreadPoolExecutor is the default asyncio event loop's default executor, so explicitly test it. Also explicitly test ForkExecutor, since it is more useful in cases where tasks may need to be forcefully cancelled. Bug: https://bugs.gentoo.org/654390 pym/portage/tests/util/futures/test_retry.py | 37 ++++++++++++++++++++++++++-- 1 file changed, 35 insertions(+), 2 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=505707a335683330ac69b1c7f428e8a1c66d0b2e commit 505707a335683330ac69b1c7f428e8a1c66d0b2e Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-05-07 08:14:19 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-05-07 08:17:18 +0000 rsync: explicitly use ForkExecutor for key refresh retry (bug 654390) The ThreadPoolExecutor that asyncio uses by default does not support cancellation of tasks, therefore use ForkExecutor for task cancellation support, in order to enforce timeouts. Bug: https://bugs.gentoo.org/654390 pym/portage/sync/modules/rsync/rsync.py | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
I've posted a patch for review, which will make portage use asyncio's default event loop for python3.4 and later: https://archives.gentoo.org/gentoo-portage-dev/message/ce688df3a3a73e734efbc59bb8994fad https://github.com/gentoo/portage/pull/320
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=96cc073263910e7c1b0f48cc08a4db4b56ffe408 commit 96cc073263910e7c1b0f48cc08a4db4b56ffe408 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-05-01 08:15:39 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-05-08 16:42:03 +0000 global_event_loop: use asyncio event loop (bug 654390) For python3.4 and later, in the main process, replace portage's internal event loop with the standard library's asyncio event loop. Continue to use portage's internal event loop in subprocesses, since asyncio's event loop is not guaranteed to work well in subprocesses (see upstream python issues 22087 and 29703). An _AsyncioEventLoopPolicy class, derived from _PortageEventLoopPolicy, is needed for some unit tests that modify asyncio's event loop policy. This policy is not needed for anything other than unit testing. Portage uses asyncio's default event loop policy, and API consumers are free to use any desired event loop policy. Portage's asynchronous functions that accept a 'loop' parameter will work with any compatible asyncio.AbstractEventLoop implementation, since an internal _wrap_loop function automatically adapts the loop for internal use. Bug: https://bugs.gentoo.org/654390 Reviewed-by: Brian Dolbec <dolsen@gentoo.org> Reviewed-by: Alec Warner <antarus@gentoo.org> pym/portage/util/_async/SchedulerInterface.py | 3 + pym/portage/util/_eventloop/asyncio_event_loop.py | 77 +++++++++++++++++++++++ pym/portage/util/_eventloop/global_event_loop.py | 5 +- pym/portage/util/futures/_asyncio/__init__.py | 17 +++++ pym/portage/util/futures/unix_events.py | 43 ++++++++++++- 5 files changed, 143 insertions(+), 2 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=b414df90b50aaf73dda3708680c6951e7dc8bad9 commit b414df90b50aaf73dda3708680c6951e7dc8bad9 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-05-16 23:28:13 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-05-16 23:28:13 +0000 repoman: explicitly close event loops (bug 654390) The default asyncio event loop triggers a resource warning if it is not explicitly closed, therefore close it when appropriate. Bug: https://bugs.gentoo.org/654390 repoman/bin/repoman | 3 +++ repoman/pym/repoman/tests/runTests.py | 6 +++++- 2 files changed, 8 insertions(+), 1 deletion(-)
Fixed in portage-2.3.40-r1.
