This will allow portage API consumers to mix portage API usage with coroutines, standard library asyncio functions, and third party libraries that support asyncio.
The patch series for bug 649276 includes a patch that adds a ForkExecutor class and uses it to implement an EventLoop run_in_executor method. Also, in this branch I've got an experimental implementation of AbstractEventLoop which uses portage's EventLoop internally: https://github.com/zmedico/portage/tree/asyncio-bug_649588
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=4095be74985c5c2eead5fb480cf37baa11308d62 commit 4095be74985c5c2eead5fb480cf37baa11308d62 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-03-14 08:01:26 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-02 16:53:23 +0000 Add ForkExecutor (bug 649588) This is useful for asynchronous operations that we might need to cancel if they take too long, since (concurrent. futures.ProcessPoolExecutor tasks are not cancellable). The ability to cancel tasks makes this executor useful as an alternative to portage.exception.AlarmSignal. Also add an asyncio-compatible EventLoop.run_in_executor method that uses ForkExecutor as the default executor, which will later be used to implement the corresponding asyncio.AbstractEventLoop run_in_executor method. Bug: https://bugs.gentoo.org/649588 Reviewed-by: Alec Warner <antarus@gentoo.org> pym/portage/util/_eventloop/EventLoop.py | 45 ++++++++- pym/portage/util/futures/executor/__init__.py | 0 pym/portage/util/futures/executor/fork.py | 134 ++++++++++++++++++++++++++ 3 files changed, 178 insertions(+), 1 deletion(-)}
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=24f861173ebe747a470deb8489887c067cd46b0f commit 24f861173ebe747a470deb8489887c067cd46b0f Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-02 03:46:10 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-08 22:04:37 +0000 EventLoop: implement add/remove_reader/writer for asyncio compat (bug 649588) Bug: https://bugs.gentoo.org/649588 pym/portage/util/_eventloop/EventLoop.py | 51 ++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+)}
I've posted a patch that adds a minimal asyncio.AbstractEventLoop implementation: https://archives.gentoo.org/gentoo-portage-dev/message/06399256701c8c52358f0be4a205e039 https://github.com/gentoo/portage/pull/295
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=82759203b9b39389de9ab27aeae58319b604f1ae commit 82759203b9b39389de9ab27aeae58319b604f1ae Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-10 15:16:21 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-11 01:44:34 +0000 async_aux_get: support asyncio via _PortageEventLoopPolicy (bug 649588) Support portage's internal EventLoop as well as the _PortageEventLoop asyncio compatibility wrapper, by using the respective _loop and _asyncio_wrapper attributes where appropriate. Example usage for coroutine with PEP 492 async and await syntax: import asyncio import portage from portage.exception import PortageKeyError from portage.util.futures.unix_events import DefaultEventLoopPolicy asyncio.set_event_loop_policy(DefaultEventLoopPolicy()) async def aux_get_demo(): portdb = portage.portdb for cpv in portdb.cp_list('sys-apps/portage'): try: result = await portdb.async_aux_get(cpv, portage.auxdbkeys) except PortageKeyError as e: # aux_get failed print('error:', cpv, e) else: print(cpv, result) asyncio.get_event_loop().run_until_complete(aux_get_demo()) Bug: https://bugs.gentoo.org/649588 pym/portage/dbapi/porttree.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=f2bece120b1e9bcd7c74fc782cef9016f2147555 commit f2bece120b1e9bcd7c74fc782cef9016f2147555 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-10 06:52:58 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-11 01:44:34 +0000 iter_completed: support asyncio via _PortageEventLoopPolicy (bug 649588) Support portage's internal EventLoop as well as the _PortageEventLoop asyncio compatibility wrapper, by using the respective _loop and _asyncio_wrapper attributes where appropriate. Bug: https://bugs.gentoo.org/649588 pym/portage/tests/util/futures/test_iter_completed.py | 4 ++-- pym/portage/util/futures/iter_completed.py | 3 ++- 2 files changed, 4 insertions(+), 3 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=142d08c0636b172fbc00a7f2b10dc07479a57e2d commit 142d08c0636b172fbc00a7f2b10dc07479a57e2d Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-03-04 20:10:55 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-11 01:44:34 +0000 Add minimal asyncio.AbstractEventLoop implementation (bug 649588) This provides minimal interoperability with existing asyncio code, by adding a portage.util.futures.unix_events.DefaultEventLoopPolicy class that makes asyncio use portage's internal event loop when an instance is passed into asyncio.set_event_loop_policy(). The get_event_loop() method of this policy returns an instance of a _PortageEventLoop class that wraps portage's internal event loop and implements asyncio's AbstractEventLoop interface. The portage.util.futures.asyncio module refers to the real asyncio module when available, and otherwise falls back to a minimal implementation that works with python2.7. The included EventLoopInForkTestCase demonstrates usage, and works with all supported versions of python, include python2.7. In python3.4 and later, API consumers can use asyncio coroutines, since _PortageEventLoop is compatible with asyncio.Task! Bug: https://bugs.gentoo.org/649588 pym/portage/tests/util/futures/asyncio/__init__.py | 0 pym/portage/tests/util/futures/asyncio/__test__.py | 0 .../futures/asyncio/test_event_loop_in_fork.py | 59 +++++++ pym/portage/util/_eventloop/EventLoop.py | 11 +- pym/portage/util/futures/__init__.py | 11 ++ pym/portage/util/futures/_asyncio.py | 114 ++++++++++++ pym/portage/util/futures/events.py | 191 +++++++++++++++++++++ pym/portage/util/futures/futures.py | 9 +- pym/portage/util/futures/unix_events.py | 91 ++++++++++ 9 files changed, 477 insertions(+), 9 deletions(-)}
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=52d4689740f5f9fcf1cf7423e3fe4089dbb4c718 commit 52d4689740f5f9fcf1cf7423e3fe4089dbb4c718 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-11 02:01:43 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-11 02:01:43 +0000 ForkExecutor: support asyncio via _PortageEventLoopPolicy (bug 649588) Support portage's internal EventLoop as well as the _PortageEventLoop asyncio compatibility wrapper, by using the respective _loop and _asyncio_wrapper attributes where appropriate. Bug: https://bugs.gentoo.org/649588 pym/portage/util/futures/executor/fork.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)}
I've posted a patch that implements AbstractEventLoopPolicy.get_child_watcher(): https://archives.gentoo.org/gentoo-portage-dev/message/af1ad9da35b2cb767d3074d848d20a87 https://github.com/gentoo/portage/pull/298 The next step is to implement the AbstractEventLoop.subprocess_exec method.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=a78dca7e47f79ad48aee4909ee10688604996b86 commit a78dca7e47f79ad48aee4909ee10688604996b86 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-11 06:44:41 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-12 08:35:05 +0000 Implement AbstractEventLoopPolicy.get_child_watcher() (bug 649588) Use a _PortageChildWatcher class to wrap portage's internal event loop and implement asyncio's AbstractChildWatcher interface. Bug: https://bugs.gentoo.org/649588 .../util/futures/asyncio/test_child_watcher.py | 45 +++++++++++ pym/portage/util/_eventloop/EventLoop.py | 7 +- pym/portage/util/futures/_asyncio.py | 13 ++++ pym/portage/util/futures/unix_events.py | 90 ++++++++++++++++++++++ 4 files changed, 152 insertions(+), 3 deletions(-)}
I've posted a patch that implements the AbstractEventLoop.subprocess_exec method: https://archives.gentoo.org/gentoo-portage-dev/message/a8d0f9c43d682bb1867fa5427a042e8c https://github.com/gentoo/portage/pull/299
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=449b7b9f30c869781f7012ca53e9cda4efef6f9b commit 449b7b9f30c869781f7012ca53e9cda4efef6f9b Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-12 19:28:47 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-12 19:28:47 +0000 Implement AbstractEventLoop.call_exception_handler (bug 649588) When using asyncio with _PortageEventLoopPolicy, this method is required in order to avoid a NotImplementedError if a coroutine raises an unexpected exception. Bug: https://bugs.gentoo.org/649588 pym/portage/util/_eventloop/EventLoop.py | 68 ++++++++++++++++++++++++++++++++ pym/portage/util/futures/unix_events.py | 2 + 2 files changed, 70 insertions(+)}
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=d31db4dfb58fcd95f2590dfaed19bce4ef31bbd2 commit d31db4dfb58fcd95f2590dfaed19bce4ef31bbd2 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-12 03:56:25 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-13 07:10:10 +0000 Implement _PortageEventLoop.subprocess_exec (bug 649588) In python versions that support asyncio, this allows API consumers to use the asyncio.create_subprocess_exec() function with portage's internal event loop. Currently, subprocess.PIPE is not implemented because that would require an implementation of asyncio's private asyncio.unix_events._UnixReadPipeTransport class. However, it's possible to use pipes created with os.pipe() for stdin, stdout, and stderr, as demonstrated in the included unit tests. Bug: https://bugs.gentoo.org/649588 .../util/futures/asyncio/test_subprocess_exec.py | 163 +++++++++++++++++++++ pym/portage/util/futures/unix_events.py | 98 +++++++++++++ 2 files changed, 261 insertions(+)}
I've posted a patch that implements the AbstractEventLoop.connect_read_pipe method: https://archives.gentoo.org/gentoo-portage-dev/message/8ecb03a38cf2a270945ae08e5a0d1f26 https://github.com/gentoo/portage/pull/301
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=608663259a8a6fa78c32205389dfa58d00b6f11c commit 608663259a8a6fa78c32205389dfa58d00b6f11c Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-15 18:56:43 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-15 18:58:00 +0000 Implement AbstractEventLoop.is_running() (bug 649588) Bug: https://bugs.gentoo.org/649588 pym/portage/util/_eventloop/EventLoop.py | 14 ++++++++++++++ pym/portage/util/futures/unix_events.py | 1 + 2 files changed, 15 insertions(+)}
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=2b6e90fadfb1adcd8ccd2f313aa009b3d19ffefe commit 2b6e90fadfb1adcd8ccd2f313aa009b3d19ffefe Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-15 23:42:49 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-15 23:58:05 +0000 EventLoop: support add_reader/writer fd overlap (bug 649588) The AbstractEventLoop add_reader and add_writer methods need to support simultaneous registration of reader and writer callbacks for the same fd. For example, this feature is used by the standard library's asyncio.unix_events._UnixWritePipeTransport class, which is used to implement AbstractEventLoop.subprocess_exec(stdin=subprocess.PIPE). Bug: https://bugs.gentoo.org/649588 pym/portage/util/_eventloop/EventLoop.py | 83 ++++++++++++++++++++++++++++---- 1 file changed, 73 insertions(+), 10 deletions(-)}
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=9a7b0a006e65f8683716d60574e4f19f8ffd603d commit 9a7b0a006e65f8683716d60574e4f19f8ffd603d Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-14 21:29:29 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-16 00:04:26 +0000 Implement AbstractEventLoop.connect_read_pipe (bug 649588) In python versions that support asyncio, this allows API consumers to use subprocess.PIPE for asyncio.create_subprocess_exec() stdout and stderr parameters. Bug: https://bugs.gentoo.org/649588 .../util/futures/asyncio/test_subprocess_exec.py | 30 ++++ pym/portage/util/futures/unix_events.py | 157 ++++++++++++++++++++- 2 files changed, 184 insertions(+), 3 deletions(-)}
I've posted a patch that implements the AbstractEventLoop.connect_write_pipe method: https://archives.gentoo.org/gentoo-portage-dev/message/766f291eba6f15faa66585a9b420017b https://github.com/gentoo/portage/pull/302
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=72914c0dbae1dfab7565a627451b616e330b8889 commit 72914c0dbae1dfab7565a627451b616e330b8889 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-15 10:11:21 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-17 02:19:57 +0000 Implement AbstractEventLoop.connect_write_pipe (bug 649588) In python versions that support asyncio, this allows API consumers to use subprocess.PIPE for asyncio.create_subprocess_exec() stdin parameters. Bug: https://bugs.gentoo.org/649588 .../util/futures/asyncio/test_subprocess_exec.py | 34 +++ pym/portage/util/futures/transports.py | 90 +++++++ pym/portage/util/futures/unix_events.py | 259 ++++++++++++++++++++- 3 files changed, 373 insertions(+), 10 deletions(-)}
Basic support is complete now.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=90fa156df0e6ef4fa9ef1a80c495511f4630de86 commit 90fa156df0e6ef4fa9ef1a80c495511f4630de86 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-20 15:21:58 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-20 15:50:43 +0000 EbuildBuildDir: remove synchronous unlock method (bug 614108) The synchronous unlock method can trigger event loop recursion if the event loop is already running, which is incompatible with asyncio's default event loop. Therefore, migrate the last consumers to use the async_unlock method, and remove the synchronous unlock method. Bug: https://bugs.gentoo.org/614108 Bug: https://bugs.gentoo.org/649588 pym/_emerge/EbuildBuildDir.py | 21 --------------------- pym/_emerge/Scheduler.py | 2 +- pym/portage/dbapi/vartree.py | 3 ++- pym/portage/package/ebuild/doebuild.py | 9 ++++++--- pym/portage/tests/ebuild/test_ipc_daemon.py | 2 +- 5 files changed, 10 insertions(+), 27 deletions(-)}
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=a68bf18050580eaf60bc0447558c63346b985049 commit a68bf18050580eaf60bc0447558c63346b985049 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-21 08:47:07 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-22 18:30:54 +0000 AsynchronousLock: remove synchronous unlock method (bug 614108) The synchronous unlock method can trigger event loop recursion if the event loop is already running, which is incompatible with asyncio's default event loop. Therefore, migrate the last consumers to use the async_unlock method, and remove the synchronous unlock method. Bug: https://bugs.gentoo.org/614108 Bug: https://bugs.gentoo.org/649588 pym/_emerge/AsynchronousLock.py | 32 ----------------------- pym/portage/tests/locks/test_asynchronous_lock.py | 22 +++++----------- 2 files changed, 7 insertions(+), 47 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=aa6ae16b606c6ed177279f6ef03651c006ff051e commit aa6ae16b606c6ed177279f6ef03651c006ff051e Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-21 08:07:32 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-22 18:30:53 +0000 EbuildBuildDir: remove synchronous lock method (bug 614112) The synchronous lock method can trigger event loop recursion if the event loop is already running, which is incompatible with asyncio's default event loop. Therefore, migrate the last consumers to use the async_lock method, and remove the synchronous lock method. Bug: https://bugs.gentoo.org/614112 Bug: https://bugs.gentoo.org/649588 pym/_emerge/EbuildBuildDir.py | 47 ----------------------------- pym/_emerge/Scheduler.py | 2 +- pym/portage/dbapi/vartree.py | 2 +- pym/portage/package/ebuild/doebuild.py | 9 ++++-- pym/portage/tests/ebuild/test_ipc_daemon.py | 2 +- 5 files changed, 9 insertions(+), 53 deletions(-)}
Fixed in portage-2.3.40-r1.
