| Summary: | sys-apps/portage: eliminate EventLoop.iteration() calls for asyncio compat | ||
|---|---|---|---|
| Product: | Portage Development | Reporter: | Zac Medico <zmedico> |
| Component: | Core | Assignee: | Portage team <dev-portage> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | Keywords: | InVCS |
| Priority: | Normal | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Bug Depends on: | 653810, 653946, 654038 | ||
| Bug Blocks: | 591760, 651804 | ||
| Attachments: | AsynchronousTask: disable event loop recursion | ||
|
Description
Zac Medico
2018-04-23 03:49:46 UTC
The synchronous AsynchronousTask.wait() method should really be eliminated, and replaced with an asynchronous method that returns a Future. Than way, any code that needs to run the event loop will need to explicitly call loop.run_until_complete(future). (In reply to Zac Medico from comment #1) > The synchronous AsynchronousTask.wait() method should really be eliminated, > and replaced with an asynchronous method that returns a Future. Than way, > any code that needs to run the event loop will need to explicitly call > loop.run_until_complete(future). I've post a patch series for review, which will add an AsynchronousTask.async_wait() method to replace the synchronous AsynchronousTask.wait() method: https://archives.gentoo.org/gentoo-portage-dev/message/27cb02d75ea82b8f30c1905dbd8b55aa https://github.com/gentoo/portage/pull/312 The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=71a5a82313226f7be0d966d49392a53139a96f6b commit 71a5a82313226f7be0d966d49392a53139a96f6b Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-24 02:47:11 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-26 03:19:22 +0000 AsynchronousTask: add async_wait() method (bug 653856) Since the AsynchronousTask.wait() method is prone to event loop recursion, deprecate it, and add an async_wait() method method to replace it. Instead of using task.wait() in order to implicitly run the event loop, now loop.run_until_complete(task.async_wait()) will be used to explicitly run the event loop. This explicit approach will make it more obvious when code will trigger event loop recursion which would not be compatible with asyncio's default event loop. Bug: https://bugs.gentoo.org/653856 pym/_emerge/AsynchronousTask.py | 23 +++++++++++++++++++++++ pym/portage/tests/ebuild/test_ipc_daemon.py | 2 +- 2 files changed, 24 insertions(+), 1 deletion(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=6c865d6b3130de63dc3d534d2a0596dd59d5e662 commit 6c865d6b3130de63dc3d534d2a0596dd59d5e662 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-24 02:21:59 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-26 03:19:21 +0000 AsynchronousTask: add scheduler attribute (bug 653856) Add an AsynchronousTask.scheduler attribute, which will hold an event loop instance in order to work with Future instances. Name the attribute "scheduler" because this is the name used by many existing subclasses of AsynchronousTask. Since the AsyncScheduler class already has an _event_loop attribute that it inherits from PollScheduler, use a property for compatibility. Also update the SlotObject constructor to allow a subclass to override an attribute from __slots__ with a property, so that AsyncScheduler can use a property for the scheduler attribute. Bug: https://bugs.gentoo.org/653856 pym/_emerge/AbstractPollTask.py | 3 +-- pym/_emerge/AsynchronousLock.py | 2 +- pym/_emerge/AsynchronousTask.py | 2 +- pym/_emerge/CompositeTask.py | 2 +- pym/portage/util/SlotObject.py | 9 ++++++++- pym/portage/util/_async/AsyncScheduler.py | 7 +++++++ pym/portage/util/_async/AsyncTaskFuture.py | 2 +- 7 files changed, 20 insertions(+), 7 deletions(-)} Created attachment 528560 [details, diff] AsynchronousTask: disable event loop recursion With the patches from bug 653946 and bug 654038, everything appears to be working with event loop recursion disabled in AsynchronousTask, and this patch will make any attempts to trigger recursion provide a helpful InvalidStateError. With early recursion detection like this patch has, it's relatively easy to see what has gone wrong. Comment on attachment 528560 [details, diff] AsynchronousTask: disable event loop recursion (In reply to Zac Medico from comment #4) > Created attachment 528560 [details, diff] [details, diff] > AsynchronousTask: disable event loop recursion Updated and posted for review: https://archives.gentoo.org/gentoo-portage-dev/message/8cfae20921fa805a67799fd258dcc861 https://github.com/gentoo/portage/pull/316 The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=efc4905546f505f69d5b1355aaa0c80e665a6f4a commit efc4905546f505f69d5b1355aaa0c80e665a6f4a Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-28 20:18:06 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-28 20:18:06 +0000 MetadataRegen: fix deprecated _wait usage (bug 653856) Override PollScheduler._cleanup() instead of AsynchronousTask._wait(), since AsynchronousTask._wait() is deprecated. Bug: https://bugs.gentoo.org/653856 pym/_emerge/MetadataRegen.py | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-)} The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=27801746e9aa7e9ffc02408b2b005aa106c0d8a7 commit 27801746e9aa7e9ffc02408b2b005aa106c0d8a7 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-28 21:51:01 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-28 21:55:46 +0000 MirrorDistTask: fix deprecated _wait usage (bug 653856) Override AsynchronousTask._async_wait() for cleanup, since AsynchronousTask._wait() is deprecated. Bug: https://bugs.gentoo.org/653856 pym/portage/_emirrordist/MirrorDistTask.py | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-)} The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=54256075a4b525e86d6837a3ecf6502eb484d3a0 commit 54256075a4b525e86d6837a3ecf6502eb484d3a0 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-28 22:25:21 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-28 22:25:21 +0000 AsynchronousTask: fix deprecated _wait usage (bug 653856) Make the default _start implementation call _async_wait() instead of wait(). Bug: https://bugs.gentoo.org/653856 pym/_emerge/AsynchronousTask.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)} The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=3d81af6440cfd20cd007568779d6129083f1de24 commit 3d81af6440cfd20cd007568779d6129083f1de24 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-28 22:45:56 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-28 22:45:56 +0000 AsyncScheduler: fix deprecated _wait usage (bug 653856) Override AsynchronousTask._async_wait() for cleanup, since AsynchronousTask._wait() is deprecated. Bug: https://bugs.gentoo.org/653856 pym/portage/util/_async/AsyncScheduler.py | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)} The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=98e5852117cbda19b4f4266eb71a053dced64a88 commit 98e5852117cbda19b4f4266eb71a053dced64a88 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-28 23:17:30 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-28 23:17:30 +0000 AsyncTaskFuture: fix deprecated _wait usage (bug 653856) Fix _done_callback to use _async_wait() instead of wait(). Bug: https://bugs.gentoo.org/653856 pym/portage/util/_async/AsyncTaskFuture.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)} The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=532366feaf8e3cb04e0d19d90498cfe0b9361ab1 commit 532366feaf8e3cb04e0d19d90498cfe0b9361ab1 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-28 23:27:49 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-28 23:27:49 +0000 PipeReaderBlockingIO: fix deprecated _wait usage (bug 653856) Use _async_wait() instead of wait(). Bug: https://bugs.gentoo.org/653856 pym/portage/util/_async/PipeReaderBlockingIO.py | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)} The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=01698b9f3f3145b3b902db9f4f5cb07f2f7c737f commit 01698b9f3f3145b3b902db9f4f5cb07f2f7c737f Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-28 23:36:47 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-28 23:36:47 +0000 AsynchronousLock: fix deprecated _wait usage (bug 653856) Use _async_wait() instead of wait(). Bug: https://bugs.gentoo.org/653856 pym/_emerge/AsynchronousLock.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)} The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=a914433a17122776e79456d33c72ecfaee199715 commit a914433a17122776e79456d33c72ecfaee199715 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-28 23:51:03 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-28 23:51:03 +0000 FifoIpcDaemon: fix deprecated _wait usage (bug 653856) Use _async_wait() instead of wait(). Bug: https://bugs.gentoo.org/653856 pym/_emerge/FifoIpcDaemon.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)} The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=5ca431566538965927a33565c222c8994ad288d5 commit 5ca431566538965927a33565c222c8994ad288d5 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-29 00:24:21 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-29 00:24:21 +0000 SubProcess._poll(): fix deprecated wait() usage (bug 653856) Use _async_wait() instead of wait(). Bug: https://bugs.gentoo.org/653856 pym/_emerge/SubProcess.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)} The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=bcbf9ec87252658900f26aa957653a4db718152a commit bcbf9ec87252658900f26aa957653a4db718152a Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-29 02:02:08 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-29 02:04:24 +0000 AbstractPollTask._wait_loop: asyncio compat (bug 653856) Use the asyncio shim for asyncio compatibility. Bug: https://bugs.gentoo.org/653856 pym/_emerge/AbstractPollTask.py | 26 +++++++++++--------------- 1 file changed, 11 insertions(+), 15 deletions(-)} The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=88ce275deabd4b260be916572639de800591a288 commit 88ce275deabd4b260be916572639de800591a288 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-29 03:39:46 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-29 03:39:46 +0000 Scheduler._main_loop: asyncio compat (bug 653856) Use create_future, call_later, and run_until_complete for asyncio compatibility. Bug: https://bugs.gentoo.org/653856 pym/_emerge/Scheduler.py | 44 +++++++++++++++++++++----------------------- 1 file changed, 21 insertions(+), 23 deletions(-)} The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=c984e65e743d049c715f66e96bacb4e830922a34 commit c984e65e743d049c715f66e96bacb4e830922a34 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-29 03:52:18 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-29 03:52:18 +0000 AsynchronousTask._async_wait: asyncio compat (bug 653856) Use call_soon for asyncio compatibility. Bug: https://bugs.gentoo.org/653856 pym/_emerge/AsynchronousTask.py | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)} The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=123aee71d3e15c2b8b2d354902d83a8f8596fbeb commit 123aee71d3e15c2b8b2d354902d83a8f8596fbeb Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-29 04:01:46 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-29 04:01:46 +0000 _LockThread._run_lock: asyncio compat (bug 653856) Use call_soon_threadsafe for asyncio compatibility. Bug: https://bugs.gentoo.org/653856 pym/_emerge/AsynchronousLock.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)} The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=2ab6eed08c9eafa29de6100b19c9c072b17b9167 commit 2ab6eed08c9eafa29de6100b19c9c072b17b9167 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-29 04:06:25 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-29 04:06:25 +0000 PipeReaderBlockingIO._reader_thread: asyncio compat (bug 653856) Use call_soon_threadsafe for asyncio compatibility, and also remove idle_add from SchedulerInterface since there are no more consumers. Bug: https://bugs.gentoo.org/653856 pym/portage/util/_async/PipeReaderBlockingIO.py | 3 +-- pym/portage/util/_async/SchedulerInterface.py | 3 +-- 2 files changed, 2 insertions(+), 4 deletions(-)} The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=f01f504fc8d4ea7442379482889b1cee4be0fcf4 commit f01f504fc8d4ea7442379482889b1cee4be0fcf4 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-29 01:28:42 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-29 04:31:45 +0000 AsynchronousTask: remove unused _wait() methods (bug 653856) Remove _wait() and _waitpid_loop() methods which are unused since AsynchronousTask.wait() now uses run_until_complete(self.async_wait()) instead. Also remove the iteration method from SchedulerInterface, since there are no more consumers. Bug: https://bugs.gentoo.org/653856 pym/_emerge/AbstractEbuildProcess.py | 27 ------------ pym/_emerge/AbstractPollTask.py | 8 +--- pym/_emerge/AsynchronousLock.py | 9 ---- pym/_emerge/AsynchronousTask.py | 6 --- pym/_emerge/CompositeTask.py | 48 +-------------------- pym/_emerge/FifoIpcDaemon.py | 13 +----- pym/_emerge/PipeReader.py | 7 ---- pym/_emerge/SpawnProcess.py | 12 +----- pym/_emerge/SubProcess.py | 56 +------------------------ pym/portage/_emirrordist/MirrorDistTask.py | 7 ---- pym/portage/util/_async/AsyncScheduler.py | 25 +---------- pym/portage/util/_async/AsyncTaskFuture.py | 8 ---- pym/portage/util/_async/PipeLogger.py | 7 ---- pym/portage/util/_async/PipeReaderBlockingIO.py | 10 ----- pym/portage/util/_async/SchedulerInterface.py | 1 - 15 files changed, 6 insertions(+), 238 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=bca4f6a58512471cdf1caf644cee9858ca3bd1eb commit bca4f6a58512471cdf1caf644cee9858ca3bd1eb Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-26 09:51:43 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-29 04:21:56 +0000 AsynchronousTask: disable event loop recursion (bug 653856) Make the wait() and _async_wait() methods raise InvalidStateError when the event loop is running and the returncode is not available, since these cases would trigger event loop recursion. There are no known remaining cases that cause event loop recursion via wait() and _async_wait(), and this patch protects against changes that would accidentally re-introduce event loop recursion. Since the wait() method now raises InvalidStateError in cases where it previously would have called the _wait() method, this patch makes it possible to remove the _wait() method implementations from all subclasses of AsynchronousTask. Subclasses that used the _wait() method to perform cleanup now use the _async_wait() method instead. Bug: https://bugs.gentoo.org/653856 pym/_emerge/AbstractEbuildProcess.py | 5 ++++- pym/_emerge/AsynchronousTask.py | 24 ++++++++++++++---------- 2 files changed, 18 insertions(+), 11 deletions(-)} Fixed in portage-2.3.40-r1. |