Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 653856 - sys-apps/portage: eliminate EventLoop.iteration() calls for asyncio compat
Summary: sys-apps/portage: eliminate EventLoop.iteration() calls for asyncio compat
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on: 653810 653946 654038
Blocks: 591760 651804
  Show dependency tree
 
Reported: 2018-04-23 03:49 UTC by Zac Medico
Modified: 2018-07-02 18:52 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments
AsynchronousTask: disable event loop recursion (0001-AsynchronousTask-disable-event-loop-recursion-bug-65.patch,2.04 KB, patch)
2018-04-26 10:01 UTC, Zac Medico
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Zac Medico gentoo-dev 2018-04-23 03:49:46 UTC
For compatibility with asyncio's default event loop, we'll have to eliminate all of the EventLoop.iteration() calls. With the patch series from bug 653810, all unit tests now pass with event loop recursion disabled. Since the EventLoop.iteration() calls are only triggered *during* event loop recursion, it should be fairly easy to eliminate them:

> pym/_emerge/AbstractPollTask.py:                                self.scheduler.iteration()
> pym/_emerge/AbstractPollTask.py:                                self.scheduler.iteration()
> pym/_emerge/CompositeTask.py:                                           self.scheduler.iteration()
> pym/_emerge/CompositeTask.py:                           self.scheduler.iteration()
> pym/_emerge/Scheduler.py:                               self._event_loop.iteration()
> pym/_emerge/Scheduler.py:                               self._event_loop.iteration()
> pym/_emerge/SubProcess.py:                              self.scheduler.iteration()
> pym/portage/util/_async/AsyncScheduler.py:                      self._event_loop.iteration()
> pym/portage/util/_async/AsyncScheduler.py:                      self._event_loop.iteration()
Comment 1 Zac Medico gentoo-dev 2018-04-23 15:11:10 UTC
The synchronous AsynchronousTask.wait() method should really be eliminated, and replaced with an asynchronous method that returns a Future. Than way, any code that needs to run the event loop will need to explicitly call loop.run_until_complete(future).
Comment 2 Zac Medico gentoo-dev 2018-04-24 03:34:54 UTC
(In reply to Zac Medico from comment #1)
> The synchronous AsynchronousTask.wait() method should really be eliminated,
> and replaced with an asynchronous method that returns a Future. Than way,
> any code that needs to run the event loop will need to explicitly call
> loop.run_until_complete(future).

I've post a patch series for review, which will add an AsynchronousTask.async_wait() method to replace the synchronous AsynchronousTask.wait() method:

https://archives.gentoo.org/gentoo-portage-dev/message/27cb02d75ea82b8f30c1905dbd8b55aa
https://github.com/gentoo/portage/pull/312
Comment 3 Larry the Git Cow gentoo-dev 2018-04-26 03:45:36 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=71a5a82313226f7be0d966d49392a53139a96f6b

commit 71a5a82313226f7be0d966d49392a53139a96f6b
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-04-24 02:47:11 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-04-26 03:19:22 +0000

    AsynchronousTask: add async_wait() method (bug 653856)
    
    Since the AsynchronousTask.wait() method is prone to event loop
    recursion, deprecate it, and add an async_wait() method method to
    replace it. Instead of using task.wait() in order to implicitly run
    the event loop, now loop.run_until_complete(task.async_wait()) will
    be used to explicitly run the event loop. This explicit approach will
    make it more obvious when code will trigger event loop recursion
    which would not be compatible with asyncio's default event loop.
    
    Bug: https://bugs.gentoo.org/653856

 pym/_emerge/AsynchronousTask.py             | 23 +++++++++++++++++++++++
 pym/portage/tests/ebuild/test_ipc_daemon.py |  2 +-
 2 files changed, 24 insertions(+), 1 deletion(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=6c865d6b3130de63dc3d534d2a0596dd59d5e662

commit 6c865d6b3130de63dc3d534d2a0596dd59d5e662
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-04-24 02:21:59 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-04-26 03:19:21 +0000

    AsynchronousTask: add scheduler attribute (bug 653856)
    
    Add an AsynchronousTask.scheduler attribute, which will hold an event
    loop instance in order to work with Future instances. Name the
    attribute "scheduler" because this is the name used by many existing
    subclasses of AsynchronousTask. Since the AsyncScheduler class already
    has an _event_loop attribute that it inherits from PollScheduler, use
    a property for compatibility. Also update the SlotObject constructor
    to allow a subclass to override an attribute from __slots__ with a
    property, so that AsyncScheduler can use a property for the scheduler
    attribute.
    
    Bug: https://bugs.gentoo.org/653856

 pym/_emerge/AbstractPollTask.py            | 3 +--
 pym/_emerge/AsynchronousLock.py            | 2 +-
 pym/_emerge/AsynchronousTask.py            | 2 +-
 pym/_emerge/CompositeTask.py               | 2 +-
 pym/portage/util/SlotObject.py             | 9 ++++++++-
 pym/portage/util/_async/AsyncScheduler.py  | 7 +++++++
 pym/portage/util/_async/AsyncTaskFuture.py | 2 +-
 7 files changed, 20 insertions(+), 7 deletions(-)}
Comment 4 Zac Medico gentoo-dev 2018-04-26 10:01:33 UTC
Created attachment 528560 [details, diff]
AsynchronousTask: disable event loop recursion

With the patches from bug 653946 and bug 654038, everything appears to be working with event loop recursion disabled in AsynchronousTask, and this patch will make any attempts to trigger recursion provide a helpful InvalidStateError. With early recursion detection like this patch has, it's relatively easy to see what has gone wrong.
Comment 5 Zac Medico gentoo-dev 2018-04-28 14:46:54 UTC
Comment on attachment 528560 [details, diff]
AsynchronousTask: disable event loop recursion

(In reply to Zac Medico from comment #4)
> Created attachment 528560 [details, diff] [details, diff]
> AsynchronousTask: disable event loop recursion

Updated and posted for review:

https://archives.gentoo.org/gentoo-portage-dev/message/8cfae20921fa805a67799fd258dcc861
https://github.com/gentoo/portage/pull/316
Comment 6 Larry the Git Cow gentoo-dev 2018-04-28 20:19:46 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=efc4905546f505f69d5b1355aaa0c80e665a6f4a

commit efc4905546f505f69d5b1355aaa0c80e665a6f4a
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-04-28 20:18:06 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-04-28 20:18:06 +0000

    MetadataRegen: fix deprecated _wait usage (bug 653856)
    
    Override PollScheduler._cleanup() instead of AsynchronousTask._wait(),
    since AsynchronousTask._wait() is deprecated.
    
    Bug: https://bugs.gentoo.org/653856

 pym/_emerge/MetadataRegen.py | 14 +++++---------
 1 file changed, 5 insertions(+), 9 deletions(-)}
Comment 7 Larry the Git Cow gentoo-dev 2018-04-28 21:57:05 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=27801746e9aa7e9ffc02408b2b005aa106c0d8a7

commit 27801746e9aa7e9ffc02408b2b005aa106c0d8a7
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-04-28 21:51:01 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-04-28 21:55:46 +0000

    MirrorDistTask: fix deprecated _wait usage (bug 653856)
    
    Override AsynchronousTask._async_wait() for cleanup, since
    AsynchronousTask._wait() is deprecated.
    
    Bug: https://bugs.gentoo.org/653856

 pym/portage/_emirrordist/MirrorDistTask.py | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)}
Comment 8 Larry the Git Cow gentoo-dev 2018-04-28 22:26:15 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=54256075a4b525e86d6837a3ecf6502eb484d3a0

commit 54256075a4b525e86d6837a3ecf6502eb484d3a0
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-04-28 22:25:21 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-04-28 22:25:21 +0000

    AsynchronousTask: fix deprecated _wait usage (bug 653856)
    
    Make the default _start implementation call _async_wait() instead
    of wait().
    
    Bug: https://bugs.gentoo.org/653856

 pym/_emerge/AsynchronousTask.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)}
Comment 9 Larry the Git Cow gentoo-dev 2018-04-28 23:08:25 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=3d81af6440cfd20cd007568779d6129083f1de24

commit 3d81af6440cfd20cd007568779d6129083f1de24
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-04-28 22:45:56 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-04-28 22:45:56 +0000

    AsyncScheduler: fix deprecated _wait usage (bug 653856)
    
    Override AsynchronousTask._async_wait() for cleanup, since
    AsynchronousTask._wait() is deprecated.
    
    Bug: https://bugs.gentoo.org/653856

 pym/portage/util/_async/AsyncScheduler.py | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)}
Comment 10 Larry the Git Cow gentoo-dev 2018-04-28 23:20:15 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=98e5852117cbda19b4f4266eb71a053dced64a88

commit 98e5852117cbda19b4f4266eb71a053dced64a88
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-04-28 23:17:30 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-04-28 23:17:30 +0000

    AsyncTaskFuture: fix deprecated _wait usage (bug 653856)
    
    Fix _done_callback to use _async_wait() instead of wait().
    
    Bug: https://bugs.gentoo.org/653856

 pym/portage/util/_async/AsyncTaskFuture.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)}
Comment 11 Larry the Git Cow gentoo-dev 2018-04-28 23:29:04 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=532366feaf8e3cb04e0d19d90498cfe0b9361ab1

commit 532366feaf8e3cb04e0d19d90498cfe0b9361ab1
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-04-28 23:27:49 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-04-28 23:27:49 +0000

    PipeReaderBlockingIO: fix deprecated _wait usage (bug 653856)
    
    Use _async_wait() instead of wait().
    
    Bug: https://bugs.gentoo.org/653856

 pym/portage/util/_async/PipeReaderBlockingIO.py | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)}
Comment 12 Larry the Git Cow gentoo-dev 2018-04-28 23:41:10 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=01698b9f3f3145b3b902db9f4f5cb07f2f7c737f

commit 01698b9f3f3145b3b902db9f4f5cb07f2f7c737f
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-04-28 23:36:47 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-04-28 23:36:47 +0000

    AsynchronousLock: fix deprecated _wait usage (bug 653856)
    
    Use _async_wait() instead of wait().
    
    Bug: https://bugs.gentoo.org/653856

 pym/_emerge/AsynchronousLock.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)}
Comment 13 Larry the Git Cow gentoo-dev 2018-04-28 23:52:46 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=a914433a17122776e79456d33c72ecfaee199715

commit a914433a17122776e79456d33c72ecfaee199715
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-04-28 23:51:03 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-04-28 23:51:03 +0000

    FifoIpcDaemon: fix deprecated _wait usage (bug 653856)
    
    Use _async_wait() instead of wait().
    
    Bug: https://bugs.gentoo.org/653856

 pym/_emerge/FifoIpcDaemon.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)}
Comment 14 Larry the Git Cow gentoo-dev 2018-04-29 00:26:55 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=5ca431566538965927a33565c222c8994ad288d5

commit 5ca431566538965927a33565c222c8994ad288d5
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-04-29 00:24:21 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-04-29 00:24:21 +0000

    SubProcess._poll(): fix deprecated wait() usage (bug 653856)
    
    Use _async_wait() instead of wait().
    
    Bug: https://bugs.gentoo.org/653856

 pym/_emerge/SubProcess.py | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)}
Comment 15 Larry the Git Cow gentoo-dev 2018-04-29 02:11:20 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=bcbf9ec87252658900f26aa957653a4db718152a

commit bcbf9ec87252658900f26aa957653a4db718152a
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-04-29 02:02:08 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-04-29 02:04:24 +0000

    AbstractPollTask._wait_loop: asyncio compat (bug 653856)
    
    Use the asyncio shim for asyncio compatibility.
    
    Bug: https://bugs.gentoo.org/653856

 pym/_emerge/AbstractPollTask.py | 26 +++++++++++---------------
 1 file changed, 11 insertions(+), 15 deletions(-)}
Comment 16 Larry the Git Cow gentoo-dev 2018-04-29 03:47:20 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=88ce275deabd4b260be916572639de800591a288

commit 88ce275deabd4b260be916572639de800591a288
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-04-29 03:39:46 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-04-29 03:39:46 +0000

    Scheduler._main_loop: asyncio compat (bug 653856)
    
    Use create_future, call_later, and run_until_complete for
    asyncio compatibility.
    
    Bug: https://bugs.gentoo.org/653856

 pym/_emerge/Scheduler.py | 44 +++++++++++++++++++++-----------------------
 1 file changed, 21 insertions(+), 23 deletions(-)}
Comment 17 Larry the Git Cow gentoo-dev 2018-04-29 03:55:15 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=c984e65e743d049c715f66e96bacb4e830922a34

commit c984e65e743d049c715f66e96bacb4e830922a34
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-04-29 03:52:18 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-04-29 03:52:18 +0000

    AsynchronousTask._async_wait: asyncio compat (bug 653856)
    
    Use call_soon for asyncio compatibility.
    
    Bug: https://bugs.gentoo.org/653856

 pym/_emerge/AsynchronousTask.py | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)}
Comment 18 Larry the Git Cow gentoo-dev 2018-04-29 04:02:25 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=123aee71d3e15c2b8b2d354902d83a8f8596fbeb

commit 123aee71d3e15c2b8b2d354902d83a8f8596fbeb
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-04-29 04:01:46 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-04-29 04:01:46 +0000

    _LockThread._run_lock: asyncio compat (bug 653856)
    
    Use call_soon_threadsafe for asyncio compatibility.
    
    Bug: https://bugs.gentoo.org/653856

 pym/_emerge/AsynchronousLock.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)}
Comment 19 Larry the Git Cow gentoo-dev 2018-04-29 04:12:43 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=2ab6eed08c9eafa29de6100b19c9c072b17b9167

commit 2ab6eed08c9eafa29de6100b19c9c072b17b9167
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-04-29 04:06:25 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-04-29 04:06:25 +0000

    PipeReaderBlockingIO._reader_thread: asyncio compat (bug 653856)
    
    Use call_soon_threadsafe for asyncio compatibility, and also
    remove idle_add from SchedulerInterface since there are no more
    consumers.
    
    Bug: https://bugs.gentoo.org/653856

 pym/portage/util/_async/PipeReaderBlockingIO.py | 3 +--
 pym/portage/util/_async/SchedulerInterface.py   | 3 +--
 2 files changed, 2 insertions(+), 4 deletions(-)}
Comment 20 Larry the Git Cow gentoo-dev 2018-04-29 04:38:19 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=f01f504fc8d4ea7442379482889b1cee4be0fcf4

commit f01f504fc8d4ea7442379482889b1cee4be0fcf4
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-04-29 01:28:42 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-04-29 04:31:45 +0000

    AsynchronousTask: remove unused _wait() methods (bug 653856)
    
    Remove _wait() and _waitpid_loop() methods which are unused since
    AsynchronousTask.wait() now uses run_until_complete(self.async_wait())
    instead. Also remove the iteration method from SchedulerInterface,
    since there are no more consumers.
    
    Bug: https://bugs.gentoo.org/653856

 pym/_emerge/AbstractEbuildProcess.py            | 27 ------------
 pym/_emerge/AbstractPollTask.py                 |  8 +---
 pym/_emerge/AsynchronousLock.py                 |  9 ----
 pym/_emerge/AsynchronousTask.py                 |  6 ---
 pym/_emerge/CompositeTask.py                    | 48 +--------------------
 pym/_emerge/FifoIpcDaemon.py                    | 13 +-----
 pym/_emerge/PipeReader.py                       |  7 ----
 pym/_emerge/SpawnProcess.py                     | 12 +-----
 pym/_emerge/SubProcess.py                       | 56 +------------------------
 pym/portage/_emirrordist/MirrorDistTask.py      |  7 ----
 pym/portage/util/_async/AsyncScheduler.py       | 25 +----------
 pym/portage/util/_async/AsyncTaskFuture.py      |  8 ----
 pym/portage/util/_async/PipeLogger.py           |  7 ----
 pym/portage/util/_async/PipeReaderBlockingIO.py | 10 -----
 pym/portage/util/_async/SchedulerInterface.py   |  1 -
 15 files changed, 6 insertions(+), 238 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=bca4f6a58512471cdf1caf644cee9858ca3bd1eb

commit bca4f6a58512471cdf1caf644cee9858ca3bd1eb
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-04-26 09:51:43 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-04-29 04:21:56 +0000

    AsynchronousTask: disable event loop recursion (bug 653856)
    
    Make the wait() and _async_wait() methods raise InvalidStateError
    when the event loop is running and the returncode is not available,
    since these cases would trigger event loop recursion. There are no
    known remaining cases that cause event loop recursion via wait()
    and _async_wait(), and this patch protects against changes that would
    accidentally re-introduce event loop recursion.
    
    Since the wait() method now raises InvalidStateError in cases where
    it previously would have called the _wait() method, this patch makes
    it possible to remove the _wait() method implementations from all
    subclasses of AsynchronousTask. Subclasses that used the _wait()
    method to perform cleanup now use the _async_wait() method instead.
    
    Bug: https://bugs.gentoo.org/653856

 pym/_emerge/AbstractEbuildProcess.py |  5 ++++-
 pym/_emerge/AsynchronousTask.py      | 24 ++++++++++++++----------
 2 files changed, 18 insertions(+), 11 deletions(-)}
Comment 21 Zac Medico gentoo-dev 2018-07-02 18:52:14 UTC
Fixed in portage-2.3.40-r1.