Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 763339 - sys-apps/portage: global_event_loop should return the running loop for current thread if one exists
Summary: sys-apps/portage: global_event_loop should return the running loop for curren...
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core - Ebuild Support (show other bugs)
Hardware: All All
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on:
Blocks: 761538 764923
  Show dependency tree
 
Reported: 2021-01-03 23:07 UTC by Zac Medico
Modified: 2021-01-25 04:38 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Zac Medico gentoo-dev 2021-01-03 23:07:50 UTC
The global_event_loop function currently returns one event loop instance per pid, but we really want it to behave like asycio.get_event_loop() which returns the running loop for current thread if one exists, and even bypasses the event loop policy:

> def get_event_loop():
>     """Return an asyncio event loop.
> 
>     When called from a coroutine or a callback (e.g. scheduled with call_soon
>     or similar API), this function will always return the running event loop.
> 
>     If there is no running event loop set, the function will return
>     the result of `get_event_loop_policy().get_event_loop()` call.
>     """
>     # NOTE: this function is implemented in C (see _asynciomodule.c)
>     current_loop = _get_running_loop()
>     if current_loop is not None:
>         return current_loop
>     return get_event_loop_policy().get_event_loop()
Comment 2 Zac Medico gentoo-dev 2021-01-04 09:27:46 UTC
For the case of "loop running in non-main thread" of API consumer, this change makes portage compatible with PEP 492 coroutines with async and await syntax.
Comment 3 Zac Medico gentoo-dev 2021-01-11 06:32:17 UTC
With the current patch, unit tests consistently hang with python3.6 and python3.7, but not for python3.8 and later. If I disable this set_wakeup_fd code from bug 655656 then it suppresses the problem:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=adee194534f0b3d9762efd1e8e8713c316b93f5a

I think the problem must be related to thread usage in the child watcher (also see bug 764905). However, I've tried using a child watcher wrapper to deliver the callbacks via call_soon_threadsafe, and that didn't solve the problem
Comment 4 Larry the Git Cow gentoo-dev 2021-01-11 09:51:25 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=386178481eb86ac603cd90ef1bb6ac6b68e51c50

commit 386178481eb86ac603cd90ef1bb6ac6b68e51c50
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2021-01-04 09:14:36 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2021-01-11 09:36:48 +0000

    global_event_loop: return running loop for current thread
    
    Like asyncio.get_event_loop(), return the running loop for the
    current thread if there is one, and otherwise construct a new
    one if needed. This allows the _safe_loop function to become
    synonymous with the global_event_loop function.
    
    For the case of "loop running in non-main thread" of API
    consumer, this change makes portage compatible with PEP
    492 coroutines with async and await syntax. Portage
    internals can safely begin using async / await syntax instead
    of compat_coroutine.
    
    Bug: https://bugs.gentoo.org/763339
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/util/_eventloop/global_event_loop.py | 28 +++-------------------
 lib/portage/util/futures/_asyncio/__init__.py    | 30 +++++++++++++++++-------
 2 files changed, 24 insertions(+), 34 deletions(-)
Comment 5 Larry the Git Cow gentoo-dev 2021-01-11 09:54:11 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=2d9df956e086c0766ee352320ecc346c27d23df3

commit 2d9df956e086c0766ee352320ecc346c27d23df3
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2021-01-11 09:45:19 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2021-01-11 09:47:00 +0000

    sys-apps/portage: Bump to version 3.0.13
    
     #763339 always allow event loops to run in threads
     #764764 fix virtual/dist-kernel slot operator rebuilds
     #764905 wrap asyncio child watcher for thread safety
    
    Bug: https://bugs.gentoo.org/764923
    Bug: https://bugs.gentoo.org/763339
    Bug: https://bugs.gentoo.org/764764
    Bug: https://bugs.gentoo.org/764905
    Package-Manager: Portage-3.0.13, Repoman-3.0.2
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-3.0.13.ebuild | 268 +++++++++++++++++++++++++++++++++
 2 files changed, 269 insertions(+)
Comment 6 Zac Medico gentoo-dev 2021-01-11 10:03:37 UTC
The problem with python3.6 and python3.7 was just that a strong reference was needed for the main loop.
Comment 7 Larry the Git Cow gentoo-dev 2021-01-18 12:20:17 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=8d962cb5cc97a5092ff45446c0f8da55b27d2434

commit 8d962cb5cc97a5092ff45446c0f8da55b27d2434
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2021-01-18 10:09:05 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2021-01-18 11:18:43 +0000

    coroutine: do not require loop parameter
    
    The loop parameter is not needed since global_event_loop now
    returns the running event loop for the current thread.
    
    Bug: https://bugs.gentoo.org/737698
    Bug: https://bugs.gentoo.org/763339
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/util/futures/compat_coroutine.py | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)