Bug 758740 – sys-apps/portage: use default asyncio event loop implementation in child processes

• Product: Portage Development
• Status: RESOLVED FIXED

[Description Zac Medico 2020-12-06 08:40:10 UTC]

It looks like we can workaround https://bugs.python.org/issue22087 by instantiating a new event loop in the subprocess, which is supported since https://bugs.python.org/issue29703 was fixed:

> diff --git a/lib/portage/__init__.py b/lib/portage/__init__.py
> index 4d4b590a8..15cad251f 100644
> --- a/lib/portage/__init__.py
> +++ b/lib/portage/__init__.py
> @@ -9,6 +9,7 @@ VERSION = "HEAD"
>  # ===========================================================================
>  
>  try:
> +	import asyncio
>  	import sys
>  	import errno
>  	if not hasattr(errno, 'ESTALE'):
> @@ -373,6 +374,8 @@ class _ForkWatcher:
>  	@staticmethod
>  	def hook(_ForkWatcher):
>  		_ForkWatcher.current_pid = _os.getpid()
> +		# Workaround https://bugs.python.org/issue22087.
> +		asyncio.set_event_loop_policy(asyncio.DefaultEventLoopPolicy())
>  
>  _ForkWatcher.hook(_ForkWatcher)
>  
> diff --git a/lib/portage/util/_eventloop/global_event_loop.py b/lib/portage/util/_eventloop/global_event_loop.py
> index 21a1d1970..413011178 100644
> --- a/lib/portage/util/_eventloop/global_event_loop.py
> +++ b/lib/portage/util/_eventloop/global_event_loop.py
> @@ -2,11 +2,8 @@
>  # Distributed under the terms of the GNU General Public License v2
>  
>  import portage
> -from .EventLoop import EventLoop
>  from portage.util._eventloop.asyncio_event_loop import AsyncioEventLoop
>  
> -
> -_MAIN_PID = portage.getpid()
>  _instances = {}
>  
>  
> @@ -22,10 +19,6 @@ def global_event_loop():
>  		return instance
>  
>  	constructor = AsyncioEventLoop
> -	# If the default constructor doesn't support multiprocessing,
> -	# then multiprocessing constructor is used in subprocesses.
> -	if not constructor.supports_multiprocessing and pid != _MAIN_PID:
> -		constructor = EventLoop
>  
>  	# Use the _asyncio_wrapper attribute, so that unit tests can compare
>  	# the reference to one retured from _wrap_loop(), since they should

[Comment 1 Zac Medico 2020-12-06 10:10:26 UTC]

Patch posted for review:

https://archives.gentoo.org/gentoo-portage-dev/message/99c01f37b89ab98d16a0ef0f1e2164f9
https://github.com/gentoo/portage/pull/645

[Comment 2 Zac Medico 2020-12-07 01:07:21 UTC]

Python v3.6.1 was the first version with the fix for https://bugs.python.org/issue29703:

https://github.com/python/cpython/commit/01e5230ef0b28658cf7311be199363eda98808bd

Since 3.6.3-r1 was the first stable 3.6.x in gentoo (bug 639070), for portage dependencies it's reasonable ignore the fact that the bug existed in v3.6.0.

[Comment 3 Larry the Git Cow 2020-12-07 02:31:39 UTC]

The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=58d8e4f6a917e262596108eb7bed1c2ef981e9de

commit 58d8e4f6a917e262596108eb7bed1c2ef981e9de
Author:     Zac Medico <[e-mail]>
AuthorDate: 2020-12-06 08:44:20 +0000
Commit:     Zac Medico <[e-mail]>
CommitDate: 2020-12-07 02:28:28 +0000

    Use default asyncio event loop implementation in child processes
    
    Use the default asyncio event loop implementation in child
    processes, instead of portage's internal EventLoop. After
    fork, force instantiation of a new event loop policy as
    a workaround for https://bugs.python.org/issue22087, which
    is necessary for RetryTestCase to succeed.
    
    Bug: https://bugs.gentoo.org/758740
    Signed-off-by: Zac Medico <[e-mail]>

 lib/portage/__init__.py                          | 4 ++++
 lib/portage/util/_eventloop/global_event_loop.py | 7 -------
 2 files changed, 4 insertions(+), 7 deletions(-)

[Comment 4 Larry the Git Cow 2020-12-07 08:49:27 UTC]

The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6a1a30033dd0e79e4678a5a6a8ce8d3ceddf265c

commit 6a1a30033dd0e79e4678a5a6a8ce8d3ceddf265c
Author:     Zac Medico <[e-mail]>
AuthorDate: 2020-12-07 08:13:48 +0000
Commit:     Zac Medico <[e-mail]>
CommitDate: 2020-12-07 08:48:09 +0000

    sys-apps/portage: Bump to version 3.0.12
    
     #758740 Use default asyncio event loop in child processes
     #758755 Use default asyncio event loop in API consumer threads
    
    Bug: https://bugs.gentoo.org/758740
    Bug: https://bugs.gentoo.org/758755
    Bug: https://bugs.gentoo.org/756028
    Package-Manager: Portage-3.0.12, Repoman-3.0.2
    Signed-off-by: Zac Medico <[e-mail]>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-3.0.12.ebuild | 267 +++++++++++++++++++++++++++++++++
 2 files changed, 268 insertions(+)

[Comment 5 Zac Medico 2024-08-18 05:28:51 UTC]

(In reply to Zac Medico from comment #0)
> > +		# Workaround https://bugs.python.org/issue22087.
> > +		asyncio.set_event_loop_policy(asyncio.DefaultEventLoopPolicy())

We can skip this set_event_loop_policy call when sys.version_info >= (3, 12) since it was fixed in v3.12.0a2-134-g191708c56cf here:

https://github.com/python/cpython/pull/99769