Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 916566 - sys-apps/portage: support alternative process cloning implementations in portage.process.spawn
Summary: sys-apps/portage: support alternative process cloning implementations in port...
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All Linux
: Normal normal
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on: 921380 923750 923755 923852 923854 924287
Blocks: 914876
  Show dependency tree
 
Reported: 2023-10-31 07:08 UTC by Zac Medico
Modified: 2024-08-11 18:49 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Zac Medico gentoo-dev 2023-10-31 07:08:39 UTC
Since os.fork() is unsafe in threaded processes (bug 914876), it would be useful for portage.process.spawn to support alternative process cloning implementations.

The process cloning interface can be a function that takes a callable "target" argument, and returns an implementation dependent object. The backward compatible os.fork() implementation of this interface can simply return an int pid number. An implementation based on multiprocessing.Process can return a multiprocessing.Process instance instead of just a pid. It can be a simple function like this:

def multiprocessing_start(target):
    proc = multiprocessing.Process(target=target)
    proc.start()
    return proc

A very naive os.fork() implementation looks something like this (a better implementation would account for bug 345289):

def fork_start(target):
    pid = os.fork()
    if pid == 0:
        os._exit(target())
    return pid

Callers of the portage.process.spawn API can choose the process cloning implementations based on environmental details, such as threading.active_count() and multiprocessing.get_start_method(), in order to avoid mixing threads with os.fork(). Since ForkProcess uses a thread to send fd_pipes for the multiprocessing spawn start method, it would be safest to avoid calling os.fork() when using the multiprocessing spawn start method.
Comment 1 Zac Medico gentoo-dev 2023-11-01 05:36:13 UTC
For the multiprocessing spawn start method, fd_pipes only works via asynchronous send_handle calls, so the existing support for this in ForkProcess can be used here. However, this means that fd_pipes will be accessed by a background thread after the spawn function has returned (can trigger timing issues like bug 916601).
Comment 2 Zac Medico gentoo-dev 2024-01-27 04:38:01 UTC
In the tracker issue for making the GIL optional (https://github.com/python/cpython/issues/108219), it lists this commit from nogil-3.12 as something to merge upstream:

c1befd7689 Stop the world before fork() and Python shutdown

It calls a _PyRuntimeState_StopTheWorld function to stop all other threads just before fork(). However, the stopped threads will still trigger a warning about possible deadlocks in the child, via the warn_about_fork_with_threads function.

(In reply to Zac Medico from comment #0)
> def multiprocessing_start(target):
>     proc = multiprocessing.Process(target=target)
>     proc.start()
>     return proc

Maybe multiprocessing.Process is not the best choice, since it is specialized for execution of python code. We could instead rely on the subprocess module's use of posix_spawn to avoid fork, but posix_spawn will still require use to use send_handle for fd_pipes support.
Comment 3 Larry the Git Cow gentoo-dev 2024-01-29 16:09:01 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=7ec7a647ef6e2d94e6f2b387d0a012e78cafbaff

commit 7ec7a647ef6e2d94e6f2b387d0a012e78cafbaff
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2024-01-29 08:20:55 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2024-01-29 08:43:52 +0000

    process.spawn: add abstraction layer for os.fork()
    
    Since os.fork() is unsafe in threaded processes, add a basic
    abstraction layer that will ultimately allow support of other process
    cloning implementations.
    
    Usage of os.fork() is now wrapped in a _start_fork function which can
    easily be replaced with an alternative implementation.  This function
    takes target, args, and kwargs arguments which are equivalent to
    the corresponding multiprocessing.Process parameters. It also has
    fd_pipes and close_fds parameters, since implementation of these is
    dependent on the process cloning implementation.
    
    The process cloning implementation does not need to be bothered with
    the details of the _exec function, since spawn uses an _exec_wrapper
    function as a target function which calls _exec and handles any
    exceptions appropriately (special exception handling is required for
    success of test_spawnE2big related to bug 830187).
    
    Bug: https://bugs.gentoo.org/916566
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/process.py | 203 +++++++++++++++++++++++++++++++++----------------
 1 file changed, 139 insertions(+), 64 deletions(-)
Comment 4 Zac Medico gentoo-dev 2024-01-30 20:50:26 UTC
If we make all of the process cloning implementations return a asyncio.subprocess.Process like object, this object can provide an interface to wait for the pid that is independent of the process cloning implementation, and can encapsulate implementation-dependent objects like multiprocessing.Process which which are designed to manage the pid lifecycle and need to persist until the pid exits. A spawn keyword argument similar to returnpid which we might called returnproc can be used to make spawn return Process objects instead of raw pids, and we can migrate everything to use this new returnproc argument instead of returnpid.
Comment 5 Larry the Git Cow gentoo-dev 2024-02-02 00:32:10 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=c556fee09b6b103a9bb8c2afa1c7a7ef25022598

commit c556fee09b6b103a9bb8c2afa1c7a7ef25022598
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2024-02-01 05:48:54 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2024-02-01 16:10:28 +0000

    process.spawn: Add returnproc parameter
    
    In order to migrate away from unsafe os.fork() usage in threaded
    processes (https://github.com/python/cpython/issues/84559), add a
    returnproc parameter that is similar to returnpid, which causes
    spawn to return a single Process object instead of a list of pids.
    The Process API is a subset of asyncio.subprocess.Process. The
    returnproc parameter conflicts with the logfile parameter, since
    the caller is expected to use the fd_pipes parameter to implement
    logging (this was also true for the returnpid parameter).
    
    In the future, spawn will return objects of a different type but
    with a compatible interface to Process, in order to encapsulate
    implementation-dependent objects like multiprocessing.Process which
    are designed to manage the process lifecycle and need to persist
    until it exits.
    
    Trigger a UserWarning when the returnpid parameter is used, in
    order to encourage migration to returnproc (do not use
    DeprecationWarning since it is hidden by default). This warning
    will be temporarily suppressed for portage internals, until they
    finish migrating to returnproc. There are probably very few if
    any external consumers of spawn with the returnpid parameter,
    so it seems safe to move quickly with this deprecation.
    
    Bug: https://bugs.gentoo.org/916566
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/process.py                             | 84 ++++++++++++++++++++++
 lib/portage/tests/process/meson.build              |  1 +
 lib/portage/tests/process/test_spawn_returnproc.py | 39 ++++++++++
 3 files changed, 124 insertions(+)
Comment 6 Zac Medico gentoo-dev 2024-02-02 17:59:30 UTC
With portage.process.MultiprocessingProcess added in https://github.com/gentoo/portage/pull/1248, it will be very easy to replace _start_fork with a function that starts a ForkProcess instance and returns the associated MultiprocessingProcess instance.

Then we might just use _start_fork for temporary returnpid backward compatiblity, since ForkProcess will use fork if multiprocessing.get_start_method() == "fork".

We can add a FEATURES setting to make portage programs call multiprocessing.set_start_method("spawn"), or perhaps even do it unconditionally so that there will be no change when https://github.com/python/cpython/issues/84559 makes "spawn" the default multiprocessing start method.
Comment 7 Larry the Git Cow gentoo-dev 2024-02-03 02:03:29 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=055c66ec9482064aaaf51bfb6b01e260ea27808e

commit 055c66ec9482064aaaf51bfb6b01e260ea27808e
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2024-02-02 16:01:18 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2024-02-02 16:01:18 +0000

    SpawnProcess: Use spawn returnproc parameter
    
    Migrate SpawnProcess to use the spawn returnproc parameter,
    and make adaptations to descendent classes as needed. Introduce a
    portage.process.MultiprocessingProcess class for ForkProcess to wrap
    multiprocessing.Process instances, needed because ForkProcess
    inherits from SpawnProcess. Use portage.process.Process to wrap
    the pid in EbuildMetadataPhase, so that returnproc support in
    the doebuild function can be reserved for a later commit.
    
    Bug: https://bugs.gentoo.org/916566
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/EbuildMetadataPhase.py      |   4 +-
 lib/_emerge/SpawnProcess.py             |  16 ++---
 lib/_emerge/SubProcess.py               |  25 +++----
 lib/portage/package/ebuild/doebuild.py  |   4 +-
 lib/portage/process.py                  | 120 ++++++++++++++++++++++++++++----
 lib/portage/util/_async/ForkProcess.py  |  87 +++--------------------
 lib/portage/util/_async/PopenProcess.py |   5 +-
 7 files changed, 143 insertions(+), 118 deletions(-)
Comment 8 Larry the Git Cow gentoo-dev 2024-02-03 22:54:18 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=305612d1b04aa06d3d1a1c8b51d046a644742fd5

commit 305612d1b04aa06d3d1a1c8b51d046a644742fd5
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2024-02-03 21:27:45 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2024-02-03 21:27:45 +0000

    process.spawn: Use multiprocessing.Process for returnproc
    
    Use multiprocessing.Process for returnproc, so that
    fork will stop being used when python makes "spawn"
    the default multiprocessing start method.
    
    Continue to use _start_fork when returnproc is not
    enabled, for backward compatibility. Ultimately,
    it can be removed at the same time as the returnpid
    parameter.
    
    Bug: https://bugs.gentoo.org/916566
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/SpawnProcess.py                        |  4 +-
 lib/portage/process.py                             | 72 ++++++++++++++++++----
 lib/portage/tests/ebuild/test_doebuild_fd_pipes.py |  6 +-
 3 files changed, 67 insertions(+), 15 deletions(-)
Comment 9 Zac Medico gentoo-dev 2024-02-04 00:27:24 UTC
All of the remaining returnpid consumers are eliminated in https://github.com/gentoo/portage/pull/1250, so that basically completes this bug.
Comment 10 Zac Medico gentoo-dev 2024-02-04 00:35:12 UTC
Actually, the returnproc=False case still uses fork, so that needs to be fixed by making it use _start_proc and use the event loop to wait for MultiprocessingProcess objects.
Comment 11 Larry the Git Cow gentoo-dev 2024-02-04 06:00:20 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=6aeee45d9b88d7dc8c450ba86975e4b017db203c

commit 6aeee45d9b88d7dc8c450ba86975e4b017db203c
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2024-02-03 21:24:49 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2024-02-03 22:40:34 +0000

    doebuild: Add returnproc parameter and deprecate returnpid
    
    Raise NotImplementedError if returnproc is enabled for anything
    other than the "depend" phase, since corresponding returnpid
    support has long been deprecated.
    
    Bug: https://bugs.gentoo.org/916566
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/EbuildMetadataPhase.py     |  5 ++--
 lib/portage/package/ebuild/doebuild.py | 47 ++++++++++++++++++++++++++--------
 2 files changed, 40 insertions(+), 12 deletions(-)
Comment 12 Larry the Git Cow gentoo-dev 2024-02-06 02:24:19 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=e8b31c86eaed645a8740fb2844e2935aee161e43

commit e8b31c86eaed645a8740fb2844e2935aee161e43
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2024-02-05 05:55:11 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2024-02-06 01:30:21 +0000

    ForkProcess: Prevent redundant pipe and set_term_size recursion
    
    When process.spawn is updated to call ForkProcess
    for bug 916566, it needs to avoid recursion via
    set_term_size.
    
    Bug: https://bugs.gentoo.org/916566
    Bug: https://bugs.gentoo.org/923750
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/SpawnProcess.py            | 31 ++++++++++++++++++++++++-------
 lib/portage/util/_async/ForkProcess.py | 28 +++++++++++++++++++++++-----
 2 files changed, 47 insertions(+), 12 deletions(-)
Comment 13 Larry the Git Cow gentoo-dev 2024-02-07 00:36:40 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=66d8f8388e5b9da7e07510f78ec487913e2ceaf5

commit 66d8f8388e5b9da7e07510f78ec487913e2ceaf5
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2024-02-04 00:16:29 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2024-02-07 00:36:17 +0000

    ChildWatcherTestCase: Remove obsolete test which uses spawn returnpid
    
    This test was added for bug 649588 when there was still an
    internal event loop implementation for python2. It is no
    longer relevant and uses the deprecated spawn returnpid
    parameter, so remove it.
    
    Bug: https://bugs.gentoo.org/916566
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/tests/util/futures/asyncio/meson.build |  1 -
 .../util/futures/asyncio/test_child_watcher.py     | 50 ----------------------
 2 files changed, 51 deletions(-)
Comment 14 Larry the Git Cow gentoo-dev 2024-02-07 02:35:16 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=3e10368e52ecead86b75478ca448ef0f59333e3e

commit 3e10368e52ecead86b75478ca448ef0f59333e3e
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2024-02-04 04:34:45 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2024-02-07 00:55:46 +0000

    process.spawn: Use _start_proc for returnpid=False
    
    This essentially completes the implementation of bug 916566,
    eliminating os.fork() usage when "spawn" becomes the default
    multiprocessing start method.
    
    Bug: https://bugs.gentoo.org/916566
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/process.py | 35 +++++++++++++++++------------------
 1 file changed, 17 insertions(+), 18 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=1ae4f574f69eca3146137ffb40c5afd8a4872777

commit 1ae4f574f69eca3146137ffb40c5afd8a4872777
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2024-02-04 00:22:08 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2024-02-07 00:55:46 +0000

    process.spawn: Enable returnpid warning for internals
    
    All internal returnpid consumers have been migrated to
    use the new returnproc parameter instead.
    
    Bug: https://bugs.gentoo.org/916566
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/process.py | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=62332ee82b8b88fa5a65aafa7c221ccdaa7d65a8

commit 62332ee82b8b88fa5a65aafa7c221ccdaa7d65a8
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2024-02-04 00:11:07 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2024-02-07 00:55:46 +0000

    RsyncSync: Migrate to spawn returnproc parameter
    
    Bug: https://bugs.gentoo.org/916566
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/sync/modules/rsync/rsync.py       | 40 +++++++++++++--------------
 lib/portage/util/futures/_asyncio/__init__.py |  6 +++-
 2 files changed, 24 insertions(+), 22 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=fa8e8f1895ed889aece2f67725df55d6ccf127fb

commit fa8e8f1895ed889aece2f67725df55d6ccf127fb
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2024-02-03 23:41:45 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2024-02-07 00:55:46 +0000

    socks5: Migrate to spawn returnproc parameter
    
    Bug: https://bugs.gentoo.org/916566
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/util/socks5.py | 36 +++++++++++++++++++-----------------
 1 file changed, 19 insertions(+), 17 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=190e8ad1a238d20d782235dd1faa6b00d1b3fd4a

commit 190e8ad1a238d20d782235dd1faa6b00d1b3fd4a
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2024-02-03 21:24:49 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2024-02-07 00:55:45 +0000

    doebuild: Add returnproc parameter and deprecate returnpid
    
    Raise NotImplementedError if returnproc is enabled for anything
    other than the "depend" phase, since corresponding returnpid
    support has long been deprecated.
    
    Bug: https://bugs.gentoo.org/916566
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/EbuildMetadataPhase.py     |  4 +--
 lib/portage/package/ebuild/doebuild.py | 47 ++++++++++++++++++++++++++--------
 2 files changed, 39 insertions(+), 12 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=a69c1b853a47346192950c91b088163490287350

commit a69c1b853a47346192950c91b088163490287350
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2024-02-03 21:27:45 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2024-02-07 00:49:26 +0000

    process.spawn: Use multiprocessing.Process for returnproc
    
    Use multiprocessing.Process for returnproc, so that
    fork will stop being used when python makes "spawn"
    the default multiprocessing start method.
    
    Continue to use _start_fork when returnproc is not
    enabled, for backward compatibility. Ultimately,
    it can be removed at the same time as the returnpid
    parameter.
    
    The _setup_pipes_after_fork wrapper prevents a
    "Bad file descriptor" error by making fd_pipes
    inheritable on exec for bug 923755. ForkProcess
    does not handle this because its target function
    does not necessarily exec.
    
    Bug: https://bugs.gentoo.org/916566
    Bug: https://bugs.gentoo.org/923755
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/SpawnProcess.py |  4 +-
 lib/portage/process.py      | 92 ++++++++++++++++++++++++++++++++++++++-------
 2 files changed, 82 insertions(+), 14 deletions(-)
Comment 15 Larry the Git Cow gentoo-dev 2024-02-12 07:58:07 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=419cce79f9082308c848df0a98f367de4d1c50a3

commit 419cce79f9082308c848df0a98f367de4d1c50a3
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2024-02-11 21:58:10 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2024-02-12 07:56:10 +0000

    process._exec: Use _start_fork for os.fork() error handling
    
    Use _start_fork for os.fork() error handling, ensuring
    that if exec fails then the child process will display
    a traceback before it exits via os._exit to suppress any
    finally blocks from parent's call stack (bug 345289).
    
    Bug: https://bugs.gentoo.org/345289
    Bug: https://bugs.gentoo.org/916566
    Bug: https://bugs.gentoo.org/924313
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/process.py | 259 ++++++++++++++++++++++++++++---------------------
 1 file changed, 151 insertions(+), 108 deletions(-)
Comment 16 Zac Medico gentoo-dev 2024-02-14 17:09:35 UTC
Sometimes lib/portage/tests/process/test_spawn_returnproc.py fails because sleep  exits with status 1 even though it was killed by SIGTERM:

https://github.com/gentoo/portage/actions/runs/7904644492/job/21575430214?pr=1270

> =================================== FAILURES ===================================
> _____________ SpawnReturnProcTestCase.testSpawnReturnProcTerminate _____________
> [gw1] linux -- Python 3.12.2 /opt/hostedtoolcache/Python/3.12.2/x64/bin/python
> 
> self = <portage.tests.process.test_spawn_returnproc.SpawnReturnProcTestCase testMethod=testSpawnReturnProcTerminate>
> 
>     def testSpawnReturnProcTerminate(self):
>         sleep_binary = find_binary("sleep")
>         self.assertNotEqual(sleep_binary, None)
>     
>         loop = global_event_loop()
>     
>         async def watch_pid():
>             proc = spawn([sleep_binary, 9999], returnproc=True)
>             proc.terminate()
>             self.assertEqual(await proc.wait(), -signal.SIGTERM)
>     
> >       loop.run_until_complete(watch_pid())
> 
> loop       = <portage.util._eventloop.asyncio_event_loop.AsyncioEventLoop object at 0x7f020f707e90>
> self       = <portage.tests.process.test_spawn_returnproc.SpawnReturnProcTestCase testMethod=testSpawnReturnProcTerminate>
> sleep_binary = '/usr/bin/sleep'
> watch_pid  = <function SpawnReturnProcTestCase.testSpawnReturnProcTerminate.<locals>.watch_pid at 0x7f020f396340>
Comment 17 Zac Medico gentoo-dev 2024-02-14 17:22:20 UTC
(In reply to Zac Medico from comment #16)
> Sometimes lib/portage/tests/process/test_spawn_returnproc.py fails because
> sleep exits with status 1 even though it was killed by SIGTERM:

Maybe it was killed before the exec to sleep. In ForkProcess._bootstrap there's this SIGTERM setup related to bug 353239:

>     @staticmethod
>     def _bootstrap(child_connection, have_send_handle, fd_pipes, target, args, kwargs):
>         # Use default signal handlers in order to avoid problems
>         # killing subprocesses as reported in bug #353239.
>         signal.signal(signal.SIGINT, signal.SIG_DFL)
>         signal.signal(signal.SIGTERM, signal.SIG_DFL)
Comment 18 Larry the Git Cow gentoo-dev 2024-02-22 07:24:11 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=77c44c46194922509bc4f2b5cfc099412a560a69

commit 77c44c46194922509bc4f2b5cfc099412a560a69
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2024-02-22 07:23:40 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2024-02-22 07:23:50 +0000

    sys-apps/portage: add 3.0.62
    
    Closes: https://bugs.gentoo.org/663324
    Closes: https://bugs.gentoo.org/728046
    Closes: https://bugs.gentoo.org/891137
    Closes: https://bugs.gentoo.org/906368
    Closes: https://bugs.gentoo.org/916566
    Closes: https://bugs.gentoo.org/921170
    Closes: https://bugs.gentoo.org/921208
    Closes: https://bugs.gentoo.org/921400
    Closes: https://bugs.gentoo.org/922038
    Closes: https://bugs.gentoo.org/922142
    Closes: https://bugs.gentoo.org/923368
    Closes: https://bugs.gentoo.org/923750
    Closes: https://bugs.gentoo.org/923841
    Closes: https://bugs.gentoo.org/923852
    Closes: https://bugs.gentoo.org/923854
    Closes: https://bugs.gentoo.org/924192
    Closes: https://bugs.gentoo.org/924273
    Closes: https://bugs.gentoo.org/924585
    Closes: https://bugs.gentoo.org/921380
    Signed-off-by: Sam James <sam@gentoo.org>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-3.0.62.ebuild | 246 +++++++++++++++++++++++++++++++++
 2 files changed, 247 insertions(+)
Comment 19 Larry the Git Cow gentoo-dev 2024-02-23 08:02:51 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=3cc986f87ddda86ee93770e03cca06346aee54c5

commit 3cc986f87ddda86ee93770e03cca06346aee54c5
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2024-02-23 06:06:14 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2024-02-23 06:48:29 +0000

    AsyncioEventLoop: Call process.run_exitfuncs() before close
    
    For the event loop running in the main thread, call
    process.run_exitfuncs() before close with the event loop
    running so that anything attached can clean itself up (like
    the socks5 ProxyManager for bug 925240). This is necessary
    because process.spawn uses the event loop to implement the
    new returnproc parameter related to bug 916566.
    
    Bug: https://bugs.gentoo.org/916566
    Bug: https://bugs.gentoo.org/925240
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/tests/util/test_socks5.py             | 51 ++++++++++++++++++++++-
 lib/portage/util/_eventloop/asyncio_event_loop.py | 44 +++++++++++++++----
 lib/portage/util/socks5.py                        | 16 ++++++-
 3 files changed, 101 insertions(+), 10 deletions(-)
Comment 20 Zac Medico gentoo-dev 2024-02-25 00:27:22 UTC
(In reply to Zac Medico from comment #17)
> (In reply to Zac Medico from comment #16)
> > Sometimes lib/portage/tests/process/test_spawn_returnproc.py fails because
> > sleep exits with status 1 even though it was killed by SIGTERM:
> 
> Maybe it was killed before the exec to sleep. In ForkProcess._bootstrap
> there's this SIGTERM setup related to bug 353239:

Actually the test log shows an unexpected TypeError which I also triggered with https://github.com/gentoo/portage/pull/1283:

----------------------------- Captured stderr call -----------------------------
Process ForkProcess-23:
Traceback (most recent call last):
  File "/home/runner/work/portage/portage/lib/portage/process.py", line 818, in _exec_wrapper
    _exec(
  File "/home/runner/work/portage/portage/lib/portage/process.py", line 1068, in _exec
    _exec2(
  File "/home/runner/work/portage/portage/lib/portage/process.py", line 1166, in _exec2
    os.execve(binary, myargs, env)
TypeError: expected str, bytes or os.PathLike object, not int

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/runner/work/portage/portage/lib/portage/util/_async/ForkProcess.py", line 328, in _bootstrap
    sys.exit(target(*(args or []), **(kwargs or {})))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/portage/portage/lib/portage/process.py", line 1441, in __call__
    return self._target(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/portage/portage/lib/portage/process.py", line 853, in _exec_wrapper
    writemsg(f"{e}:\n   {' '.join(mycommand)}\n", noiselevel=-1)
                         ^^^^^^^^^^^^^^^^^^^
TypeError: sequence item 1: expected str instance, int found
Comment 21 Larry the Git Cow gentoo-dev 2024-02-25 00:45:11 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=9e84ef57ba747766c9147c1ac1b247faa1f05956

commit 9e84ef57ba747766c9147c1ac1b247faa1f05956
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2024-02-25 00:31:13 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2024-02-25 00:42:46 +0000

    testSpawnReturnProcTerminate: Fix integer in spawn command argument
    
    The invalid integer in the spawn command argument intermittently
    triggered this error when SIGTERM did not arrive until after the
    exec call:
    
    ----------------------------- Captured stderr call -----------------------------
    Process ForkProcess-23:
    Traceback (most recent call last):
      File "/home/runner/work/portage/portage/lib/portage/process.py", line 818, in _exec_wrapper
        _exec(
      File "/home/runner/work/portage/portage/lib/portage/process.py", line 1068, in _exec
        _exec2(
      File "/home/runner/work/portage/portage/lib/portage/process.py", line 1166, in _exec2
        os.execve(binary, myargs, env)
    TypeError: expected str, bytes or os.PathLike object, not int
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
        self.run()
      File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/multiprocessing/process.py", line 108, in run
        self._target(*self._args, **self._kwargs)
      File "/home/runner/work/portage/portage/lib/portage/util/_async/ForkProcess.py", line 328, in _bootstrap
        sys.exit(target(*(args or []), **(kwargs or {})))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/work/portage/portage/lib/portage/process.py", line 1441, in __call__
        return self._target(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/work/portage/portage/lib/portage/process.py", line 853, in _exec_wrapper
        writemsg(f"{e}:\n   {' '.join(mycommand)}\n", noiselevel=-1)
                             ^^^^^^^^^^^^^^^^^^^
    TypeError: sequence item 1: expected str instance, int found
    
    Bug: https://bugs.gentoo.org/916566#c20
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/tests/process/test_spawn_returnproc.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 22 Larry the Git Cow gentoo-dev 2024-03-04 15:31:13 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=6f9a10d38259dd61b948837e193b047464791845

commit 6f9a10d38259dd61b948837e193b047464791845
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2024-03-04 05:31:08 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2024-03-04 05:33:44 +0000

    SpawnProcess: Optimize away null input for create_pipe=False
    
    When create_pipe=False support was added in commit e8b31c86eaed,
    a null input file descriptor was used for PipeLogger and BuildLogger
    instances. Optimize this away, eliminating the unnecessary loggers.
    
    Fixes: e8b31c86eaed ("ForkProcess: Prevent redundant pipe and set_term_size recursion")
    Bug: https://bugs.gentoo.org/916566
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/SpawnProcess.py            | 51 +++++++++++++++++-----------------
 lib/portage/util/_async/ForkProcess.py |  7 ++---
 2 files changed, 28 insertions(+), 30 deletions(-)