Since os.fork() is unsafe in threaded processes (bug 914876), it would be useful for portage.process.spawn to support alternative process cloning implementations. The process cloning interface can be a function that takes a callable "target" argument, and returns an implementation dependent object. The backward compatible os.fork() implementation of this interface can simply return an int pid number. An implementation based on multiprocessing.Process can return a multiprocessing.Process instance instead of just a pid. It can be a simple function like this: def multiprocessing_start(target): proc = multiprocessing.Process(target=target) proc.start() return proc A very naive os.fork() implementation looks something like this (a better implementation would account for bug 345289): def fork_start(target): pid = os.fork() if pid == 0: os._exit(target()) return pid Callers of the portage.process.spawn API can choose the process cloning implementations based on environmental details, such as threading.active_count() and multiprocessing.get_start_method(), in order to avoid mixing threads with os.fork(). Since ForkProcess uses a thread to send fd_pipes for the multiprocessing spawn start method, it would be safest to avoid calling os.fork() when using the multiprocessing spawn start method.
For the multiprocessing spawn start method, fd_pipes only works via asynchronous send_handle calls, so the existing support for this in ForkProcess can be used here. However, this means that fd_pipes will be accessed by a background thread after the spawn function has returned (can trigger timing issues like bug 916601).
In the tracker issue for making the GIL optional (https://github.com/python/cpython/issues/108219), it lists this commit from nogil-3.12 as something to merge upstream: c1befd7689 Stop the world before fork() and Python shutdown It calls a _PyRuntimeState_StopTheWorld function to stop all other threads just before fork(). However, the stopped threads will still trigger a warning about possible deadlocks in the child, via the warn_about_fork_with_threads function. (In reply to Zac Medico from comment #0) > def multiprocessing_start(target): > proc = multiprocessing.Process(target=target) > proc.start() > return proc Maybe multiprocessing.Process is not the best choice, since it is specialized for execution of python code. We could instead rely on the subprocess module's use of posix_spawn to avoid fork, but posix_spawn will still require use to use send_handle for fd_pipes support.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=7ec7a647ef6e2d94e6f2b387d0a012e78cafbaff commit 7ec7a647ef6e2d94e6f2b387d0a012e78cafbaff Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-01-29 08:20:55 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-01-29 08:43:52 +0000 process.spawn: add abstraction layer for os.fork() Since os.fork() is unsafe in threaded processes, add a basic abstraction layer that will ultimately allow support of other process cloning implementations. Usage of os.fork() is now wrapped in a _start_fork function which can easily be replaced with an alternative implementation. This function takes target, args, and kwargs arguments which are equivalent to the corresponding multiprocessing.Process parameters. It also has fd_pipes and close_fds parameters, since implementation of these is dependent on the process cloning implementation. The process cloning implementation does not need to be bothered with the details of the _exec function, since spawn uses an _exec_wrapper function as a target function which calls _exec and handles any exceptions appropriately (special exception handling is required for success of test_spawnE2big related to bug 830187). Bug: https://bugs.gentoo.org/916566 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/process.py | 203 +++++++++++++++++++++++++++++++++---------------- 1 file changed, 139 insertions(+), 64 deletions(-)
If we make all of the process cloning implementations return a asyncio.subprocess.Process like object, this object can provide an interface to wait for the pid that is independent of the process cloning implementation, and can encapsulate implementation-dependent objects like multiprocessing.Process which which are designed to manage the pid lifecycle and need to persist until the pid exits. A spawn keyword argument similar to returnpid which we might called returnproc can be used to make spawn return Process objects instead of raw pids, and we can migrate everything to use this new returnproc argument instead of returnpid.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=c556fee09b6b103a9bb8c2afa1c7a7ef25022598 commit c556fee09b6b103a9bb8c2afa1c7a7ef25022598 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-02-01 05:48:54 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-02-01 16:10:28 +0000 process.spawn: Add returnproc parameter In order to migrate away from unsafe os.fork() usage in threaded processes (https://github.com/python/cpython/issues/84559), add a returnproc parameter that is similar to returnpid, which causes spawn to return a single Process object instead of a list of pids. The Process API is a subset of asyncio.subprocess.Process. The returnproc parameter conflicts with the logfile parameter, since the caller is expected to use the fd_pipes parameter to implement logging (this was also true for the returnpid parameter). In the future, spawn will return objects of a different type but with a compatible interface to Process, in order to encapsulate implementation-dependent objects like multiprocessing.Process which are designed to manage the process lifecycle and need to persist until it exits. Trigger a UserWarning when the returnpid parameter is used, in order to encourage migration to returnproc (do not use DeprecationWarning since it is hidden by default). This warning will be temporarily suppressed for portage internals, until they finish migrating to returnproc. There are probably very few if any external consumers of spawn with the returnpid parameter, so it seems safe to move quickly with this deprecation. Bug: https://bugs.gentoo.org/916566 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/process.py | 84 ++++++++++++++++++++++ lib/portage/tests/process/meson.build | 1 + lib/portage/tests/process/test_spawn_returnproc.py | 39 ++++++++++ 3 files changed, 124 insertions(+)
With portage.process.MultiprocessingProcess added in https://github.com/gentoo/portage/pull/1248, it will be very easy to replace _start_fork with a function that starts a ForkProcess instance and returns the associated MultiprocessingProcess instance. Then we might just use _start_fork for temporary returnpid backward compatiblity, since ForkProcess will use fork if multiprocessing.get_start_method() == "fork". We can add a FEATURES setting to make portage programs call multiprocessing.set_start_method("spawn"), or perhaps even do it unconditionally so that there will be no change when https://github.com/python/cpython/issues/84559 makes "spawn" the default multiprocessing start method.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=055c66ec9482064aaaf51bfb6b01e260ea27808e commit 055c66ec9482064aaaf51bfb6b01e260ea27808e Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-02-02 16:01:18 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-02-02 16:01:18 +0000 SpawnProcess: Use spawn returnproc parameter Migrate SpawnProcess to use the spawn returnproc parameter, and make adaptations to descendent classes as needed. Introduce a portage.process.MultiprocessingProcess class for ForkProcess to wrap multiprocessing.Process instances, needed because ForkProcess inherits from SpawnProcess. Use portage.process.Process to wrap the pid in EbuildMetadataPhase, so that returnproc support in the doebuild function can be reserved for a later commit. Bug: https://bugs.gentoo.org/916566 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/EbuildMetadataPhase.py | 4 +- lib/_emerge/SpawnProcess.py | 16 ++--- lib/_emerge/SubProcess.py | 25 +++---- lib/portage/package/ebuild/doebuild.py | 4 +- lib/portage/process.py | 120 ++++++++++++++++++++++++++++---- lib/portage/util/_async/ForkProcess.py | 87 +++-------------------- lib/portage/util/_async/PopenProcess.py | 5 +- 7 files changed, 143 insertions(+), 118 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=305612d1b04aa06d3d1a1c8b51d046a644742fd5 commit 305612d1b04aa06d3d1a1c8b51d046a644742fd5 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-02-03 21:27:45 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-02-03 21:27:45 +0000 process.spawn: Use multiprocessing.Process for returnproc Use multiprocessing.Process for returnproc, so that fork will stop being used when python makes "spawn" the default multiprocessing start method. Continue to use _start_fork when returnproc is not enabled, for backward compatibility. Ultimately, it can be removed at the same time as the returnpid parameter. Bug: https://bugs.gentoo.org/916566 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/SpawnProcess.py | 4 +- lib/portage/process.py | 72 ++++++++++++++++++---- lib/portage/tests/ebuild/test_doebuild_fd_pipes.py | 6 +- 3 files changed, 67 insertions(+), 15 deletions(-)
All of the remaining returnpid consumers are eliminated in https://github.com/gentoo/portage/pull/1250, so that basically completes this bug.
Actually, the returnproc=False case still uses fork, so that needs to be fixed by making it use _start_proc and use the event loop to wait for MultiprocessingProcess objects.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=6aeee45d9b88d7dc8c450ba86975e4b017db203c commit 6aeee45d9b88d7dc8c450ba86975e4b017db203c Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-02-03 21:24:49 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-02-03 22:40:34 +0000 doebuild: Add returnproc parameter and deprecate returnpid Raise NotImplementedError if returnproc is enabled for anything other than the "depend" phase, since corresponding returnpid support has long been deprecated. Bug: https://bugs.gentoo.org/916566 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/EbuildMetadataPhase.py | 5 ++-- lib/portage/package/ebuild/doebuild.py | 47 ++++++++++++++++++++++++++-------- 2 files changed, 40 insertions(+), 12 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=e8b31c86eaed645a8740fb2844e2935aee161e43 commit e8b31c86eaed645a8740fb2844e2935aee161e43 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-02-05 05:55:11 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-02-06 01:30:21 +0000 ForkProcess: Prevent redundant pipe and set_term_size recursion When process.spawn is updated to call ForkProcess for bug 916566, it needs to avoid recursion via set_term_size. Bug: https://bugs.gentoo.org/916566 Bug: https://bugs.gentoo.org/923750 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/SpawnProcess.py | 31 ++++++++++++++++++++++++------- lib/portage/util/_async/ForkProcess.py | 28 +++++++++++++++++++++++----- 2 files changed, 47 insertions(+), 12 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=66d8f8388e5b9da7e07510f78ec487913e2ceaf5 commit 66d8f8388e5b9da7e07510f78ec487913e2ceaf5 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-02-04 00:16:29 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-02-07 00:36:17 +0000 ChildWatcherTestCase: Remove obsolete test which uses spawn returnpid This test was added for bug 649588 when there was still an internal event loop implementation for python2. It is no longer relevant and uses the deprecated spawn returnpid parameter, so remove it. Bug: https://bugs.gentoo.org/916566 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/tests/util/futures/asyncio/meson.build | 1 - .../util/futures/asyncio/test_child_watcher.py | 50 ---------------------- 2 files changed, 51 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=3e10368e52ecead86b75478ca448ef0f59333e3e commit 3e10368e52ecead86b75478ca448ef0f59333e3e Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-02-04 04:34:45 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-02-07 00:55:46 +0000 process.spawn: Use _start_proc for returnpid=False This essentially completes the implementation of bug 916566, eliminating os.fork() usage when "spawn" becomes the default multiprocessing start method. Bug: https://bugs.gentoo.org/916566 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/process.py | 35 +++++++++++++++++------------------ 1 file changed, 17 insertions(+), 18 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=1ae4f574f69eca3146137ffb40c5afd8a4872777 commit 1ae4f574f69eca3146137ffb40c5afd8a4872777 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-02-04 00:22:08 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-02-07 00:55:46 +0000 process.spawn: Enable returnpid warning for internals All internal returnpid consumers have been migrated to use the new returnproc parameter instead. Bug: https://bugs.gentoo.org/916566 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/process.py | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=62332ee82b8b88fa5a65aafa7c221ccdaa7d65a8 commit 62332ee82b8b88fa5a65aafa7c221ccdaa7d65a8 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-02-04 00:11:07 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-02-07 00:55:46 +0000 RsyncSync: Migrate to spawn returnproc parameter Bug: https://bugs.gentoo.org/916566 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/sync/modules/rsync/rsync.py | 40 +++++++++++++-------------- lib/portage/util/futures/_asyncio/__init__.py | 6 +++- 2 files changed, 24 insertions(+), 22 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=fa8e8f1895ed889aece2f67725df55d6ccf127fb commit fa8e8f1895ed889aece2f67725df55d6ccf127fb Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-02-03 23:41:45 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-02-07 00:55:46 +0000 socks5: Migrate to spawn returnproc parameter Bug: https://bugs.gentoo.org/916566 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/util/socks5.py | 36 +++++++++++++++++++----------------- 1 file changed, 19 insertions(+), 17 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=190e8ad1a238d20d782235dd1faa6b00d1b3fd4a commit 190e8ad1a238d20d782235dd1faa6b00d1b3fd4a Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-02-03 21:24:49 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-02-07 00:55:45 +0000 doebuild: Add returnproc parameter and deprecate returnpid Raise NotImplementedError if returnproc is enabled for anything other than the "depend" phase, since corresponding returnpid support has long been deprecated. Bug: https://bugs.gentoo.org/916566 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/EbuildMetadataPhase.py | 4 +-- lib/portage/package/ebuild/doebuild.py | 47 ++++++++++++++++++++++++++-------- 2 files changed, 39 insertions(+), 12 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=a69c1b853a47346192950c91b088163490287350 commit a69c1b853a47346192950c91b088163490287350 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-02-03 21:27:45 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-02-07 00:49:26 +0000 process.spawn: Use multiprocessing.Process for returnproc Use multiprocessing.Process for returnproc, so that fork will stop being used when python makes "spawn" the default multiprocessing start method. Continue to use _start_fork when returnproc is not enabled, for backward compatibility. Ultimately, it can be removed at the same time as the returnpid parameter. The _setup_pipes_after_fork wrapper prevents a "Bad file descriptor" error by making fd_pipes inheritable on exec for bug 923755. ForkProcess does not handle this because its target function does not necessarily exec. Bug: https://bugs.gentoo.org/916566 Bug: https://bugs.gentoo.org/923755 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/SpawnProcess.py | 4 +- lib/portage/process.py | 92 ++++++++++++++++++++++++++++++++++++++------- 2 files changed, 82 insertions(+), 14 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=419cce79f9082308c848df0a98f367de4d1c50a3 commit 419cce79f9082308c848df0a98f367de4d1c50a3 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-02-11 21:58:10 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-02-12 07:56:10 +0000 process._exec: Use _start_fork for os.fork() error handling Use _start_fork for os.fork() error handling, ensuring that if exec fails then the child process will display a traceback before it exits via os._exit to suppress any finally blocks from parent's call stack (bug 345289). Bug: https://bugs.gentoo.org/345289 Bug: https://bugs.gentoo.org/916566 Bug: https://bugs.gentoo.org/924313 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/process.py | 259 ++++++++++++++++++++++++++++--------------------- 1 file changed, 151 insertions(+), 108 deletions(-)
Sometimes lib/portage/tests/process/test_spawn_returnproc.py fails because sleep exits with status 1 even though it was killed by SIGTERM: https://github.com/gentoo/portage/actions/runs/7904644492/job/21575430214?pr=1270 > =================================== FAILURES =================================== > _____________ SpawnReturnProcTestCase.testSpawnReturnProcTerminate _____________ > [gw1] linux -- Python 3.12.2 /opt/hostedtoolcache/Python/3.12.2/x64/bin/python > > self = <portage.tests.process.test_spawn_returnproc.SpawnReturnProcTestCase testMethod=testSpawnReturnProcTerminate> > > def testSpawnReturnProcTerminate(self): > sleep_binary = find_binary("sleep") > self.assertNotEqual(sleep_binary, None) > > loop = global_event_loop() > > async def watch_pid(): > proc = spawn([sleep_binary, 9999], returnproc=True) > proc.terminate() > self.assertEqual(await proc.wait(), -signal.SIGTERM) > > > loop.run_until_complete(watch_pid()) > > loop = <portage.util._eventloop.asyncio_event_loop.AsyncioEventLoop object at 0x7f020f707e90> > self = <portage.tests.process.test_spawn_returnproc.SpawnReturnProcTestCase testMethod=testSpawnReturnProcTerminate> > sleep_binary = '/usr/bin/sleep' > watch_pid = <function SpawnReturnProcTestCase.testSpawnReturnProcTerminate.<locals>.watch_pid at 0x7f020f396340>
(In reply to Zac Medico from comment #16) > Sometimes lib/portage/tests/process/test_spawn_returnproc.py fails because > sleep exits with status 1 even though it was killed by SIGTERM: Maybe it was killed before the exec to sleep. In ForkProcess._bootstrap there's this SIGTERM setup related to bug 353239: > @staticmethod > def _bootstrap(child_connection, have_send_handle, fd_pipes, target, args, kwargs): > # Use default signal handlers in order to avoid problems > # killing subprocesses as reported in bug #353239. > signal.signal(signal.SIGINT, signal.SIG_DFL) > signal.signal(signal.SIGTERM, signal.SIG_DFL)
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=77c44c46194922509bc4f2b5cfc099412a560a69 commit 77c44c46194922509bc4f2b5cfc099412a560a69 Author: Sam James <sam@gentoo.org> AuthorDate: 2024-02-22 07:23:40 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-02-22 07:23:50 +0000 sys-apps/portage: add 3.0.62 Closes: https://bugs.gentoo.org/663324 Closes: https://bugs.gentoo.org/728046 Closes: https://bugs.gentoo.org/891137 Closes: https://bugs.gentoo.org/906368 Closes: https://bugs.gentoo.org/916566 Closes: https://bugs.gentoo.org/921170 Closes: https://bugs.gentoo.org/921208 Closes: https://bugs.gentoo.org/921400 Closes: https://bugs.gentoo.org/922038 Closes: https://bugs.gentoo.org/922142 Closes: https://bugs.gentoo.org/923368 Closes: https://bugs.gentoo.org/923750 Closes: https://bugs.gentoo.org/923841 Closes: https://bugs.gentoo.org/923852 Closes: https://bugs.gentoo.org/923854 Closes: https://bugs.gentoo.org/924192 Closes: https://bugs.gentoo.org/924273 Closes: https://bugs.gentoo.org/924585 Closes: https://bugs.gentoo.org/921380 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.62.ebuild | 246 +++++++++++++++++++++++++++++++++ 2 files changed, 247 insertions(+)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=3cc986f87ddda86ee93770e03cca06346aee54c5 commit 3cc986f87ddda86ee93770e03cca06346aee54c5 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-02-23 06:06:14 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-02-23 06:48:29 +0000 AsyncioEventLoop: Call process.run_exitfuncs() before close For the event loop running in the main thread, call process.run_exitfuncs() before close with the event loop running so that anything attached can clean itself up (like the socks5 ProxyManager for bug 925240). This is necessary because process.spawn uses the event loop to implement the new returnproc parameter related to bug 916566. Bug: https://bugs.gentoo.org/916566 Bug: https://bugs.gentoo.org/925240 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/tests/util/test_socks5.py | 51 ++++++++++++++++++++++- lib/portage/util/_eventloop/asyncio_event_loop.py | 44 +++++++++++++++---- lib/portage/util/socks5.py | 16 ++++++- 3 files changed, 101 insertions(+), 10 deletions(-)
(In reply to Zac Medico from comment #17) > (In reply to Zac Medico from comment #16) > > Sometimes lib/portage/tests/process/test_spawn_returnproc.py fails because > > sleep exits with status 1 even though it was killed by SIGTERM: > > Maybe it was killed before the exec to sleep. In ForkProcess._bootstrap > there's this SIGTERM setup related to bug 353239: Actually the test log shows an unexpected TypeError which I also triggered with https://github.com/gentoo/portage/pull/1283: ----------------------------- Captured stderr call ----------------------------- Process ForkProcess-23: Traceback (most recent call last): File "/home/runner/work/portage/portage/lib/portage/process.py", line 818, in _exec_wrapper _exec( File "/home/runner/work/portage/portage/lib/portage/process.py", line 1068, in _exec _exec2( File "/home/runner/work/portage/portage/lib/portage/process.py", line 1166, in _exec2 os.execve(binary, myargs, env) TypeError: expected str, bytes or os.PathLike object, not int During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/home/runner/work/portage/portage/lib/portage/util/_async/ForkProcess.py", line 328, in _bootstrap sys.exit(target(*(args or []), **(kwargs or {}))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/work/portage/portage/lib/portage/process.py", line 1441, in __call__ return self._target(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/work/portage/portage/lib/portage/process.py", line 853, in _exec_wrapper writemsg(f"{e}:\n {' '.join(mycommand)}\n", noiselevel=-1) ^^^^^^^^^^^^^^^^^^^ TypeError: sequence item 1: expected str instance, int found
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=9e84ef57ba747766c9147c1ac1b247faa1f05956 commit 9e84ef57ba747766c9147c1ac1b247faa1f05956 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-02-25 00:31:13 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-02-25 00:42:46 +0000 testSpawnReturnProcTerminate: Fix integer in spawn command argument The invalid integer in the spawn command argument intermittently triggered this error when SIGTERM did not arrive until after the exec call: ----------------------------- Captured stderr call ----------------------------- Process ForkProcess-23: Traceback (most recent call last): File "/home/runner/work/portage/portage/lib/portage/process.py", line 818, in _exec_wrapper _exec( File "/home/runner/work/portage/portage/lib/portage/process.py", line 1068, in _exec _exec2( File "/home/runner/work/portage/portage/lib/portage/process.py", line 1166, in _exec2 os.execve(binary, myargs, env) TypeError: expected str, bytes or os.PathLike object, not int During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/home/runner/work/portage/portage/lib/portage/util/_async/ForkProcess.py", line 328, in _bootstrap sys.exit(target(*(args or []), **(kwargs or {}))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/work/portage/portage/lib/portage/process.py", line 1441, in __call__ return self._target(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/work/portage/portage/lib/portage/process.py", line 853, in _exec_wrapper writemsg(f"{e}:\n {' '.join(mycommand)}\n", noiselevel=-1) ^^^^^^^^^^^^^^^^^^^ TypeError: sequence item 1: expected str instance, int found Bug: https://bugs.gentoo.org/916566#c20 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/tests/process/test_spawn_returnproc.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=6f9a10d38259dd61b948837e193b047464791845 commit 6f9a10d38259dd61b948837e193b047464791845 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-03-04 05:31:08 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-03-04 05:33:44 +0000 SpawnProcess: Optimize away null input for create_pipe=False When create_pipe=False support was added in commit e8b31c86eaed, a null input file descriptor was used for PipeLogger and BuildLogger instances. Optimize this away, eliminating the unnecessary loggers. Fixes: e8b31c86eaed ("ForkProcess: Prevent redundant pipe and set_term_size recursion") Bug: https://bugs.gentoo.org/916566 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/SpawnProcess.py | 51 +++++++++++++++++----------------- lib/portage/util/_async/ForkProcess.py | 7 ++--- 2 files changed, 28 insertions(+), 30 deletions(-)
