While testing https://github.com/gentoo/portage/pull/1251 with multiprocessing.set_start_method("spawn", force=True), I found that testAsynchronousLockWaitCancel can fail because lock2.cancel() kills the corresponding process before _send_fd_pipes has completed, resulting in this exception: Traceback (most recent call last): File "portage/lib/_emerge/SpawnProcess.py", line 215, in _main_exit main_task.result() File "portage/lib/portage/util/_async/ForkProcess.py", line 178, in _main await self.scheduler.run_in_executor( File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "portage/lib/portage/util/_async/ForkProcess.py", line 157, in _send_fd_pipes self._files.connection.send( File "/usr/lib/python3.12/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/usr/lib/python3.12/multiprocessing/connection.py", line 427, in _send_bytes self._send(header + buf) File "/usr/lib/python3.12/multiprocessing/connection.py", line 384, in _send n = write(self._handle, buf) ^^^^^^^^^^^^^^^^^^^^^^^^ BrokenPipeError: [Errno 32] Broken pipe This exception is easily handled by converting BrokenPipeError to asyncio.CancelledError: --- a/lib/portage/util/_async/ForkProcess.py +++ b/lib/portage/util/_async/ForkProcess.py @@ -153,15 +153,18 @@ class ForkProcess(SpawnProcess): This performs blocking IO, intended for invocation via run_in_executor. """ fd_list = list(set(self._fd_pipes.values())) - self._files.connection.send( - (self._fd_pipes, fd_list), - ) - for fd in fd_list: - multiprocessing.reduction.send_handle( - self._files.connection, - fd, - self.pid, + try: + self._files.connection.send( + (self._fd_pipes, fd_list), ) + for fd in fd_list: + multiprocessing.reduction.send_handle( + self._files.connection, + fd, + self.pid, + ) + except BrokenPipeError as e: + raise asyncio.CancelledError from e # self._fd_pipes contains duplicates that must be closed. for fd in fd_list:
Patch now included in https://github.com/gentoo/portage/pull/1251: commit a842f20e45815b91fab417a9fbfbeeddfb4adf86 Author: Zac Medico <zmedico@gentoo.org> Date: Mon Feb 5 17:35:25 2024 -0800 ForkProcess: Handle BrokenPipeError in _send_fd_pipes Convert _send_fd_pipes BrokenPipeError to asyncio.CancelledError, in order to gracefully handle a concurrently terminated child process as in bug 923852. Bug: https://bugs.gentoo.org/923852 Signed-off-by: Zac Medico <zmedico@gentoo.org>
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=7d7ef237f3dddcf450fedab5aabfd57d1fb3406d commit 7d7ef237f3dddcf450fedab5aabfd57d1fb3406d Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-02-06 01:35:25 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-02-07 00:36:56 +0000 ForkProcess: Handle BrokenPipeError in _send_fd_pipes Convert _send_fd_pipes BrokenPipeError to asyncio.CancelledError, in order to gracefully handle a concurrently terminated child process as in testAsynchronousLockWaitCancel. Even if the child terminated abnormally, then there is no harm in suppressing the exception here, since the child error should have gone to stderr. Bug: https://bugs.gentoo.org/923852 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/util/_async/ForkProcess.py | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-)
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=77c44c46194922509bc4f2b5cfc099412a560a69 commit 77c44c46194922509bc4f2b5cfc099412a560a69 Author: Sam James <sam@gentoo.org> AuthorDate: 2024-02-22 07:23:40 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-02-22 07:23:50 +0000 sys-apps/portage: add 3.0.62 Closes: https://bugs.gentoo.org/663324 Closes: https://bugs.gentoo.org/728046 Closes: https://bugs.gentoo.org/891137 Closes: https://bugs.gentoo.org/906368 Closes: https://bugs.gentoo.org/916566 Closes: https://bugs.gentoo.org/921170 Closes: https://bugs.gentoo.org/921208 Closes: https://bugs.gentoo.org/921400 Closes: https://bugs.gentoo.org/922038 Closes: https://bugs.gentoo.org/922142 Closes: https://bugs.gentoo.org/923368 Closes: https://bugs.gentoo.org/923750 Closes: https://bugs.gentoo.org/923841 Closes: https://bugs.gentoo.org/923852 Closes: https://bugs.gentoo.org/923854 Closes: https://bugs.gentoo.org/924192 Closes: https://bugs.gentoo.org/924273 Closes: https://bugs.gentoo.org/924585 Closes: https://bugs.gentoo.org/921380 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.62.ebuild | 246 +++++++++++++++++++++++++++++++++ 2 files changed, 247 insertions(+)
