The ForkProcess fd_pipes parameter does not work with the multiprocessing spawn start method. This test case shows that it's possible to use a multiprocessing.Connection file descriptor as stdout in a subprocess with the multiprocessing spawn start method: > #!/usr/bin/env python > > import multiprocessing > import os > import select > import sys > > > def subprocess_target(conn): > os.dup2(conn.fileno(), sys.stdin.fileno()) > os.dup2(conn.fileno(), sys.stdout.fileno()) > print("hello world") > conn.close() > sys.stdin.close() > sys.stdout.close() > sys.exit(0) > > > if __name__ == "__main__": > multiprocessing.set_start_method("spawn", force=True) > conn1, conn2 = multiprocessing.Pipe(duplex=True) > proc = multiprocessing.Process(target=subprocess_target, args=(conn1,)) > proc.start() > conn1.close() > os.set_blocking(conn2.fileno(), False) > poller = select.poll() > poller.register( > conn2.fileno(), select.POLLIN | select.POLLPRI | select.POLLHUP | select.POLLERR > ) > eof = False > while not eof: > try: > for fd, flag in poller.poll(): > if flag & select.POLLIN: > print(os.read(fd, 4096).decode(), end="") > > if flag & select.POLLHUP: > eof = True > break > except EOFError: > break > proc.join()
The self.logfile attribute now triggers fd_pipes usage in SpawnProcess, so ForkProcess will need a way to prevent that and log via multiprocessing.Pipe instead.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=767a746035960a211229fac33aabc042d30234b7 commit 767a746035960a211229fac33aabc042d30234b7 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2023-10-17 01:54:10 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2023-10-17 16:33:57 +0000 ForkProcess: Support logging via multiprocessing.Pipe Detect the multiprocessing spawn start method, and log via multiprocessing.Pipe because file descriptor inheritance via fd_pipes does not work with the spawn start method. The included test cases set the multiprocessing start method to spawn in a subprocess, in order to prevent interference with the main process. Bug: https://bugs.gentoo.org/915123 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/SpawnProcess.py | 9 +++- lib/portage/tests/process/meson.build | 1 + lib/portage/tests/process/test_AsyncFunction.py | 23 +++++++++- lib/portage/tests/process/test_ForkProcess.py | 39 +++++++++++++++++ lib/portage/util/_async/ForkProcess.py | 58 ++++++++++++++++++++++--- 5 files changed, 120 insertions(+), 10 deletions(-)
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=3500483f75789c36e379c1b6546a09df7c11e2b1 commit 3500483f75789c36e379c1b6546a09df7c11e2b1 Author: Sam James <sam@gentoo.org> AuthorDate: 2023-10-20 00:49:33 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-10-20 00:51:00 +0000 sys-apps/portage: add 3.0.53 Closes: https://bugs.gentoo.org/915120 Closes: https://bugs.gentoo.org/821529 Closes: https://bugs.gentoo.org/914441 Closes: https://bugs.gentoo.org/914722 Closes: https://bugs.gentoo.org/914873 Closes: https://bugs.gentoo.org/915099 Closes: https://bugs.gentoo.org/915123 Closes: https://bugs.gentoo.org/915128 Closes: https://bugs.gentoo.org/915136 Closes: https://bugs.gentoo.org/915330 Closes: https://bugs.gentoo.org/915494 Closes: https://bugs.gentoo.org/915834 Closes: https://bugs.gentoo.org/915903 Closes: https://bugs.gentoo.org/900224 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.53.ebuild | 238 +++++++++++++++++++++++++++++++++ 2 files changed, 239 insertions(+)
