As noted in bug 914876 comment #3, the ForkProcess _run instance method is essentially incompatible with the multiprocessing spawn start method because passing self._run to the subprocess means that self (the ForkProcess instance and anything it refers to including a PollScheduler instance) will have to be pickled and sent to the subprocess. The solution is to deprecate the ForkProcess _run method and replace it with a "target" constructor parameter like multiprocessing.Process has.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=04184a03b74669fdb48403cb8002de6395cf8684 commit 04184a03b74669fdb48403cb8002de6395cf8684 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2023-10-03 07:33:29 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2023-10-03 14:48:01 +0000 AsyncFunction: Migrate to ForkProcess target parameter Bug: https://bugs.gentoo.org/915099 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/util/_async/AsyncFunction.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=72760ab948047cf1f5b9e1c1fcf9d2c4934afc8c commit 72760ab948047cf1f5b9e1c1fcf9d2c4934afc8c Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2023-10-03 06:24:21 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2023-10-03 14:47:37 +0000 ForkProcess: Add target constructor parameter If the _run method is not implemented, then use a new "target" constructor parameter like multiprocessing.Process. Support "args" and "kwargs" constructor parameters as well. If the _run method is implemented, then behave in a backward compatible manner, and ignore the "args" and "kwargs" constructor parameters which are used by the AsyncFunction subclass. Bug: https://bugs.gentoo.org/915099 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/util/_async/AsyncFunction.py | 7 +------ lib/portage/util/_async/ForkProcess.py | 35 +++++++++++++++++++++++++++----- 2 files changed, 31 insertions(+), 11 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=24db094b68a2024cfd23e0366d8a81e8bded5838 commit 24db094b68a2024cfd23e0366d8a81e8bded5838 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2023-10-04 02:37:00 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2023-10-04 02:39:26 +0000 tests: Migrate to ForkProcess target parameter Bug: https://bugs.gentoo.org/915099 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/tests/ebuild/test_doebuild_fd_pipes.py | 16 +++++----------- lib/portage/tests/ebuild/test_ipc_daemon.py | 16 +++------------- lib/portage/tests/util/futures/test_iter_completed.py | 7 +++---- 3 files changed, 11 insertions(+), 28 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=3dfe5e326eaaca877bc7a45ec84d6b39fc8d137a commit 3dfe5e326eaaca877bc7a45ec84d6b39fc8d137a Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2023-10-04 02:56:10 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2023-10-04 03:07:12 +0000 FileDigester: Migrate to AsyncFunction Bug: https://bugs.gentoo.org/915099 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/util/_async/FileDigester.py | 71 +++++---------------------------- 1 file changed, 11 insertions(+), 60 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=73eb44fce683a8cbfca195f01783b94eea6e7eca commit 73eb44fce683a8cbfca195f01783b94eea6e7eca Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2023-10-04 03:42:41 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2023-10-04 03:43:18 +0000 FileCopier: multiprocessing spawn compat Bug: https://bugs.gentoo.org/915099 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/util/_async/FileCopier.py | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=f45e2bb561f2f5d16bdf8f6cd31cc393d2794f92 commit f45e2bb561f2f5d16bdf8f6cd31cc393d2794f92 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2023-10-04 04:06:49 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2023-10-04 04:11:14 +0000 SyncfsProcess: Migrate to ForkProcess target parameter Bug: https://bugs.gentoo.org/915099 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/dbapi/_SyncfsProcess.py | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
These are the remaining ForkProcess._run implementations which are not yet picklable: > lib/_emerge/EbuildFetcher.py: def _run(self): > lib/portage/dbapi/_MergeProcess.py: def _run(self): > lib/portage/package/ebuild/_parallel_manifest/ManifestProcess.py: def _run(self):
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=8e88810e61b6b36bd36e8fb1e5ae76715a72dcd9 commit 8e88810e61b6b36bd36e8fb1e5ae76715a72dcd9 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2023-10-05 04:17:10 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2023-10-05 04:18:31 +0000 EbuildFetcherProcess: Migrate to ForkProcess target parameter Bug: https://bugs.gentoo.org/915099 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/EbuildFetcher.py | 35 +++++++++++++++++++++++------------ 1 file changed, 23 insertions(+), 12 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=07dbfbdf0504387fe96f865440550b720245f0bb commit 07dbfbdf0504387fe96f865440550b720245f0bb Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2023-10-05 04:36:43 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2023-10-05 04:37:06 +0000 ManifestProcess: Migrate to ForkProcess target parameter Bug: https://bugs.gentoo.org/915099 Signed-off-by: Zac Medico <zmedico@gentoo.org> .../ebuild/_parallel_manifest/ManifestProcess.py | 28 +++++++++++++++++----- 1 file changed, 22 insertions(+), 6 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=2ccc08e3b629d3f56f028cd767ebd5ff1a8edaf1 commit 2ccc08e3b629d3f56f028cd767ebd5ff1a8edaf1 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2023-10-05 04:43:50 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2023-10-05 05:18:58 +0000 MergeProcess: Migrate to ForkProcess target parameter Bug: https://bugs.gentoo.org/915099 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/dbapi/_MergeProcess.py | 73 +++++++++++++++++++++++++++----------- 1 file changed, 52 insertions(+), 21 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=ed458fa634c37c13cadd436b38498678f4ee103d commit ed458fa634c37c13cadd436b38498678f4ee103d Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2023-10-05 05:54:40 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2023-10-05 05:59:39 +0000 ForkProcess: Warn if _run method is used Bug: https://bugs.gentoo.org/915099 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/util/_async/ForkProcess.py | 6 ++++++ 1 file changed, 6 insertions(+)
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=3500483f75789c36e379c1b6546a09df7c11e2b1 commit 3500483f75789c36e379c1b6546a09df7c11e2b1 Author: Sam James <sam@gentoo.org> AuthorDate: 2023-10-20 00:49:33 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-10-20 00:51:00 +0000 sys-apps/portage: add 3.0.53 Closes: https://bugs.gentoo.org/915120 Closes: https://bugs.gentoo.org/821529 Closes: https://bugs.gentoo.org/914441 Closes: https://bugs.gentoo.org/914722 Closes: https://bugs.gentoo.org/914873 Closes: https://bugs.gentoo.org/915099 Closes: https://bugs.gentoo.org/915123 Closes: https://bugs.gentoo.org/915128 Closes: https://bugs.gentoo.org/915136 Closes: https://bugs.gentoo.org/915330 Closes: https://bugs.gentoo.org/915494 Closes: https://bugs.gentoo.org/915834 Closes: https://bugs.gentoo.org/915903 Closes: https://bugs.gentoo.org/900224 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.53.ebuild | 238 +++++++++++++++++++++++++++++++++ 2 files changed, 239 insertions(+)
