While testing https://github.com/gentoo/portage/pull/1251 with multiprocessing.set_start_method("spawn", force=True), I found that RetryForkExecutorTestCase fails due to a pickling error: > =================================== FAILURES =================================== > __________________ RetryForkExecutorTestCase.testHangForever ___________________ > > self = <portage.tests.util.futures.test_retry.RetryForkExecutorTestCase testMethod=testHangForever> > > def testHangForever(self): > loop = global_event_loop() > with self._wrap_coroutine_func(HangForever()) as func_coroutine: > decorator = retry( > try_max=2, > try_timeout=0.1, > delay_func=RandomExponentialBackoff(multiplier=0.1, base=2), > ) > decorated_func = decorator(func_coroutine, loop=loop) > done, pending = loop.run_until_complete( > > asyncio.wait([decorated_func()], loop=loop) > ) > > lib/portage/tests/util/futures/test_retry.py:126: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > lib/portage/util/futures/retry.py:97: in _retry > _Retry( > lib/portage/util/futures/retry.py:142: in __init__ > self._begin_try() > lib/portage/util/futures/retry.py:160: in _begin_try > self._current_task = asyncio.ensure_future(self._func(), loop=self._loop) > lib/portage/tests/util/futures/test_retry.py:275: in execute_wrapper > parent_loop.run_in_executor(self._executor, wrapper, kill_switch), > /usr/lib/python3.12/asyncio/base_events.py:860: in run_in_executor > executor.submit(func, *args), loop=self) > lib/portage/util/futures/executor/fork.py:46: in submit > self._schedule() > lib/portage/util/futures/executor/fork.py:62: in _schedule > proc.start() > lib/_emerge/AsynchronousTask.py:34: in start > self._start() > lib/portage/util/_async/AsyncFunction.py:35: in _start > ForkProcess._start(self) > lib/portage/util/_async/ForkProcess.py:65: in _start > self._proc = self._spawn(self.args, fd_pipes=None) > lib/portage/util/_async/ForkProcess.py:255: in _spawn > proc.start() > /usr/lib/python3.12/multiprocessing/process.py:121: in start > self._popen = self._Popen(self) > /usr/lib/python3.12/multiprocessing/context.py:224: in _Popen > return _default_context.get_context().Process._Popen(process_obj) > /usr/lib/python3.12/multiprocessing/context.py:289: in _Popen > return Popen(process_obj) > /usr/lib/python3.12/multiprocessing/popen_spawn_posix.py:32: in __init__ > super().__init__(process_obj) > /usr/lib/python3.12/multiprocessing/popen_fork.py:19: in __init__ > self._launch(process_obj) > /usr/lib/python3.12/multiprocessing/popen_spawn_posix.py:47: in _launch > reduction.dump(process_obj, fp) > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > obj = <Process name='Process-1' parent=2818913 initial> > file = <_io.BytesIO object at 0x7fb14a1cd350>, protocol = None > > def dump(obj, file, protocol=None): > '''Replacement for pickle.dump() using ForkingPickler.''' > > ForkingPickler(file, protocol).dump(obj) > E AttributeError: Can't pickle local object 'RetryForkExecutorTestCase._wrap_coroutine_func.<locals>.wrapper' > > /usr/lib/python3.12/multiprocessing/reduction.py:60: AttributeError > =========================== short test summary info ============================ > FAILED lib/portage/tests/util/futures/test_retry.py::RetryForkExecutorTestCase::testHangForever - AttributeError: Can't pickle local object 'RetryForkExecutorTestCase._wrap_...
Patch now included in https://github.com/gentoo/portage/pull/1251: commit 8a6f799062482715f296e1acae8a4cbc1d324444 Author: Zac Medico <zmedico@gentoo.org> Date: 2024-02-05 20:00:29 -0800 ForkExecutor: multiprocessing spawn compat Use the ForkProcess create_pipe=False parameter to avoid issues in the pipe code triggered with the "spawn" multiprocessing start method when spawn uses multiprocessing.Process. Also fix RetryForkExecutorTestCase to avoid pickling issues. Bug: https://bugs.gentoo.org/923854 Signed-off-by: Zac Medico <zmedico@gentoo.org>
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=6c4b542b2a830587869b6180e879b719e97fda66 commit 6c4b542b2a830587869b6180e879b719e97fda66 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-02-06 04:00:29 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-02-07 00:49:26 +0000 ForkExecutor: multiprocessing spawn compat Use the AsyncFunction create_pipe=False parameter to avoid issues in the pipe code triggered with the "spawn" multiprocessing start method when spawn uses multiprocessing.Process (bug 916566), since these jobs should inherit stdio streams and run in the foreground with no log. Also fix RetryForkExecutorTestCase to avoid pickling issues. Bug: https://bugs.gentoo.org/923854 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/tests/util/futures/test_retry.py | 42 ++++++++++++++++++---------- lib/portage/util/futures/executor/fork.py | 6 ++-- 2 files changed, 31 insertions(+), 17 deletions(-)
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=77c44c46194922509bc4f2b5cfc099412a560a69 commit 77c44c46194922509bc4f2b5cfc099412a560a69 Author: Sam James <sam@gentoo.org> AuthorDate: 2024-02-22 07:23:40 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-02-22 07:23:50 +0000 sys-apps/portage: add 3.0.62 Closes: https://bugs.gentoo.org/663324 Closes: https://bugs.gentoo.org/728046 Closes: https://bugs.gentoo.org/891137 Closes: https://bugs.gentoo.org/906368 Closes: https://bugs.gentoo.org/916566 Closes: https://bugs.gentoo.org/921170 Closes: https://bugs.gentoo.org/921208 Closes: https://bugs.gentoo.org/921400 Closes: https://bugs.gentoo.org/922038 Closes: https://bugs.gentoo.org/922142 Closes: https://bugs.gentoo.org/923368 Closes: https://bugs.gentoo.org/923750 Closes: https://bugs.gentoo.org/923841 Closes: https://bugs.gentoo.org/923852 Closes: https://bugs.gentoo.org/923854 Closes: https://bugs.gentoo.org/924192 Closes: https://bugs.gentoo.org/924273 Closes: https://bugs.gentoo.org/924585 Closes: https://bugs.gentoo.org/921380 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.62.ebuild | 246 +++++++++++++++++++++++++++++++++ 2 files changed, 247 insertions(+)
