Bug 916231 – sys-apps/portage: SyncManager usage of AsyncFunction not compatible with multprocessing spawn start method

• Product: Portage Development
• Status: RESOLVED FIXED

[Description Zac Medico 2023-10-24 15:57:01 UTC]

Using this script:

> import contextlib
> import multiprocessing
> import unittest
> 
> from portage.tests.conftest import prepare_environment
> from portage.tests.sync.test_sync_local import SyncLocalTestCase
> 
> if __name__ == "__main__":
>     multiprocessing.set_start_method("spawn", force=True)
>     with contextlib.contextmanager(prepare_environment.__wrapped__)():
>         unittest.main()

With emerge patched like this:

> --- a/bin/emerge
> +++ b/bin/emerge
> @@ -2,6 +2,9 @@
>  # Copyright 2006-2023 Gentoo Authors
>  # Distributed under the terms of the GNU General Public License v2
>  
> +import multiprocessing
> +multiprocessing.set_start_method("spawn", force=True)
> +
>  import os
>  import signal
>  import sys

It currently fails like this:

> Traceback (most recent call last):
>   File "portage/bin/emerge", line 60, in main
>     retval = emerge_main()
>              ^^^^^^^^^^^^^
>   File "portage/lib/_emerge/main.py", line 1306, in emerge_main
>     return run_action(emerge_config)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "portage/lib/_emerge/actions.py", line 3845, in run_action
>     return action_sync(emerge_config)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "portage/lib/_emerge/actions.py", line 2414, in action_sync
>     success, msgs = syncer.auto_sync(options=options)
>                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "portage/lib/portage/emaint/modules/sync/sync.py", line 89, in auto_sync
>     return self._sync(repos, return_messages, emaint_opts=options)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "portage/lib/portage/emaint/modules/sync/sync.py", line 234, in _sync
>     sync_scheduler.start()
>   File "portage/lib/_emerge/AsynchronousTask.py", line 34, in start
>     self._start()
>   File "portage/lib/portage/util/_async/AsyncScheduler.py", line 94, in _start
>     self._schedule()
>   File "portage/lib/_emerge/PollScheduler.py", line 151, in _schedule
>     self._schedule_tasks()
>   File "portage/lib/portage/util/_async/AsyncScheduler.py", line 66, in _schedule_tasks
>     task.start()
>   File "portage/lib/_emerge/AsynchronousTask.py", line 34, in start
>     self._start()
>   File "portage/lib/portage/sync/controller.py", line 399, in _start
>     self._start_task(self.sync_task, self._sync_task_exit)
>   File "portage/lib/_emerge/CompositeTask.py", line 111, in _start_task
>     task.start()
>   File "portage/lib/_emerge/AsynchronousTask.py", line 34, in start
>     self._start()
>   File "portage/lib/portage/util/_async/AsyncFunction.py", line 35, in _start
>     ForkProcess._start(self)
>   File "portage/lib/portage/util/_async/ForkProcess.py", line 61, in _start
>     retval = self._spawn(self.args, fd_pipes=self.fd_pipes)
>              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "portage/lib/portage/util/_async/ForkProcess.py", line 189, in _spawn
>     self._proc.start()
>   File "/usr/lib/python3.11/multiprocessing/process.py", line 121, in start
>     self._popen = self._Popen(self)
>                   ^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.11/multiprocessing/context.py", line 224, in _Popen
>     return _default_context.get_context().Process._Popen(process_obj)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.11/multiprocessing/context.py", line 288, in _Popen
>     return Popen(process_obj)
>            ^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 32, in __init__
>     super().__init__(process_obj)
>   File "/usr/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__
>     self._launch(process_obj)
>   File "/usr/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 47, in _launch
>     reduction.dump(process_obj, fp)
>   File "/usr/lib/python3.11/multiprocessing/reduction.py", line 60, in dump
>     ForkingPickler(file, protocol).dump(obj)
> TypeError: cannot pickle 'module' object

[Comment 1 Larry the Git Cow 2023-10-24 17:45:16 UTC]

The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=cebe9363cc348cea8d997afc638abf38ca79ed47

commit cebe9363cc348cea8d997afc638abf38ca79ed47
Author:     Zac Medico <[e-mail]>
AuthorDate: 2023-10-24 16:39:29 +0000
Commit:     Zac Medico <[e-mail]>
CommitDate: 2023-10-24 16:41:15 +0000

    SyncManager: multiprocessing spawn compat
    
    Tested via lib/portage/tests/ebuild/test_fetch.py with
    bin/emerge patched to call:
    
    multiprocessing.set_start_method("spawn", force=True)
    
    Bug: https://bugs.gentoo.org/916231
    Signed-off-by: Zac Medico <[e-mail]>

 lib/portage/sync/controller.py | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

[Comment 2 Larry the Git Cow 2023-10-25 05:49:10 UTC]

The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=2b9128b401def05cac46f9e6d66048e5a0d888b8

commit 2b9128b401def05cac46f9e6d66048e5a0d888b8
Author:     Sam James <[e-mail]>
AuthorDate: 2023-10-25 05:48:31 +0000
Commit:     Sam James <[e-mail]>
CommitDate: 2023-10-25 05:48:40 +0000

    sys-apps/portage: add 3.0.54
    
    Closes: https://bugs.gentoo.org/905869
    Closes: https://bugs.gentoo.org/915551
    Closes: https://bugs.gentoo.org/915896
    Closes: https://bugs.gentoo.org/916106
    Closes: https://bugs.gentoo.org/916108
    Closes: https://bugs.gentoo.org/916112
    Closes: https://bugs.gentoo.org/916116
    Closes: https://bugs.gentoo.org/916139
    Closes: https://bugs.gentoo.org/916141
    Closes: https://bugs.gentoo.org/916142
    Closes: https://bugs.gentoo.org/916149
    Closes: https://bugs.gentoo.org/916182
    Closes: https://bugs.gentoo.org/916231
    Closes: https://bugs.gentoo.org/916235
    Closes: https://bugs.gentoo.org/916240
    Closes: https://bugs.gentoo.org/916242
    Closes: https://bugs.gentoo.org/916245
    Closes: https://bugs.gentoo.org/916248
    Closes: https://bugs.gentoo.org/899898
    Closes: https://bugs.gentoo.org/906129
    Closes: https://bugs.gentoo.org/906156
    Closes: https://bugs.gentoo.org/916031
    Signed-off-by: Sam James <[e-mail]>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-3.0.54.ebuild | 238 +++++++++++++++++++++++++++++++++
 2 files changed, 239 insertions(+)