Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 916240 - sys-apps/portage: env-update not compatible with multiprocessing spawn
Summary: sys-apps/portage: env-update not compatible with multiprocessing spawn
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on: 916031
Blocks: 914876
  Show dependency tree
 
Reported: 2023-10-24 17:35 UTC by Zac Medico
Modified: 2023-10-25 05:49 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Zac Medico gentoo-dev 2023-10-24 17:35:29 UTC
The test lib/portage/tests/emerge/test_simple.py fails when bin/env-update is patched like this:

multiprocessing.set_start_method("spawn", force=True)

> Traceback (most recent call last):                                                                       
>   File "<string>", line 1, in <module>                                                                   
>   File "/usr/lib/python3.12/multiprocessing/spawn.py", line 122, in spawn_main
>     exitcode = _main(fd, parent_sentinel)                                                                
>                ^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.12/multiprocessing/spawn.py", line 131, in _main
>     prepare(preparation_data)                                                                            
>   File "/usr/lib/python3.12/multiprocessing/spawn.py", line 246, in prepare
>     _fixup_main_from_path(data['init_main_from_path'])
>   File "/usr/lib/python3.12/multiprocessing/spawn.py", line 297, in _fixup_main_from_path
>     main_content = runpy.run_path(main_path,                                                             
>                    ^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "<frozen runpy>", line 286, in run_path
>   File "<frozen runpy>", line 98, in _run_module_code
>   File "<frozen runpy>", line 88, in _run_code
>   File "portage/bin/env-update", line 42, in <module>
>     portage.env_update(makelinks)
>   File "portage/lib/portage/proxy/objectproxy.py", line 31, in __call__
>     return result(*args, **kwargs)                                                                       
>            ^^^^^^^^^^^^^^^^^^^^^^^
>   File "portage/lib/portage/util/env_update.py", line 79, in env_update
>     vardbapi._fs_lock()                                                                                  
>   File "portage/lib/portage/dbapi/vartree.py", line 313, in _fs_lock
>     self._fs_lock_obj = lockfile(self._conf_mem_file)                  
>                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "portage/lib/portage/locks.py", line 184, in lockfile
>     lock = _lockfile_iteration(
>            ^^^^^^^^^^^^^^^^^^^^
>   File "portage/lib/portage/locks.py", line 314, in _lockfile_iteration
>     locking_method = _get_lock_fn()
>                      ^^^^^^^^^^^^^^
>   File "portage/lib/portage/locks.py", line 96, in _get_lock_fn
>     if _test_lock_fn(_lockf_test_lock_fn):
>        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "portage/lib/portage/locks.py", line 128, in _test_lock_fn
>     proc.start()
>   File "/usr/lib/python3.12/multiprocessing/process.py", line 121, in start
>     self._popen = self._Popen(self)
>                   ^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.12/multiprocessing/context.py", line 224, in _Popen
>     return _default_context.get_context().Process._Popen(process_obj)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.12/multiprocessing/context.py", line 289, in _Popen
>     return Popen(process_obj)
>            ^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.12/multiprocessing/popen_spawn_posix.py", line 32, in __init__
>     super().__init__(process_obj)
>   File "/usr/lib/python3.12/multiprocessing/popen_fork.py", line 19, in __init__
>     self._launch(process_obj)
>   File "/usr/lib/python3.12/multiprocessing/popen_spawn_posix.py", line 42, in _launch
>     prep_data = spawn.get_preparation_data(process_obj._name)
>                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.12/multiprocessing/spawn.py", line 164, in get_preparation_data
>     _check_not_importing_main()
>   File "/usr/lib/python3.12/multiprocessing/spawn.py", line 140, in _check_not_importing_main
>     raise RuntimeError('''
> RuntimeError: 
>         An attempt has been made to start a new process before the
>         current process has finished its bootstrapping phase.
> 
>         This probably means that you are not using fork to start your
>         child processes and you have forgotten to use the proper idiom
>         in the main module:
> 
>             if __name__ == '__main__':
>                 freeze_support()
>                 ...
> 
>         The "freeze_support()" line can be omitted if the program
>         is not going to be frozen to produce an executable.
> 
>         To fix this issue, refer to the "Safe importing of main module"
>         section in https://docs.python.org/3/library/multiprocessing.html

It's easy to fix like this:

> diff --git a/bin/env-update b/bin/env-update
> index 6ba80c3ef..2b4afb8e5 100755
> --- a/bin/env-update
> +++ b/bin/env-update
> @@ -37,11 +38,12 @@ import portage
>  
>  portage._internal_caller = True
>  
> -try:
> -    portage.env_update(makelinks)
> -except OSError as e:
> -    if e.errno == errno.EACCES:
> -        print("env-update: Need superuser access")
> -        sys.exit(1)
> -    else:
> -        raise
> +if __name__ == "__main__":
> +       try:
> +               portage.env_update(makelinks)
> +       except OSError as e:
> +               if e.errno == errno.EACCES:
> +                       print("env-update: Need superuser access")
> +                       sys.exit(1)
> +               else:
> +                       raise
Comment 1 Larry the Git Cow gentoo-dev 2023-10-24 17:49:02 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=0706d3d3fe6268560efbe4f422b07f2155923168

commit 0706d3d3fe6268560efbe4f422b07f2155923168
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2023-10-24 17:48:15 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2023-10-24 17:48:39 +0000

    env-update: multiprocessing spawn compat
    
    Bug: https://bugs.gentoo.org/916240
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 bin/env-update | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)
Comment 2 Larry the Git Cow gentoo-dev 2023-10-25 05:49:13 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=2b9128b401def05cac46f9e6d66048e5a0d888b8

commit 2b9128b401def05cac46f9e6d66048e5a0d888b8
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2023-10-25 05:48:31 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2023-10-25 05:48:40 +0000

    sys-apps/portage: add 3.0.54
    
    Closes: https://bugs.gentoo.org/905869
    Closes: https://bugs.gentoo.org/915551
    Closes: https://bugs.gentoo.org/915896
    Closes: https://bugs.gentoo.org/916106
    Closes: https://bugs.gentoo.org/916108
    Closes: https://bugs.gentoo.org/916112
    Closes: https://bugs.gentoo.org/916116
    Closes: https://bugs.gentoo.org/916139
    Closes: https://bugs.gentoo.org/916141
    Closes: https://bugs.gentoo.org/916142
    Closes: https://bugs.gentoo.org/916149
    Closes: https://bugs.gentoo.org/916182
    Closes: https://bugs.gentoo.org/916231
    Closes: https://bugs.gentoo.org/916235
    Closes: https://bugs.gentoo.org/916240
    Closes: https://bugs.gentoo.org/916242
    Closes: https://bugs.gentoo.org/916245
    Closes: https://bugs.gentoo.org/916248
    Closes: https://bugs.gentoo.org/899898
    Closes: https://bugs.gentoo.org/906129
    Closes: https://bugs.gentoo.org/906156
    Closes: https://bugs.gentoo.org/916031
    Signed-off-by: Sam James <sam@gentoo.org>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-3.0.54.ebuild | 238 +++++++++++++++++++++++++++++++++
 2 files changed, 239 insertions(+)