Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 916141

Summary: sys-apps/portage: BinpkgVerifier uses unpicklable generator expression as input to FileDigester
Product: Portage Development Reporter: Zac Medico <zmedico>
Component: CoreAssignee: Portage team <dev-portage>
Status: RESOLVED FIXED    
Severity: normal Keywords: InVCS
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on: 916031    
Bug Blocks: 914876    

Description Zac Medico gentoo-dev 2023-10-22 22:22:54 UTC
> Traceback (most recent call last):
>   File "portage/lib/_emerge/Scheduler.py", line 964, in _run_pkg_pretend
>     verifier.start()
>   File "portage/lib/_emerge/AsynchronousTask.py", line 34, in start
>     self._start()
>   File "portage/lib/_emerge/BinpkgVerifier.py", line 59, in _start
>     self._start_task(
>   File "portage/lib/_emerge/CompositeTask.py", line 111, in _start_task
>     task.start()
>   File "portage/lib/_emerge/AsynchronousTask.py", line 34, in start
>     self._start()
>   File "portage/lib/portage/util/_async/FileDigester.py", line 27, in _start
>     super()._start()
>   File "portage/lib/portage/util/_async/AsyncFunction.py", line 35, in _start
>     ForkProcess._start(self)
>   File "portage/lib/portage/util/_async/ForkProcess.py", line 61, in _start
>     retval = self._spawn(self.args, fd_pipes=self.fd_pipes)
>              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "portage/lib/portage/util/_async/ForkProcess.py", line 189, in _spawn
>     self._proc.start()
>   File "/usr/lib/python3.12/multiprocessing/process.py", line 121, in start
>     self._popen = self._Popen(self)
>                   ^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.12/multiprocessing/context.py", line 224, in _Popen
>     return _default_context.get_context().Process._Popen(process_obj)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.12/multiprocessing/context.py", line 289, in _Popen
>     return Popen(process_obj)
>            ^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.12/multiprocessing/popen_spawn_posix.py", line 32, in __init__
>     super().__init__(process_obj)
>   File "/usr/lib/python3.12/multiprocessing/popen_fork.py", line 19, in __init__
>     self._launch(process_obj)
>   File "/usr/lib/python3.12/multiprocessing/popen_spawn_posix.py", line 47, in _launch
>     reduction.dump(process_obj, fp)
>   File "/usr/lib/python3.12/multiprocessing/reduction.py", line 60, in dump
>     ForkingPickler(file, protocol).dump(obj)
> TypeError: cannot pickle 'generator' object

The fix is easy:

> diff --git a/lib/_emerge/BinpkgVerifier.py b/lib/_emerge/BinpkgVerifier.py
> index 58691e68b..2a45d3316 100644
> --- a/lib/_emerge/BinpkgVerifier.py
> +++ b/lib/_emerge/BinpkgVerifier.py
> @@ -59,7 +59,7 @@ class BinpkgVerifier(CompositeTask):
>          self._start_task(
>              FileDigester(
>                  file_path=self._pkg_path,
> -                hash_names=(k for k in digests if k != "size"),
> +                hash_names=[k for k in digests if k != "size"],
>                  background=self.background,
>                  logfile=self.logfile,
>                  scheduler=self.scheduler,
Comment 1 Larry the Git Cow gentoo-dev 2023-10-22 22:51:55 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=6963901bebf5403be5042fcbc54f4c950608204e

commit 6963901bebf5403be5042fcbc54f4c950608204e
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2023-10-22 22:50:23 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2023-10-22 22:51:28 +0000

    BinpkgVerifier: multiprocessing spawn compat
    
    Solves this error with the multiprocessing spawn start method:
    
    TypeError: cannot pickle 'generator' object
    
    Bug: https://bugs.gentoo.org/916141
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/BinpkgVerifier.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Comment 2 Larry the Git Cow gentoo-dev 2023-10-25 05:49:04 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=2b9128b401def05cac46f9e6d66048e5a0d888b8

commit 2b9128b401def05cac46f9e6d66048e5a0d888b8
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2023-10-25 05:48:31 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2023-10-25 05:48:40 +0000

    sys-apps/portage: add 3.0.54
    
    Closes: https://bugs.gentoo.org/905869
    Closes: https://bugs.gentoo.org/915551
    Closes: https://bugs.gentoo.org/915896
    Closes: https://bugs.gentoo.org/916106
    Closes: https://bugs.gentoo.org/916108
    Closes: https://bugs.gentoo.org/916112
    Closes: https://bugs.gentoo.org/916116
    Closes: https://bugs.gentoo.org/916139
    Closes: https://bugs.gentoo.org/916141
    Closes: https://bugs.gentoo.org/916142
    Closes: https://bugs.gentoo.org/916149
    Closes: https://bugs.gentoo.org/916182
    Closes: https://bugs.gentoo.org/916231
    Closes: https://bugs.gentoo.org/916235
    Closes: https://bugs.gentoo.org/916240
    Closes: https://bugs.gentoo.org/916242
    Closes: https://bugs.gentoo.org/916245
    Closes: https://bugs.gentoo.org/916248
    Closes: https://bugs.gentoo.org/899898
    Closes: https://bugs.gentoo.org/906129
    Closes: https://bugs.gentoo.org/906156
    Closes: https://bugs.gentoo.org/916031
    Signed-off-by: Sam James <sam@gentoo.org>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-3.0.54.ebuild | 238 +++++++++++++++++++++++++++++++++
 2 files changed, 239 insertions(+)