Bug 916112 – sys-apps/portage: vardbapi.unpack_contents not compatible with multiprocessing spawn start method

• Product: Portage Development
• Status: RESOLVED FIXED

[Description Zac Medico 2023-10-22 09:01:37 UTC]

After I fixed bug 916106 and bug 916108, I found that vardbapi.unpack_contents contains an unpicklable _io.FileIO object that triggers this error with the multiprocessing spawn start method:

> Traceback (most recent call last):
>   File "/usr/lib/python3.10/asyncio/events.py", line 80, in _run
>     self._context.run(self._callback, *self._args)
>   File "portage/lib/_emerge/AsynchronousTask.py", line 209, in _exit_listener_cb
>     listener(self)
>   File "portage/lib/_emerge/Binpkg.py", line 431, in _unpack_contents_exit
>     unpack_contents.future.result()
>   File "portage/lib/portage/dbapi/bintree.py", line 391, in unpack_contents
>     await add_pkg._db.unpack_contents(pkg, dest_dir, loop=loop)
>   File "portage/lib/portage/dbapi/vartree.py", line 1101, in unpack_contents
>     excluded_config_files = await loop.run_in_executor(
>   File "/usr/lib/python3.10/asyncio/base_events.py", line 821, in run_in_executor
>     executor.submit(func, *args), loop=self)
>   File "portage/lib/portage/util/futures/executor/fork.py", line 46, in submit
>     self._schedule()
>   File "portage/lib/portage/util/futures/executor/fork.py", line 62, in _schedule
>     proc.start()
>   File "portage/lib/_emerge/AsynchronousTask.py", line 34, in start
>     self._start()
>   File "portage/lib/portage/util/_async/AsyncFunction.py", line 35, in _start
>     ForkProcess._start(self)
>   File "portage/lib/portage/util/_async/ForkProcess.py", line 61, in _start
>     retval = self._spawn(self.args, fd_pipes=self.fd_pipes)
>   File "portage/lib/portage/util/_async/ForkProcess.py", line 189, in _spawn
>     self._proc.start()
>   File "/usr/lib/python3.10/multiprocessing/process.py", line 121, in start
>     self._popen = self._Popen(self)
>   File "/usr/lib/python3.10/multiprocessing/context.py", line 224, in _Popen
>     return _default_context.get_context().Process._Popen(process_obj)
>   File "/usr/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
>     return Popen(process_obj)
>   File "/usr/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
>     super().__init__(process_obj)
>   File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
>     self._launch(process_obj)
>   File "/usr/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 47, in _launch
>     reduction.dump(process_obj, fp)
>   File "/usr/lib/python3.10/multiprocessing/reduction.py", line 60, in dump
>     ForkingPickler(file, protocol).dump(obj)
> TypeError: cannot pickle '_io.FileIO' object

[Comment 1 Zac Medico 2023-10-22 09:09:29 UTC]

It looks like it's the pw_file variable here:

> if binpkg_format == "xpak":
> 	tar_cmd = ("tar", "-x", "--xattrs", "--xattrs-include=*", "-C", dest_dir)
> 	pr, pw = os.pipe()
> 	proc = await asyncio.create_subprocess_exec(*tar_cmd, stdin=pr)
> 	os.close(pr)
> 	with os.fdopen(pw, "wb", 0) as pw_file:
> 		excluded_config_files = await loop.run_in_executor(
> 			ForkExecutor(loop=loop),
> 			functools.partial(
> 				self._dblink(cpv).quickpkg,
> 				pw_file,
> 				include_config=opts.include_config == "y",
> 				include_unmodified_config=opts.include_unmodified_config == "y",
> 			),
> 		)

[Comment 2 Larry the Git Cow 2023-10-22 20:58:45 UTC]

The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=0b9fbe389db3c3b9e625e1f5f526b3a921d2e0ce

commit 0b9fbe389db3c3b9e625e1f5f526b3a921d2e0ce
Author:     Zac Medico <[e-mail]>
AuthorDate: 2023-10-22 19:57:16 +0000
Commit:     Zac Medico <[e-mail]>
CommitDate: 2023-10-22 19:58:20 +0000

    vardbapi.unpack_contents: Use multiprocessing.Pipe for spawn compat
    
    Bug: https://bugs.gentoo.org/916112
    Signed-off-by: Zac Medico <[e-mail]>

 lib/portage/dbapi/vartree.py | 28 +++++++++++++++-------------
 1 file changed, 15 insertions(+), 13 deletions(-)

[Comment 3 Larry the Git Cow 2023-10-25 05:49:07 UTC]

The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=2b9128b401def05cac46f9e6d66048e5a0d888b8

commit 2b9128b401def05cac46f9e6d66048e5a0d888b8
Author:     Sam James <[e-mail]>
AuthorDate: 2023-10-25 05:48:31 +0000
Commit:     Sam James <[e-mail]>
CommitDate: 2023-10-25 05:48:40 +0000

    sys-apps/portage: add 3.0.54
    
    Closes: https://bugs.gentoo.org/905869
    Closes: https://bugs.gentoo.org/915551
    Closes: https://bugs.gentoo.org/915896
    Closes: https://bugs.gentoo.org/916106
    Closes: https://bugs.gentoo.org/916108
    Closes: https://bugs.gentoo.org/916112
    Closes: https://bugs.gentoo.org/916116
    Closes: https://bugs.gentoo.org/916139
    Closes: https://bugs.gentoo.org/916141
    Closes: https://bugs.gentoo.org/916142
    Closes: https://bugs.gentoo.org/916149
    Closes: https://bugs.gentoo.org/916182
    Closes: https://bugs.gentoo.org/916231
    Closes: https://bugs.gentoo.org/916235
    Closes: https://bugs.gentoo.org/916240
    Closes: https://bugs.gentoo.org/916242
    Closes: https://bugs.gentoo.org/916245
    Closes: https://bugs.gentoo.org/916248
    Closes: https://bugs.gentoo.org/899898
    Closes: https://bugs.gentoo.org/906129
    Closes: https://bugs.gentoo.org/906156
    Closes: https://bugs.gentoo.org/916031
    Signed-off-by: Sam James <[e-mail]>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-3.0.54.ebuild | 238 +++++++++++++++++++++++++++++++++
 2 files changed, 239 insertions(+)