Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 916245 - sys-apps/portage: test_auxdb.py incompatible with multiprocessing spawn start method
Summary: sys-apps/portage: test_auxdb.py incompatible with multiprocessing spawn start...
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Conceptual/Abstract Ideas (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on: 916031
Blocks: 914876
  Show dependency tree
 
Reported: 2023-10-24 19:34 UTC by Zac Medico
Modified: 2023-10-25 05:49 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Zac Medico gentoo-dev 2023-10-24 19:34:14 UTC 
When launching lib/portage/tests/dbapi/test_auxdb.py via this script that runs pytest with the multiprocessing spawn start method, it fails:

> import multiprocessing
> import sys
> import pytest
> if __name__ == "__main__":
>     multiprocessing.set_start_method("spawn", force=True)
>     sys.exit(pytest.console_main())

> ============================= test session starts ==============================
> platform linux -- Python 3.12.0, pytest-7.4.2, pluggy-1.3.0
> rootdir: portage
> collected 4 items
> 
> lib/portage/tests/dbapi/test_auxdb.py FFFF
> 
> =================================== FAILURES ===================================
> __________________________ AuxdbTestCase.test_anydbm ___________________________
> 
> self = <portage.tests.dbapi.test_auxdb.AuxdbTestCase testMethod=test_anydbm>
> 
>     def test_anydbm(self):
>         try:
>             from portage.cache.anydbm import database
>         except ImportError:
>             self.skipTest("dbm import failed")
> >       self._test_mod("portage.cache.anydbm.database", multiproc=False)
> 
> lib/portage/tests/dbapi/test_auxdb.py:16: 
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> lib/portage/tests/dbapi/test_auxdb.py:80: in _test_mod
>     loop.run_until_complete(loop.run_in_executor(ForkExecutor(), test_func))
> /usr/lib/python3.12/asyncio/base_events.py:840: in run_in_executor
>     executor.submit(func, *args), loop=self)
> lib/portage/util/futures/executor/fork.py:46: in submit
>     self._schedule()
> lib/portage/util/futures/executor/fork.py:62: in _schedule
>     proc.start()
> lib/_emerge/AsynchronousTask.py:34: in start
>     self._start()
> lib/portage/util/_async/AsyncFunction.py:35: in _start
>     ForkProcess._start(self)
> lib/portage/util/_async/ForkProcess.py:61: in _start
>     retval = self._spawn(self.args, fd_pipes=self.fd_pipes)
> lib/portage/util/_async/ForkProcess.py:189: in _spawn
>     self._proc.start()
> /usr/lib/python3.12/multiprocessing/process.py:121: in start
>     self._popen = self._Popen(self)
> /usr/lib/python3.12/multiprocessing/context.py:224: in _Popen
>     return _default_context.get_context().Process._Popen(process_obj)
> /usr/lib/python3.12/multiprocessing/context.py:289: in _Popen
>     return Popen(process_obj)
> /usr/lib/python3.12/multiprocessing/popen_spawn_posix.py:32: in __init__
>     super().__init__(process_obj)
> /usr/lib/python3.12/multiprocessing/popen_fork.py:19: in __init__
>     self._launch(process_obj)
> /usr/lib/python3.12/multiprocessing/popen_spawn_posix.py:47: in _launch
>     reduction.dump(process_obj, fp)
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> 
> obj = <Process name='Process-1' parent=2281947 initial>
> file = <_io.BytesIO object at 0x7f22467d6070>, protocol = None
> 
>     def dump(obj, file, protocol=None):
>         '''Replacement for pickle.dump() using ForkingPickler.'''
> >       ForkingPickler(file, protocol).dump(obj)
> E       AttributeError: Can't pickle local object 'AuxdbTestCase._test_mod.<locals>.test_func'
> 
> /usr/lib/python3.12/multiprocessing/reduction.py:60: AttributeError
> _______________________ AuxdbTestCase.test_flat_hash_md5 _______________________
> 
> self = <portage.tests.dbapi.test_auxdb.AuxdbTestCase testMethod=test_flat_hash_md5>
> 
>     def test_flat_hash_md5(self):
> >       self._test_mod("portage.cache.flat_hash.md5_database")
> 
> lib/portage/tests/dbapi/test_auxdb.py:19: 
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> lib/portage/tests/dbapi/test_auxdb.py:80: in _test_mod
>     loop.run_until_complete(loop.run_in_executor(ForkExecutor(), test_func))
> /usr/lib/python3.12/asyncio/base_events.py:840: in run_in_executor
>     executor.submit(func, *args), loop=self)
> lib/portage/util/futures/executor/fork.py:46: in submit
>     self._schedule()
> lib/portage/util/futures/executor/fork.py:62: in _schedule
>     proc.start()
> lib/_emerge/AsynchronousTask.py:34: in start
>     self._start()
> lib/portage/util/_async/AsyncFunction.py:35: in _start
>     ForkProcess._start(self)
> lib/portage/util/_async/ForkProcess.py:61: in _start
>     retval = self._spawn(self.args, fd_pipes=self.fd_pipes)
> lib/portage/util/_async/ForkProcess.py:189: in _spawn
>     self._proc.start()
> /usr/lib/python3.12/multiprocessing/process.py:121: in start
>     self._popen = self._Popen(self)
> /usr/lib/python3.12/multiprocessing/context.py:224: in _Popen
>     return _default_context.get_context().Process._Popen(process_obj)
> /usr/lib/python3.12/multiprocessing/context.py:289: in _Popen
>     return Popen(process_obj)
> /usr/lib/python3.12/multiprocessing/popen_spawn_posix.py:32: in __init__
>     super().__init__(process_obj)
> /usr/lib/python3.12/multiprocessing/popen_fork.py:19: in __init__
>     self._launch(process_obj)
> /usr/lib/python3.12/multiprocessing/popen_spawn_posix.py:47: in _launch
>     reduction.dump(process_obj, fp)
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> 
> obj = <Process name='Process-2' parent=2281947 initial>
> file = <_io.BytesIO object at 0x7f224623f510>, protocol = None
> 
>     def dump(obj, file, protocol=None):
>         '''Replacement for pickle.dump() using ForkingPickler.'''
> >       ForkingPickler(file, protocol).dump(obj)
> E       AttributeError: Can't pickle local object 'AuxdbTestCase._test_mod.<locals>.test_func'
> 
> /usr/lib/python3.12/multiprocessing/reduction.py:60: AttributeError
> ___________________________ AuxdbTestCase.test_sqite ___________________________
> 
> self = <portage.tests.dbapi.test_auxdb.AuxdbTestCase testMethod=test_sqite>
> 
>     def test_sqite(self):
>         try:
>             import sqlite3
>         except ImportError:
>             self.skipTest("sqlite3 import failed")
> >       self._test_mod("portage.cache.sqlite.database")
> 
> lib/portage/tests/dbapi/test_auxdb.py:29: 
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> lib/portage/tests/dbapi/test_auxdb.py:80: in _test_mod
>     loop.run_until_complete(loop.run_in_executor(ForkExecutor(), test_func))
> /usr/lib/python3.12/asyncio/base_events.py:840: in run_in_executor
>     executor.submit(func, *args), loop=self)
> lib/portage/util/futures/executor/fork.py:46: in submit
>     self._schedule()
> lib/portage/util/futures/executor/fork.py:62: in _schedule
>     proc.start()
> lib/_emerge/AsynchronousTask.py:34: in start
>     self._start()
> lib/portage/util/_async/AsyncFunction.py:35: in _start
>     ForkProcess._start(self)
> lib/portage/util/_async/ForkProcess.py:61: in _start
>     retval = self._spawn(self.args, fd_pipes=self.fd_pipes)
> lib/portage/util/_async/ForkProcess.py:189: in _spawn
>     self._proc.start()
> /usr/lib/python3.12/multiprocessing/process.py:121: in start
>     self._popen = self._Popen(self)
> /usr/lib/python3.12/multiprocessing/context.py:224: in _Popen
>     return _default_context.get_context().Process._Popen(process_obj)
> /usr/lib/python3.12/multiprocessing/context.py:289: in _Popen
>     return Popen(process_obj)
> /usr/lib/python3.12/multiprocessing/popen_spawn_posix.py:32: in __init__
>     super().__init__(process_obj)
> /usr/lib/python3.12/multiprocessing/popen_fork.py:19: in __init__
>     self._launch(process_obj)
> /usr/lib/python3.12/multiprocessing/popen_spawn_posix.py:47: in _launch
>     reduction.dump(process_obj, fp)
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> 
> obj = <Process name='Process-3' parent=2281947 initial>
> file = <_io.BytesIO object at 0x7f224666e7f0>, protocol = None
> 
>     def dump(obj, file, protocol=None):
>         '''Replacement for pickle.dump() using ForkingPickler.'''
> >       ForkingPickler(file, protocol).dump(obj)
> E       AttributeError: Can't pickle local object 'AuxdbTestCase._test_mod.<locals>.test_func'
> 
> /usr/lib/python3.12/multiprocessing/reduction.py:60: AttributeError
> _________________________ AuxdbTestCase.test_volatile __________________________
> 
> self = <portage.tests.dbapi.test_auxdb.AuxdbTestCase testMethod=test_volatile>
> 
>     def test_volatile(self):
> >       self._test_mod("portage.cache.volatile.database", multiproc=False)
> 
> lib/portage/tests/dbapi/test_auxdb.py:22: 
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> lib/portage/tests/dbapi/test_auxdb.py:80: in _test_mod
>     loop.run_until_complete(loop.run_in_executor(ForkExecutor(), test_func))
> /usr/lib/python3.12/asyncio/base_events.py:840: in run_in_executor
>     executor.submit(func, *args), loop=self)
> lib/portage/util/futures/executor/fork.py:46: in submit
>     self._schedule()
> lib/portage/util/futures/executor/fork.py:62: in _schedule
>     proc.start()
> lib/_emerge/AsynchronousTask.py:34: in start
>     self._start()
> lib/portage/util/_async/AsyncFunction.py:35: in _start
>     ForkProcess._start(self)
> lib/portage/util/_async/ForkProcess.py:61: in _start
>     retval = self._spawn(self.args, fd_pipes=self.fd_pipes)
> lib/portage/util/_async/ForkProcess.py:189: in _spawn
>     self._proc.start()
> /usr/lib/python3.12/multiprocessing/process.py:121: in start
>     self._popen = self._Popen(self)
> /usr/lib/python3.12/multiprocessing/context.py:224: in _Popen
>     return _default_context.get_context().Process._Popen(process_obj)
> /usr/lib/python3.12/multiprocessing/context.py:289: in _Popen
>     return Popen(process_obj)
> /usr/lib/python3.12/multiprocessing/popen_spawn_posix.py:32: in __init__
>     super().__init__(process_obj)
> /usr/lib/python3.12/multiprocessing/popen_fork.py:19: in __init__
>     self._launch(process_obj)
> /usr/lib/python3.12/multiprocessing/popen_spawn_posix.py:47: in _launch
>     reduction.dump(process_obj, fp)
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> 
> obj = <Process name='Process-4' parent=2281947 initial>
> file = <_io.BytesIO object at 0x7f224667e8e0>, protocol = None
> 
>     def dump(obj, file, protocol=None):
>         '''Replacement for pickle.dump() using ForkingPickler.'''
> >       ForkingPickler(file, protocol).dump(obj)
> E       AttributeError: Can't pickle local object 'AuxdbTestCase._test_mod.<locals>.test_func'
> 
> /usr/lib/python3.12/multiprocessing/reduction.py:60: AttributeError
> =========================== short test summary info ============================
> FAILED lib/portage/tests/dbapi/test_auxdb.py::AuxdbTestCase::test_anydbm - At...
> FAILED lib/portage/tests/dbapi/test_auxdb.py::AuxdbTestCase::test_flat_hash_md5
> FAILED lib/portage/tests/dbapi/test_auxdb.py::AuxdbTestCase::test_sqite - Att...
> FAILED lib/portage/tests/dbapi/test_auxdb.py::AuxdbTestCase::test_volatile - ...
> ============================== 4 failed in 2.35s ===============================
Comment 1 Zac Medico gentoo-dev 2023-10-24 23:39:54 UTC 
After fixing the unpicklable local functions, there's this:

FAILED lib/portage/tests/dbapi/test_auxdb.py::AuxdbTestCase::test_anydbm - TypeError: cannot pickle '_dbm.dbm' object
FAILED lib/portage/tests/dbapi/test_auxdb.py::AuxdbTestCase::test_sqite - TypeError: cannot pickle 'module' object
Comment 2 Larry the Git Cow gentoo-dev 2023-10-25 01:35:10 UTC 
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=d7e64317e3a0d789e17dfea7b451c6b3b6cdb3c0

commit d7e64317e3a0d789e17dfea7b451c6b3b6cdb3c0
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2023-10-24 23:52:46 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2023-10-24 23:56:15 +0000

    test_auxdb: multiprocessing spawn compat
    
    Use staticmethod and functools.partial to avoid unpicklable
    local functions. Also, don't try to pickle anydbm or sqlite
    modules since they currently are not picklable. Ultimately,
    it might be a good idea to implement pickling for the sqlite
    module.
    
    Bug: https://bugs.gentoo.org/916245
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/tests/dbapi/test_auxdb.py | 103 ++++++++++++++++++++++------------
 1 file changed, 66 insertions(+), 37 deletions(-)
Comment 3 Larry the Git Cow gentoo-dev 2023-10-25 05:49:15 UTC 
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=2b9128b401def05cac46f9e6d66048e5a0d888b8

commit 2b9128b401def05cac46f9e6d66048e5a0d888b8
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2023-10-25 05:48:31 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2023-10-25 05:48:40 +0000

    sys-apps/portage: add 3.0.54
    
    Closes: https://bugs.gentoo.org/905869
    Closes: https://bugs.gentoo.org/915551
    Closes: https://bugs.gentoo.org/915896
    Closes: https://bugs.gentoo.org/916106
    Closes: https://bugs.gentoo.org/916108
    Closes: https://bugs.gentoo.org/916112
    Closes: https://bugs.gentoo.org/916116
    Closes: https://bugs.gentoo.org/916139
    Closes: https://bugs.gentoo.org/916141
    Closes: https://bugs.gentoo.org/916142
    Closes: https://bugs.gentoo.org/916149
    Closes: https://bugs.gentoo.org/916182
    Closes: https://bugs.gentoo.org/916231
    Closes: https://bugs.gentoo.org/916235
    Closes: https://bugs.gentoo.org/916240
    Closes: https://bugs.gentoo.org/916242
    Closes: https://bugs.gentoo.org/916245
    Closes: https://bugs.gentoo.org/916248
    Closes: https://bugs.gentoo.org/899898
    Closes: https://bugs.gentoo.org/906129
    Closes: https://bugs.gentoo.org/906156
    Closes: https://bugs.gentoo.org/916031
    Signed-off-by: Sam James <sam@gentoo.org>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-3.0.54.ebuild | 238 +++++++++++++++++++++++++++++++++
 2 files changed, 239 insertions(+)