I ran lib/portage/tests/emerge/test_simple.py with this patch applied to make it use the multiprocessing spawn start method. The self._scheduler.is_closed() thing is a workaround for a strange issue with the asyncio loop getting closed for an unknown reason: > diff --git a/bin/emerge b/bin/emerge > index f5dbc3106..b2afe74b2 100755 > --- a/bin/emerge > +++ b/bin/emerge > @@ -4,2 +4,5 @@ > > +import multiprocessing > +multiprocessing.set_start_method("spawn", force=True) > + > import os > diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py > index 88fc52577..203eaa3bb 100644 > --- a/lib/portage/dbapi/vartree.py > +++ b/lib/portage/dbapi/vartree.py > @@ -4282,2 +4282,7 @@ class dblink: > > + if self._scheduler.is_closed(): > + asyncio._thread_weakrefs.mainloop = None > + asyncio._thread_weakrefs.loops.clear() > + self._scheduler = SchedulerInterface(asyncio._safe_loop()) > + > # run instprep internal phase > diff --git a/lib/portage/tests/emerge/test_simple.py b/lib/portage/tests/emerge/test_simple.py > index 1cc6457ef..32fe3e501 100644 > --- a/lib/portage/tests/emerge/test_simple.py > +++ b/lib/portage/tests/emerge/test_simple.py > @@ -55,3 +55,3 @@ class SimpleEmergeTestCase(TestCase): > def testSimple(self): > - debug = False > + debug = True With the above patch, I can see that QueryCommand._db is missing when MergeProcess uses the multiprocessing spawn start method, as shown by this AttributeError that happens because QueryCommand._db is missing: > Traceback (most recent call last): > File "/usr/lib/python3.10/asyncio/events.py", line 80, in _run > self._context.run(self._callback, *self._args) > File "portage/lib/_emerge/EbuildIpcDaemon.py", line 55, in _input_handler > reply = cmd_handler(obj) > File "portage/lib/portage/package/ebuild/_ipc/QueryCommand.py", line 50, in __call__ > db = self.get_db() > File "portage/lib/portage/package/ebuild/_ipc/QueryCommand.py", line 29, in get_db > return portage.db > AttributeError: module 'portage' has no attribute 'db'
A possible solution is to have EbuildIpcDaemon proxy the requests through to the parent process via a duplex multiprocessing.Pipe.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=b9a85ff987ea677e350fb70e2c3355197ee6c39b commit b9a85ff987ea677e350fb70e2c3355197ee6c39b Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2023-10-22 08:08:02 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2023-10-22 21:29:49 +0000 MergeProcess: Support QueryCommand with spawn start method Since QueryCommand._db does not propagate to the child process in MergeProcess with the multiprocessing spawn start method, handle it by passing QueryCommand._db as an explicit parameter. Since self.mydbapi can be a bindbapi instance, override the bindbapi __getstate__ method so that it is picklable (omit the unpicklable _aux_cache_slot_dict attribute and regenerate it after unpickling). Bug: https://bugs.gentoo.org/916106 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/dbapi/_MergeProcess.py | 19 ++++++++++++++++--- lib/portage/dbapi/bintree.py | 20 ++++++++++++++++++-- 2 files changed, 34 insertions(+), 5 deletions(-)
(In reply to Zac Medico from comment #0) > The self._scheduler.is_closed() thing is a workaround for a strange issue > with the asyncio loop getting closed for an unknown reason: > > > diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py > > index 88fc52577..203eaa3bb 100644 > > --- a/lib/portage/dbapi/vartree.py > > +++ b/lib/portage/dbapi/vartree.py > > @@ -4282,2 +4282,7 @@ class dblink: > > > > + if self._scheduler.is_closed(): > > + asyncio._thread_weakrefs.mainloop = None > > + asyncio._thread_weakrefs.loops.clear() > > + self._scheduler = SchedulerInterface(asyncio._safe_loop()) > > + > > # run instprep internal phase It's actually this code in bin/emerge that triggered the issue: > finally: > global_event_loop().close() For the multiprocessing spawn start method, the event loop close needs to be protected inside a __name__ == "__main__" conditional. There's a similar issue in bin/ebuild.
Something is still not quite right about bindbapi pickling, because now it fails to pickle _aux_cache_slot_dict_cache in some cases. It may be that __gatestate__ is raising some kind of exception and preventing it from working. Also, the _instance_key attribute needs to be excluded from pickling since it refers to an instance method. The __setstate__ method can use the _multi_instance attribute to set the _instance_key attribute appropriately for the new instance.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=c7768ae4820201913ba28d5fad441c1acc4659c0 commit c7768ae4820201913ba28d5fad441c1acc4659c0 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2023-10-23 04:14:05 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2023-10-23 04:41:10 +0000 bindbapi: Fix pickling for MergeProcess Both the _aux_cache_slot_dict_cache and _instance_key attributes are unpicklable, so fix __getstate__ to exclude them both from the state. Override __setstate__ to initialize _instance_key appropriately. These changes prevent the following error: AttributeError: Can't pickle local object 'slot_dict_class.<locals>.LocalSlotDict' Fixes: b9a85ff987ea ("MergeProcess: Support QueryCommand with spawn start method") Bug: https://bugs.gentoo.org/916106 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/dbapi/bintree.py | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-)
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=2b9128b401def05cac46f9e6d66048e5a0d888b8 commit 2b9128b401def05cac46f9e6d66048e5a0d888b8 Author: Sam James <sam@gentoo.org> AuthorDate: 2023-10-25 05:48:31 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-10-25 05:48:40 +0000 sys-apps/portage: add 3.0.54 Closes: https://bugs.gentoo.org/905869 Closes: https://bugs.gentoo.org/915551 Closes: https://bugs.gentoo.org/915896 Closes: https://bugs.gentoo.org/916106 Closes: https://bugs.gentoo.org/916108 Closes: https://bugs.gentoo.org/916112 Closes: https://bugs.gentoo.org/916116 Closes: https://bugs.gentoo.org/916139 Closes: https://bugs.gentoo.org/916141 Closes: https://bugs.gentoo.org/916142 Closes: https://bugs.gentoo.org/916149 Closes: https://bugs.gentoo.org/916182 Closes: https://bugs.gentoo.org/916231 Closes: https://bugs.gentoo.org/916235 Closes: https://bugs.gentoo.org/916240 Closes: https://bugs.gentoo.org/916242 Closes: https://bugs.gentoo.org/916245 Closes: https://bugs.gentoo.org/916248 Closes: https://bugs.gentoo.org/899898 Closes: https://bugs.gentoo.org/906129 Closes: https://bugs.gentoo.org/906156 Closes: https://bugs.gentoo.org/916031 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.54.ebuild | 238 +++++++++++++++++++++++++++++++++ 2 files changed, 239 insertions(+)
