Summary: | sys-apps/portage: QueryCommand._db is missing when MergeProcess uses the multiprocessing spawn start method | ||
---|---|---|---|
Product: | Portage Development | Reporter: | Zac Medico <zmedico> |
Component: | Core | Assignee: | Portage team <dev-portage> |
Status: | RESOLVED FIXED | ||
Severity: | normal | Keywords: | InVCS |
Priority: | Normal | ||
Version: | unspecified | ||
Hardware: | All | ||
OS: | Linux | ||
See Also: |
https://github.com/gentoo/portage/pull/1141 https://github.com/gentoo/portage/pull/1151 https://bugs.gentoo.org/show_bug.cgi?id=933297 |
||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Bug Depends on: | 916031 | ||
Bug Blocks: | 914876 |
Description
Zac Medico
![]() A possible solution is to have EbuildIpcDaemon proxy the requests through to the parent process via a duplex multiprocessing.Pipe. The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=b9a85ff987ea677e350fb70e2c3355197ee6c39b commit b9a85ff987ea677e350fb70e2c3355197ee6c39b Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2023-10-22 08:08:02 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2023-10-22 21:29:49 +0000 MergeProcess: Support QueryCommand with spawn start method Since QueryCommand._db does not propagate to the child process in MergeProcess with the multiprocessing spawn start method, handle it by passing QueryCommand._db as an explicit parameter. Since self.mydbapi can be a bindbapi instance, override the bindbapi __getstate__ method so that it is picklable (omit the unpicklable _aux_cache_slot_dict attribute and regenerate it after unpickling). Bug: https://bugs.gentoo.org/916106 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/dbapi/_MergeProcess.py | 19 ++++++++++++++++--- lib/portage/dbapi/bintree.py | 20 ++++++++++++++++++-- 2 files changed, 34 insertions(+), 5 deletions(-) (In reply to Zac Medico from comment #0) > The self._scheduler.is_closed() thing is a workaround for a strange issue > with the asyncio loop getting closed for an unknown reason: > > > diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py > > index 88fc52577..203eaa3bb 100644 > > --- a/lib/portage/dbapi/vartree.py > > +++ b/lib/portage/dbapi/vartree.py > > @@ -4282,2 +4282,7 @@ class dblink: > > > > + if self._scheduler.is_closed(): > > + asyncio._thread_weakrefs.mainloop = None > > + asyncio._thread_weakrefs.loops.clear() > > + self._scheduler = SchedulerInterface(asyncio._safe_loop()) > > + > > # run instprep internal phase It's actually this code in bin/emerge that triggered the issue: > finally: > global_event_loop().close() For the multiprocessing spawn start method, the event loop close needs to be protected inside a __name__ == "__main__" conditional. There's a similar issue in bin/ebuild. Something is still not quite right about bindbapi pickling, because now it fails to pickle _aux_cache_slot_dict_cache in some cases. It may be that __gatestate__ is raising some kind of exception and preventing it from working. Also, the _instance_key attribute needs to be excluded from pickling since it refers to an instance method. The __setstate__ method can use the _multi_instance attribute to set the _instance_key attribute appropriately for the new instance. The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=c7768ae4820201913ba28d5fad441c1acc4659c0 commit c7768ae4820201913ba28d5fad441c1acc4659c0 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2023-10-23 04:14:05 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2023-10-23 04:41:10 +0000 bindbapi: Fix pickling for MergeProcess Both the _aux_cache_slot_dict_cache and _instance_key attributes are unpicklable, so fix __getstate__ to exclude them both from the state. Override __setstate__ to initialize _instance_key appropriately. These changes prevent the following error: AttributeError: Can't pickle local object 'slot_dict_class.<locals>.LocalSlotDict' Fixes: b9a85ff987ea ("MergeProcess: Support QueryCommand with spawn start method") Bug: https://bugs.gentoo.org/916106 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/dbapi/bintree.py | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=2b9128b401def05cac46f9e6d66048e5a0d888b8 commit 2b9128b401def05cac46f9e6d66048e5a0d888b8 Author: Sam James <sam@gentoo.org> AuthorDate: 2023-10-25 05:48:31 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-10-25 05:48:40 +0000 sys-apps/portage: add 3.0.54 Closes: https://bugs.gentoo.org/905869 Closes: https://bugs.gentoo.org/915551 Closes: https://bugs.gentoo.org/915896 Closes: https://bugs.gentoo.org/916106 Closes: https://bugs.gentoo.org/916108 Closes: https://bugs.gentoo.org/916112 Closes: https://bugs.gentoo.org/916116 Closes: https://bugs.gentoo.org/916139 Closes: https://bugs.gentoo.org/916141 Closes: https://bugs.gentoo.org/916142 Closes: https://bugs.gentoo.org/916149 Closes: https://bugs.gentoo.org/916182 Closes: https://bugs.gentoo.org/916231 Closes: https://bugs.gentoo.org/916235 Closes: https://bugs.gentoo.org/916240 Closes: https://bugs.gentoo.org/916242 Closes: https://bugs.gentoo.org/916245 Closes: https://bugs.gentoo.org/916248 Closes: https://bugs.gentoo.org/899898 Closes: https://bugs.gentoo.org/906129 Closes: https://bugs.gentoo.org/906156 Closes: https://bugs.gentoo.org/916031 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.54.ebuild | 238 +++++++++++++++++++++++++++++++++ 2 files changed, 239 insertions(+) |