We need to replace unsafe os.fork() in check_locale for bug 914876, and also somehow move it out of async context. We can't really make it wait asynchronously like we did for set_term_size in bug 923750 since it's supposed to raise an error on failure: https://gitweb.gentoo.org/proj/portage.git/commit/?id=e5ba8d096e56495a9b516cea818d48e104d62bca commit e5ba8d096e56495a9b516cea818d48e104d62bca Author: Michał Górny <mgorny@gentoo.org> Date: Sun Nov 15 22:03:42 2015 +0100 EAPI 6: Enforce posixish LC_CTYPE
For multiprocessing start method spawn, we might pass down portage.util.locale._check_locale_cache to child processes via ForkProcess._bootstrap, since normally the lib/_emerge/actions.py calls check_locale from a non-async context, and we can later use the cached result in async contexts of child processes.
Since check_locale doesn't cache results unless an env argument is given, we'll want the check_locale call in lib/_emerge/actions.py to pass in an env argument.
(In reply to Zac Medico from comment #2) > Since check_locale doesn't cache results unless an env argument is given, > we'll want the check_locale call in lib/_emerge/actions.py to pass in an env > argument. Not sure what the correct way is to generate mylocale for the cache key in this case, since it uses whatever the current locale is. Since locale.getlocale() returns a tuple of (language code, encoding). Maybe we can simply use ".".join(locale.getlocale()) for mylocale.
If we move the code from config.environ() to someplace like the EbuildPhase class then we can make it do proper async/await for the the child process. I tried generating a pre-cache in lib/_emerge/actions.py, but ultimately I wasn't happy with it because the pre-cache did not account for possible locale settings in package.env, so it did not guarantee cache hits.
I've fixed it to use async check_locale calls in https://github.com/gentoo/portage/pull/1254 now.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=c95fc64abf9698263090b3ffd4a056e989dd2be1 commit c95fc64abf9698263090b3ffd4a056e989dd2be1 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-02-09 06:38:41 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-02-09 08:19:00 +0000 EbuildPhase: async_check_locale Change config.environ() check_locale calls to async_check_locale calls in the EbuildPhase _async_start method in order to eliminate synchronous waiting for child processes in the main event loop thread. Bug: https://bugs.gentoo.org/923841 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/EbuildMetadataPhase.py | 21 ++++++++++++++++++++ lib/_emerge/EbuildPhase.py | 28 ++++++++++++++++++++++++++- lib/portage/package/ebuild/config.py | 26 +++++++++++-------------- lib/portage/util/futures/_asyncio/__init__.py | 9 +++++++++ lib/portage/util/locale.py | 28 ++++++++++++++++++--------- 5 files changed, 87 insertions(+), 25 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=ba33bc425363977eaf549a087b2469720a79e2a4 commit ba33bc425363977eaf549a087b2469720a79e2a4 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-02-08 06:46:04 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-02-09 08:19:00 +0000 check_locale: Use multiprocessing.Process instead of os.fork() Since os.fork() is unsafe in threaded processes, use multiprocessing.Process instead. This way the fork will be eliminated when the default multiprocessing start method changes to "spawn". TODO: Make async version of check_locale and call it from EbuildPhase instead of config.environ(), since it's bad to synchronously wait for the process in the main event loop thread where config.environ() tends to be called. Bug: https://bugs.gentoo.org/923841 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/util/locale.py | 56 +++++++++++++++++++++++++++------------------- 1 file changed, 33 insertions(+), 23 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=5c528b1cf44f30d80a3ca5620a810e4fe2bd66f1 commit 5c528b1cf44f30d80a3ca5620a810e4fe2bd66f1 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-02-13 04:47:53 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-02-13 05:02:14 +0000 Revert "EbuildPhase: async_check_locale" This reverts commit c95fc64abf9698263090b3ffd4a056e989dd2be1 since we had assumed EbuildMetadataPhase._start would serialize access to the portdbapi doebuild_settings attribute, and that assumption broke when _async_start was introduced in order to call async_check_locale. Bug: https://bugs.gentoo.org/923841 Bug: https://bugs.gentoo.org/924319 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/EbuildMetadataPhase.py | 21 -------------------- lib/_emerge/EbuildPhase.py | 28 +-------------------------- lib/portage/package/ebuild/config.py | 26 ++++++++++++++----------- lib/portage/util/futures/_asyncio/__init__.py | 9 --------- lib/portage/util/locale.py | 28 +++++++++------------------ 5 files changed, 25 insertions(+), 87 deletions(-)
(In reply to Larry the Git Cow from comment #7) > The bug has been referenced in the following commit(s): > > https://gitweb.gentoo.org/proj/portage.git/commit/ > ?id=5c528b1cf44f30d80a3ca5620a810e4fe2bd66f1 > > commit 5c528b1cf44f30d80a3ca5620a810e4fe2bd66f1 > Author: Zac Medico <zmedico@gentoo.org> > AuthorDate: 2024-02-13 04:47:53 +0000 > Commit: Zac Medico <zmedico@gentoo.org> > CommitDate: 2024-02-13 05:02:14 +0000 > > Revert "EbuildPhase: async_check_locale" Addition of an _async_start method has historically been troublesome, like when I reverted changes in https://github.com/gentoo/portage/pull/541 due to bug 716636. In that case, the root problem was only an unsafe call to remove_reader in this commit: https://gitweb.gentoo.org/proj/portage.git/commit/?id=27712651aa7014a960b012dc89457df09677edc1 The problem was later corrected with a safe remove_reader call in this fixed version of the same commit: https://gitweb.gentoo.org/proj/portage.git/commit/?id=72ac22e722549833c1ee7e7ad1b585db55f7dafc
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=18cdb6331a66c1cc92f296b1aaf0538f63586875 commit 18cdb6331a66c1cc92f296b1aaf0538f63586875 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-02-13 08:44:50 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-02-21 15:27:31 +0000 EbuildPhase: async_check_locale Change config.environ() check_locale calls to async_check_locale calls in the EbuildPhase _async_start method in order to eliminate synchronous waiting for child processes in the main event loop thread. Bug: https://bugs.gentoo.org/923841 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/EbuildMetadataPhase.py | 4 ++++ lib/_emerge/EbuildPhase.py | 28 ++++++++++++++++++++++++++- lib/portage/package/ebuild/config.py | 26 +++++++++++-------------- lib/portage/util/futures/_asyncio/__init__.py | 9 +++++++++ lib/portage/util/locale.py | 28 ++++++++++++++++++--------- 5 files changed, 70 insertions(+), 25 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=789493d521eb09f4aca2a8552b5262e2c0f5bc51 commit 789493d521eb09f4aca2a8552b5262e2c0f5bc51 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-02-13 08:19:59 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-02-21 15:27:31 +0000 EbuildMetadataPhase: Migrate to _async_start Bug: https://bugs.gentoo.org/923841 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/EbuildMetadataPhase.py | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-)
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=77c44c46194922509bc4f2b5cfc099412a560a69 commit 77c44c46194922509bc4f2b5cfc099412a560a69 Author: Sam James <sam@gentoo.org> AuthorDate: 2024-02-22 07:23:40 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-02-22 07:23:50 +0000 sys-apps/portage: add 3.0.62 Closes: https://bugs.gentoo.org/663324 Closes: https://bugs.gentoo.org/728046 Closes: https://bugs.gentoo.org/891137 Closes: https://bugs.gentoo.org/906368 Closes: https://bugs.gentoo.org/916566 Closes: https://bugs.gentoo.org/921170 Closes: https://bugs.gentoo.org/921208 Closes: https://bugs.gentoo.org/921400 Closes: https://bugs.gentoo.org/922038 Closes: https://bugs.gentoo.org/922142 Closes: https://bugs.gentoo.org/923368 Closes: https://bugs.gentoo.org/923750 Closes: https://bugs.gentoo.org/923841 Closes: https://bugs.gentoo.org/923852 Closes: https://bugs.gentoo.org/923854 Closes: https://bugs.gentoo.org/924192 Closes: https://bugs.gentoo.org/924273 Closes: https://bugs.gentoo.org/924585 Closes: https://bugs.gentoo.org/921380 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.62.ebuild | 246 +++++++++++++++++++++++++++++++++ 2 files changed, 247 insertions(+)
