When you install glibc from a binpkg, you still have to go through the entire locale generation procedure. For a normal user, they can populate /etc/locale.gen to slim this process down, but when building releases, we can't do this, so generating all locales each time gets *very* annoying. Can the locale generation be moved into one of the src_*() functions instead of pkg_postinst() to avoid this needless regeneration? It would also be nice if it could be parallelized, but that won't be as big of a deal if it only needs to be done once.
(In reply to comment #0) > It would also be nice if it could be parallelized, but that won't be as big of a > deal if it only needs to be done once. Well, see Bug 137706 for this...
there was a reason i moved this code to pkg_postinst() rather than src_install(), i just cant remember why right now :P probably related to the fact that you cannot cross-compile locales, so if you ran it in src_install(), it'd segfault on you
Does pkg_postinst() not get executed when building glibc with crossdev? Or is there something more to it?
in the cross-compile toolchain scenario the locale step is skipped on purpose
Perhaps we can use some kind of system to keep a list on-disk (separate from locales.gen) that shows what's already been built, then glibc can compare them. Andrew is right, on arches like mips, it takes longer to do the glibc locales than to do everything else when building with catalyst and a full cache. On faster architectures, it's just an annoyance, but one that we can surely do without.
> Perhaps we can use some kind of system to keep a list on-disk (separate from > locales.gen) that shows what's already been built although you can already do this (`locale-gen -u`), this isnt going to be added to the ebuild as you cant [easily] tell whether the new locale files installed differ from the binary generated ones already on disk in short: locales are always going to be generated, it's just a matter of when storing all the locales increases the binpkg size significantly btw, but i guess for slower arches it's a lot easier to sacrifice disk space than cpu time
so here's the problem you cannot utilize locale.gen from ROOT while in src_install ... that means if you want to run locale-gen inside of src_install, you need to generate all locales this in turn screws everyone else as now their systems will generate all locales only to turn around and throw them away in pkg_postinst()
Created attachment 108463 [details, diff] run locale-gen in src_install sanely here's how you'd do it sanely ... however, you still need to figure out how to avoid the aforementioned issue you could utilize like the build or bootstrap USE flags, but that'd only help you in stage1/stage2 generation ... going from a stage2 to a stage3 is pretty much just like a user upgrading, so you're going to still be screwed
a possibility for us to investigate: http://sourceware.org/ml/libc-ports/2008-01/msg00011.html this addresses the cross-compiling / endianness issue we could consider tying the remaining behavior to "binpkg" or "build" ... would that be usable at all for release building of stages ?
If we made it do locale generation without reading locale.gen in src_install if USE=build or USE=bootstrap, that would save us a good bit of time. It would/could still be done in pkg_* for USE="-build -bootstrap" so users wouldn't notice a difference, except for those building from stage1, and they can suffer a few extra minutes if they're willing to do that sort of an install, anyway.
(In reply to comment #2) > there was a reason i moved this code to pkg_postinst() rather than > src_install(), i just cant remember why right now :P > > probably related to the fact that you cannot cross-compile locales, so if you > ran it in src_install(), it'd segfault on you > There's another small problem with having locale-generating code in pkg_postinst(). Namely, when e.g. glibc-2.7-r2 is installed as a tbz2 (binary) on a system without headers (or a compiler for that matter) pkg_postinst() fails. In my case, I wasn't actually cross-compiling the package either because My build machine is a server somewhere at my office, and my host machine is my EEE 701. It would be nice if all locales could be pre-generated on the build system, and then screened on the host system, in the case where cross-compilation is not happening. Regarding cross-compilation with the local-generating code in pkg_postinst(), I'm just puzzled... typically on an embedded system one wouldn't find all of the necessary headers, compiler, etc, when installing the package. It makes installing a pre-compiled glibc much more complicated.
i dont see how headers not being installed would matter at all. no code is compiled in pkg_postinst() nor does the locale generation when running `localedef`.
Created attachment 257908 [details] proposed locale.gen An acceptable work-around to me would be to simply ship a locale.gen file with glibc that selects only the most common locale, namely 'en_US.UTF-8 UTF-8'. It seems like we don't gain much by shipping every locale in stages, and certainly most (nearly all?) users don't want all locales installed on their systems. For users who wish to use a different locale, they can simply modify locale.gen and run `locale-gen`. It's a really bad feeling after you've waited for a few hours for glibc to compile when you realize that you forgot to modify locale.gen so now you've got to wait another couple hours for it to generate a bunch of stupid locales that you don't want anyway. The proposed work-around kills this possibility.
BTW do we ignore the fact that md5sums saved in /var/db/pkg/*/*/CONTENTS will differ after locale-gen?
no idea what you mean. glibc doesnt currently record any info about the locales in CONTENTS. `qcheck sys-libs/glibc` shows all md5 matches.
(In reply to comment #15) > no idea what you mean. glibc doesnt currently record any info about the > locales in CONTENTS. If generation will be moved to src_install() it will be there and thus it will differ after locale-gen.
i dont think it would generally be an issue though as few people rerun locale-gen themselves
(In reply to comment #13) > Created an attachment (id=257908) [details] > proposed locale.gen Any comments on this?
(In reply to comment #18) > (In reply to comment #13) > > Created an attachment (id=257908) [details] > > proposed locale.gen > > Any comments on this? This should be discussed in -dev ml.
Comment on attachment 257908 [details] proposed locale.gen we're not shipping with en_US only out of the box
Comment on attachment 108463 [details, diff] run locale-gen in src_install sanely i could add this behind a USE flag like "compile-locales" which stage builders could enable ...
(In reply to comment #21) > Comment on attachment 108463 [details, diff] > run locale-gen in src_install sanely > > i could add this behind a USE flag like "compile-locales" which stage builders > could enable ... That would certainly help.
Created attachment 287099 [details, diff] glibc compile locales patch see if this works for you
The addition of the USE flag compile-locales seemed like a good solution to me. Any chance of resurrecting this idea?
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=bbf35ca26d61f179cd22066eb59c466171b36829 commit bbf35ca26d61f179cd22066eb59c466171b36829 Author: Andreas K. Hüttel <dilfridge@gentoo.org> AuthorDate: 2017-10-27 22:30:59 +0000 Commit: Andreas K. Hüttel <dilfridge@gentoo.org> CommitDate: 2017-10-27 22:31:25 +0000 sys-libs/glibc: Add flag to build locale data in src_install, bug 146882 Based on the patch there by SpanKY; to be included in 2.27 and later. Bug: https://bugs.gentoo.org/146882 Package-Manager: Portage-2.3.13, Repoman-2.3.4 sys-libs/glibc/glibc-9999.ebuild | 28 ++++++++++++++++++++-------- sys-libs/glibc/metadata.xml | 1 + 2 files changed, 21 insertions(+), 8 deletions(-)}
This still needs some fixes though, since the code as it is now will fail on real upgrades (say, 2.25 to 2.26). (Reason: it calls in src_install the newly built localedef, but uses the still-present old dynamic loader of the system for it. That may fail if the new binary needs new versioning symbols etc bla bla...)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=3510931c562f92a584b859112ce36ac3bbe485e3 commit 3510931c562f92a584b859112ce36ac3bbe485e3 Author: Andreas K. Hüttel <dilfridge@gentoo.org> AuthorDate: 2018-02-02 14:33:15 +0000 Commit: Andreas K. Hüttel <dilfridge@gentoo.org> CommitDate: 2018-02-02 14:34:31 +0000 package.use.mask: mask compile-locales feature of glibc, not ready yet Bug: https://bugs.gentoo.org/146882 profiles/base/package.use.mask | 4 ++++ 1 file changed, 4 insertions(+)}
(In reply to Andreas K. Hüttel from comment #26) > This still needs some fixes though, since the code as it is now will fail on > real upgrades (say, 2.25 to 2.26). > > (Reason: it calls in src_install the newly built localedef, but uses the > still-present old dynamic loader of the system for it. That may fail if the > new binary needs new versioning symbols etc bla bla...) dwfreed suggested on IRC that we use LD_LIBRARY_PATH to point at the new libc.
(In reply to Matt Turner from comment #28) > (In reply to Andreas K. Hüttel from comment #26) > > This still needs some fixes though, since the code as it is now will fail on > > real upgrades (say, 2.25 to 2.26). > > > > (Reason: it calls in src_install the newly built localedef, but uses the > > still-present old dynamic loader of the system for it. That may fail if the > > new binary needs new versioning symbols etc bla bla...) > > dwfreed suggested on IRC that we use LD_LIBRARY_PATH to point at the new > libc. Nah... The code is nearly finished. Let's do it either with the second stabilization in 2.28 or with 2.29.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=3be9b76dbd62b8eb1a491fc6fb39a4020c9120a9 commit 3be9b76dbd62b8eb1a491fc6fb39a4020c9120a9 Author: Andreas K. Hüttel <dilfridge@gentoo.org> AuthorDate: 2019-08-04 00:14:36 +0000 Commit: Andreas K. Hüttel <dilfridge@gentoo.org> CommitDate: 2019-08-04 00:15:03 +0000 profiles: unmask compile-locales useflag for glibc-2.30 and above Signed-off-by: Andreas K. Hüttel <dilfridge@gentoo.org> Bug: https://bugs.gentoo.org/146882 profiles/base/package.use.mask | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=fb2fe75af62ad29a44aeba1b8e9e41ce5acb3992 commit fb2fe75af62ad29a44aeba1b8e9e41ce5acb3992 Author: Andreas K. Hüttel <dilfridge@gentoo.org> AuthorDate: 2019-08-06 00:58:57 +0000 Commit: Andreas K. Hüttel <dilfridge@gentoo.org> CommitDate: 2019-08-06 00:59:28 +0000 sys-libs/glibc: Add 2.29 revision with compile-locales support Bug: https://bugs.gentoo.org/146882 Package-Manager: Portage-2.3.70, Repoman-2.3.16 Signed-off-by: Andreas K. Hüttel <dilfridge@gentoo.org> sys-libs/glibc/Manifest | 1 + sys-libs/glibc/glibc-2.29-r4.ebuild | 1473 +++++++++++++++++++++++++++++++++++ 2 files changed, 1474 insertions(+) https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=90ff3d4f5d3a9fb1c8486b2d5ee29ac67b2d620d commit 90ff3d4f5d3a9fb1c8486b2d5ee29ac67b2d620d Author: Andreas K. Hüttel <dilfridge@gentoo.org> AuthorDate: 2019-08-06 00:21:07 +0000 Commit: Andreas K. Hüttel <dilfridge@gentoo.org> CommitDate: 2019-08-06 00:59:28 +0000 package.use.mask: refine compile-locales mask, bug 146882 Bug: https://bugs.gentoo.org/146882 Signed-off-by: Andreas K. Hüttel <dilfridge@gentoo.org> profiles/base/package.use.mask | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Should work now.
