Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 146882 - sys-libs/glibc locale generation should be done in src_*() instead of pkg_postinst()
Summary: sys-libs/glibc locale generation should be done in src_*() instead of pkg_pos...
Status: CONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: High enhancement (vote)
Assignee: Gentoo Toolchain Maintainers
URL:
Whiteboard: not glibc-2.28 stable blocker
Keywords:
Depends on:
Blocks: glibc-2.28
  Show dependency tree
 
Reported: 2006-09-08 14:17 UTC by Andrew Gaffney (RETIRED)
Modified: 2019-08-06 00:59 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
run locale-gen in src_install sanely (glibc-locale-gen-src-install.patch,985 bytes, patch)
2007-01-29 07:36 UTC, SpanKY
Details | Diff
proposed locale.gen (locale.gen,874 bytes, text/plain)
2010-12-24 03:14 UTC, Matt Turner
Details
glibc compile locales patch (glibc-compile-locales.patch,4.59 KB, patch)
2011-09-20 01:54 UTC, SpanKY
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Andrew Gaffney (RETIRED) gentoo-dev 2006-09-08 14:17:52 UTC
When you install glibc from a binpkg, you still have to go through the entire locale generation procedure. For a normal user, they can populate /etc/locale.gen to slim this process down, but when building releases, we can't do this, so generating all locales each time gets *very* annoying.

Can the locale generation be moved into one of the src_*() functions instead of pkg_postinst() to avoid this needless regeneration? It would also be nice if it could be parallelized, but that won't be as big of a deal if it only needs to be done once.
Comment 1 Jakub Moc (RETIRED) gentoo-dev 2006-09-08 15:32:27 UTC
(In reply to comment #0)
> It would also be nice if it could be parallelized, but that won't be as big of a > deal if it only needs to be done once.

Well, see Bug 137706 for this...
Comment 2 SpanKY gentoo-dev 2006-09-08 19:32:14 UTC
there was a reason i moved this code to pkg_postinst() rather than src_install(), i just cant remember why right now :P

probably related to the fact that you cannot cross-compile locales, so if you ran it in src_install(), it'd segfault on you
Comment 3 Andrew Gaffney (RETIRED) gentoo-dev 2006-09-08 19:48:28 UTC
Does pkg_postinst() not get executed when building glibc with crossdev? Or is there something more to it?
Comment 4 SpanKY gentoo-dev 2006-09-08 20:24:41 UTC
in the cross-compile toolchain scenario the locale step is skipped on purpose
Comment 5 Chris Gianelloni (RETIRED) gentoo-dev 2006-09-10 14:57:38 UTC
Perhaps we can use some kind of system to keep a list on-disk (separate from locales.gen) that shows what's already been built, then glibc can compare them.  Andrew is right, on arches like mips, it takes longer to do the glibc locales than to do everything else when building with catalyst and a full cache.  On faster architectures, it's just an annoyance, but one that we can surely do without.
Comment 6 SpanKY gentoo-dev 2006-09-10 20:50:13 UTC
> Perhaps we can use some kind of system to keep a list on-disk (separate from
> locales.gen) that shows what's already been built

although you can already do this (`locale-gen -u`), this isnt going to be added to the ebuild as you cant [easily] tell whether the new locale files installed differ from the binary generated ones already on disk

in short: locales are always going to be generated, it's just a matter of when

storing all the locales increases the binpkg size significantly btw, but i guess for slower arches it's a lot easier to sacrifice disk space than cpu time
Comment 7 SpanKY gentoo-dev 2007-01-29 07:30:28 UTC
so here's the problem

you cannot utilize locale.gen from ROOT while in src_install ... that means if you want to run locale-gen inside of src_install, you need to generate all locales

this in turn screws everyone else as now their systems will generate all locales only to turn around and throw them away in pkg_postinst()
Comment 8 SpanKY gentoo-dev 2007-01-29 07:36:25 UTC
Created attachment 108463 [details, diff]
run locale-gen in src_install sanely

here's how you'd do it sanely ... however, you still need to figure out how to avoid the aforementioned issue

you could utilize like the build or bootstrap USE flags, but that'd only help you in stage1/stage2 generation ... going from a stage2 to a stage3 is pretty much just like a user upgrading, so you're going to still be screwed
Comment 9 SpanKY gentoo-dev 2008-01-30 14:17:45 UTC
a possibility for us to investigate:
http://sourceware.org/ml/libc-ports/2008-01/msg00011.html
this addresses the cross-compiling / endianness issue

we could consider tying the remaining behavior to "binpkg" or "build" ... would that be usable at all for release building of stages ?
Comment 10 Chris Gianelloni (RETIRED) gentoo-dev 2008-01-31 01:09:17 UTC
If we made it do locale generation without reading locale.gen in src_install if USE=build or USE=bootstrap, that would save us a good bit of time.  It would/could still be done in pkg_* for USE="-build -bootstrap" so users wouldn't notice a difference, except for those building from stage1, and they can suffer a few extra minutes if they're willing to do that sort of an install, anyway.
Comment 11 Christopher Friedt 2009-03-25 08:26:43 UTC
(In reply to comment #2)
> there was a reason i moved this code to pkg_postinst() rather than
> src_install(), i just cant remember why right now :P
> 
> probably related to the fact that you cannot cross-compile locales, so if you
> ran it in src_install(), it'd segfault on you
> 

There's another small problem with having locale-generating code in pkg_postinst(). Namely, when e.g. glibc-2.7-r2 is installed as a tbz2 (binary) on a system without headers (or a compiler for that matter) pkg_postinst() fails. In my case, I wasn't actually cross-compiling the package either because My build machine is a server somewhere at my office, and my host machine is my EEE 701. 

It would be nice if all locales could be pre-generated on the build system, and then screened on the host system, in the case where cross-compilation is not happening. 

Regarding cross-compilation with the local-generating code in pkg_postinst(), I'm just puzzled... typically on an embedded system one wouldn't find all of the necessary headers, compiler, etc, when installing the package. It makes installing a pre-compiled glibc much more complicated. 
Comment 12 SpanKY gentoo-dev 2009-03-29 01:17:31 UTC
i dont see how headers not being installed would matter at all.  no code is compiled in pkg_postinst() nor does the locale generation when running `localedef`.
Comment 13 Matt Turner gentoo-dev 2010-12-24 03:14:38 UTC
Created attachment 257908 [details]
proposed locale.gen

An acceptable work-around to me would be to simply ship a locale.gen file with glibc that selects only the most common locale, namely 'en_US.UTF-8 UTF-8'.

It seems like we don't gain much by shipping every locale in stages, and certainly most (nearly all?) users don't want all locales installed on their systems. For users who wish to use a different locale, they can simply modify locale.gen and run `locale-gen`.

It's a really bad feeling after you've waited for a few hours for glibc to compile when you realize that you forgot to modify locale.gen so now you've got to wait another couple hours for it to generate a bunch of stupid locales that you don't want anyway. The proposed work-around kills this possibility.
Comment 14 Peter Volkov (RETIRED) gentoo-dev 2010-12-24 12:53:54 UTC
BTW do we ignore the fact that md5sums saved in /var/db/pkg/*/*/CONTENTS will differ after locale-gen?
Comment 15 SpanKY gentoo-dev 2010-12-27 21:07:30 UTC
no idea what you mean.  glibc doesnt currently record any info about the locales in CONTENTS.  `qcheck sys-libs/glibc` shows all md5 matches.
Comment 16 Peter Volkov (RETIRED) gentoo-dev 2010-12-29 07:26:14 UTC
(In reply to comment #15)
> no idea what you mean.  glibc doesnt currently record any info about the
> locales in CONTENTS. 

If generation will be moved to src_install() it will be there and thus it will differ after locale-gen.
Comment 17 SpanKY gentoo-dev 2010-12-31 22:32:23 UTC
i dont think it would generally be an issue though as few people rerun locale-gen themselves
Comment 18 Matt Turner gentoo-dev 2011-02-15 15:50:24 UTC
(In reply to comment #13)
> Created an attachment (id=257908) [details]
> proposed locale.gen

Any comments on this?
Comment 19 Peter Volkov (RETIRED) gentoo-dev 2011-04-30 19:45:25 UTC
(In reply to comment #18)
> (In reply to comment #13)
> > Created an attachment (id=257908) [details]
> > proposed locale.gen
> 
> Any comments on this?

This should be discussed in -dev ml.
Comment 20 SpanKY gentoo-dev 2011-09-18 05:58:41 UTC
Comment on attachment 257908 [details]
proposed locale.gen

we're not shipping with en_US only out of the box
Comment 21 SpanKY gentoo-dev 2011-09-18 06:00:29 UTC
Comment on attachment 108463 [details, diff]
run locale-gen in src_install sanely

i could add this behind a USE flag like "compile-locales" which stage builders could enable ...
Comment 22 Matt Turner gentoo-dev 2011-09-18 15:35:28 UTC
(In reply to comment #21)
> Comment on attachment 108463 [details, diff]
> run locale-gen in src_install sanely
> 
> i could add this behind a USE flag like "compile-locales" which stage builders
> could enable ...

That would certainly help.
Comment 23 SpanKY gentoo-dev 2011-09-20 01:54:44 UTC
Created attachment 287099 [details, diff]
glibc compile locales patch

see if this works for you
Comment 24 Marshall McMullen 2017-01-16 18:15:35 UTC
The addition of the USE flag compile-locales seemed like a good solution to me. Any chance of resurrecting this idea?
Comment 25 Larry the Git Cow gentoo-dev 2017-10-27 22:31:34 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=bbf35ca26d61f179cd22066eb59c466171b36829

commit bbf35ca26d61f179cd22066eb59c466171b36829
Author:     Andreas K. Hüttel <dilfridge@gentoo.org>
AuthorDate: 2017-10-27 22:30:59 +0000
Commit:     Andreas K. Hüttel <dilfridge@gentoo.org>
CommitDate: 2017-10-27 22:31:25 +0000

    sys-libs/glibc: Add flag to build locale data in src_install, bug 146882
    
    Based on the patch there by SpanKY; to be included in 2.27 and later.
    
    Bug: https://bugs.gentoo.org/146882
    Package-Manager: Portage-2.3.13, Repoman-2.3.4

 sys-libs/glibc/glibc-9999.ebuild | 28 ++++++++++++++++++++--------
 sys-libs/glibc/metadata.xml      |  1 +
 2 files changed, 21 insertions(+), 8 deletions(-)}
Comment 26 Andreas K. Hüttel gentoo-dev 2017-11-11 15:44:33 UTC
This still needs some fixes though, since the code as it is now will fail on real upgrades (say, 2.25 to 2.26). 

(Reason: it calls in src_install the newly built localedef, but uses the still-present old dynamic loader of the system for it. That may fail if the new binary needs new versioning symbols etc bla bla...)
Comment 27 Larry the Git Cow gentoo-dev 2018-02-02 14:34:49 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=3510931c562f92a584b859112ce36ac3bbe485e3

commit 3510931c562f92a584b859112ce36ac3bbe485e3
Author:     Andreas K. Hüttel <dilfridge@gentoo.org>
AuthorDate: 2018-02-02 14:33:15 +0000
Commit:     Andreas K. Hüttel <dilfridge@gentoo.org>
CommitDate: 2018-02-02 14:34:31 +0000

    package.use.mask: mask compile-locales feature of glibc, not ready yet
    
    Bug: https://bugs.gentoo.org/146882

 profiles/base/package.use.mask | 4 ++++
 1 file changed, 4 insertions(+)}
Comment 28 Matt Turner gentoo-dev 2018-12-19 15:57:00 UTC
(In reply to Andreas K. Hüttel from comment #26)
> This still needs some fixes though, since the code as it is now will fail on
> real upgrades (say, 2.25 to 2.26). 
> 
> (Reason: it calls in src_install the newly built localedef, but uses the
> still-present old dynamic loader of the system for it. That may fail if the
> new binary needs new versioning symbols etc bla bla...)

dwfreed suggested on IRC that we use LD_LIBRARY_PATH to point at the new libc.
Comment 29 Andreas K. Hüttel gentoo-dev 2018-12-30 20:21:48 UTC
(In reply to Matt Turner from comment #28)
> (In reply to Andreas K. Hüttel from comment #26)
> > This still needs some fixes though, since the code as it is now will fail on
> > real upgrades (say, 2.25 to 2.26). 
> > 
> > (Reason: it calls in src_install the newly built localedef, but uses the
> > still-present old dynamic loader of the system for it. That may fail if the
> > new binary needs new versioning symbols etc bla bla...)
> 
> dwfreed suggested on IRC that we use LD_LIBRARY_PATH to point at the new
> libc.

Nah... 

The code is nearly finished. Let's do it either with the second stabilization in 2.28 or with 2.29.
Comment 30 Larry the Git Cow gentoo-dev 2019-08-04 00:15:26 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=3be9b76dbd62b8eb1a491fc6fb39a4020c9120a9

commit 3be9b76dbd62b8eb1a491fc6fb39a4020c9120a9
Author:     Andreas K. Hüttel <dilfridge@gentoo.org>
AuthorDate: 2019-08-04 00:14:36 +0000
Commit:     Andreas K. Hüttel <dilfridge@gentoo.org>
CommitDate: 2019-08-04 00:15:03 +0000

    profiles: unmask compile-locales useflag for glibc-2.30 and above
    
    Signed-off-by: Andreas K. Hüttel <dilfridge@gentoo.org>
    Bug: https://bugs.gentoo.org/146882

 profiles/base/package.use.mask | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Comment 31 Larry the Git Cow gentoo-dev 2019-08-06 00:59:34 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=fb2fe75af62ad29a44aeba1b8e9e41ce5acb3992

commit fb2fe75af62ad29a44aeba1b8e9e41ce5acb3992
Author:     Andreas K. Hüttel <dilfridge@gentoo.org>
AuthorDate: 2019-08-06 00:58:57 +0000
Commit:     Andreas K. Hüttel <dilfridge@gentoo.org>
CommitDate: 2019-08-06 00:59:28 +0000

    sys-libs/glibc: Add 2.29 revision with compile-locales support
    
    Bug: https://bugs.gentoo.org/146882
    Package-Manager: Portage-2.3.70, Repoman-2.3.16
    Signed-off-by: Andreas K. Hüttel <dilfridge@gentoo.org>

 sys-libs/glibc/Manifest             |    1 +
 sys-libs/glibc/glibc-2.29-r4.ebuild | 1473 +++++++++++++++++++++++++++++++++++
 2 files changed, 1474 insertions(+)

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=90ff3d4f5d3a9fb1c8486b2d5ee29ac67b2d620d

commit 90ff3d4f5d3a9fb1c8486b2d5ee29ac67b2d620d
Author:     Andreas K. Hüttel <dilfridge@gentoo.org>
AuthorDate: 2019-08-06 00:21:07 +0000
Commit:     Andreas K. Hüttel <dilfridge@gentoo.org>
CommitDate: 2019-08-06 00:59:28 +0000

    package.use.mask: refine compile-locales mask, bug 146882
    
    Bug: https://bugs.gentoo.org/146882
    Signed-off-by: Andreas K. Hüttel <dilfridge@gentoo.org>

 profiles/base/package.use.mask | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)