Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 72596 - Bootstrap overwrites locales.build and extra locales are built
Summary: Bootstrap overwrites locales.build and extra locales are built
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Core system (show other bugs)
Hardware: All Linux
: High normal (vote)
Assignee: Gentoo Toolchain Maintainers
URL:
Whiteboard:
Keywords:
: 74219 85675 92729 (view as bug list)
Depends on:
Blocks:
 
Reported: 2004-11-26 19:16 UTC by Jeff Davidson
Modified: 2005-05-16 15:23 UTC (History)
6 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
ebuild for the rt2500-1.4.4.0 driver (rt2500-1.4.4.0.ebuild,2.21 KB, text/plain)
2004-12-08 09:42 UTC, Michael Schilling
Details
A list of default locales generated (locale-a.txt,589 bytes, text/plain)
2004-12-15 16:44 UTC, wiktorw
Details
Unnecessary /etc/locales.build file overwrite after glibc emerge (locales.build-problem.txt,1.21 KB, text/plain)
2004-12-15 17:44 UTC, wiktorw
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jeff Davidson 2004-11-26 19:16:06 UTC
During the bootstrap.sh, multiple glibc compilations are done, the reasons of which I don't know, but it's happening.  Included is /var/log/emerge.log.  My process is to USE="-* build bootstrap" emerge --oneshot linux26-headers prior to bootstrapping to bring in only these headers (otherwise, bootstrap.sh brings in 2.4 headers as well), then I bootstrap with USE="nptl" in make.conf.  The problem is that because glibc gets compiled multiple times, and bootstrap.sh enables overwriting of any configuration files, even with userlocales specified as a USE flag, multiple locales get written.  Since this is mentioned in the installation guide, I think this is a problem.  I have livecd in my use flag since I'm building a system for a LiveCD in a chroot... is this the problem?

Reproducible: Always
Steps to Reproduce:
1. In /etc/make.conf, USE="nptl userlocales livecd"
2. USE="-* build bootstrap" emerge --oneshot linux26-headers
3. scripts/bootstrap.sh
4. locale -a

Actual Results:  
Many extra locales showed up.

Expected Results:  
Only the en_US locales I select should show up.

emerge info output, paused in middle of bootstrapping in chroot:

Gentoo Base System version 1.4.16
Portage 2.0.51-r3 (default-linux/x86/2004.3, gcc-3.3.4, glibc-2.3.4.20040808-r1,
2.6.9-gentoo-r6 i686)
=================================================================
System uname: 2.6.9-gentoo-r6 i686 Intel(R) Pentium(R) 4 CPU 2.40GHz
Autoconf: 
Automake: 
Binutils: sys-devel/binutils-2.15.90.0.1.1-r3
Headers:  sys-kernel/linux26-headers-2.6.8.1
Libtools: 
ACCEPT_KEYWORDS="x86"
AUTOCLEAN="yes"
CFLAGS="-Os -march=i686 -mcpu=i686 -pipe -fomit-frame-pointer"
CHOST="i686-pc-linux-gnu"
COMPILER=""
CONFIG_PROTECT="/etc /usr/X11R6/lib/X11/xkb /usr/kde/2/share/config
/usr/kde/3/share/config /usr/share/config /var/qmail/control"
CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d"
CXXFLAGS="-Os -march=i686 -mcpu=i686 -pipe -fomit-frame-pointer"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoaddcvs autoconfig ccache distlocks nodoc noinfo noman sandbox sfperms"
GENTOO_MIRRORS="http://mirror.datapipe.net/gentoo"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY=""
SYNC="rsync://rsync.us.gentoo.org/gentoo-portage"
USE="X apm arts avi berkdb bitmap-fonts crypt cups encode f77 foomaticdb fortran
gdbm gif gnome gpm gtk gtk2 imlib jpeg kde libg++ libwww livecd mad mikmod motif
mpeg ncurses nptl oggvorbis opengl oss pam pdflib png python qt quicktime
readline sdl spell ssl svga tcpd truetype userlocales x86 xml2 xmms xv zlib"
Comment 1 Jeff Davidson 2004-11-29 19:14:48 UTC
For anyone (everyone) currently experiencing this bug, you can either remerge glibc after bootstrapping with the fixed locales.build and the userlocales use flag, or you can replace the locales.build file after each build of glibc in the bootstrap (I think there are only two, so replace after the first so the second time around it still builds the same locales).  

It's a good idea to use ccache in this case:

USE="-* build bootstrap" emerge --nodeps ccache

This will hopefully speed up the second round of glibc for the time being until the bug is fixed.  Hope this helps someone.
Comment 2 Peter Beutner 2004-11-30 09:35:40 UTC
or you use this:
*  app-admin/localepurge
      Latest version available: 0.2-r1
      Latest version installed: [ Not Installed ]
      Size of downloaded files: 3 kB
      Homepage:    http://www.gentoo.org/~bass/
      Description: Script to recover diskspace wasted for unneeded locale files and localized man pages
      License:     GPL-2

anyway, it doesnt resolves the bug ;)
Comment 3 Jeff Davidson 2004-11-30 17:35:43 UTC
Actually, no, I think that program is for a different thing - each package installs man pages, documentation, and files for internationalization, and localepurge gets rid of those for other languages.  These are the system locales from glibc.  Someone with more knowledge can explain exactly what they are.
Comment 4 SpanKY gentoo-dev 2004-11-30 19:19:23 UTC
so when you emerge linux26-headers, glibc isnt built ?

in your description you say:
then I bootstrap with USE="nptl" in make.conf

that's incorrect ... it sounds like you wanted to put USE="nptl userlocales" into make.conf and then bootstrap

i dont know why you have 'livecd' in your USE, users should never set that
Comment 5 Jeff Davidson 2004-11-30 19:31:17 UTC
Sorry if I was ambiguous (and looking back, I probably was), but not quite.

First of all, the livecd use flag was because I was attempting to build a LiveCD as per the howto in the Documentation, Tips, and Tricks section, however, it is irrelevant because I encountered this bug on another, NPTL/2.6 headers only bootstrap without that flag.

If I don't emerge linux26-headers and I do scripts/bootstrap.sh -f, it starts pulling in linux-headers and fetches linux-2.4.21.tar.bz2 (or at least, it did when I last tried, and assuming it still says to emerge the 26 headers manually prior to bootstrapping in the handbook, I think this is still the behavior).  Therefore, I emerge them with USE="-* build bootstrap" emerge --oneshot linux26-headers, which works fine, and then I bootstrap. My USE flags in the second test case were simply "nptl userlocales".  Everything compiles perfectly fine.  The issue here is that during the course of the bootstrap, glibc (and perhaps others, though I didn't check because none were as significant) is compiled twice.  Therefore, it uses the locales.build file I specified the first time through and only compiles the two en_US locales, however, in installing the first pass it replaces the locales.build because cfgprotect is disabled.  Therefore, for the second pass, it uses the new locales.build and builds extra locales specified in the example file.

Not only is the second compile a pretty large waste of time, especially for people with slow computers where it takes glibc a good few hours to compile, but this creates problems for users who use locales.build but get extra locales and spend extra time compiling the locales anyways.

Did this clarify things?  If not, please let me know. :)
Comment 6 SpanKY gentoo-dev 2004-11-30 19:39:57 UTC
dunno what docs you've been reading, but we build livecd's with catalyst now (emerge catalyst)

any other way is not supported afaik

yes, that does clarify things ... we've fixed bootstrap.sh in the past to not compile gcc/glibc twice ... lets see if we can do it again
Comment 7 Jeff Davidson 2004-11-30 19:50:11 UTC
I was just tooling around with the livecd stuff,  here's the page:

http://forums.gentoo.org/viewtopic.php?t=244837

Here's a quote:

This mini-HowTo will show you how simple it is to create your own LiveCD by hand. It will also allow you to have the following advantages over the catalyst way:
1. The build source will be kept intact and will not get deleted between iterations of LiveCD creation. This will allow you to sync, update, merge, and customize your environment incrementally the same way you do it with a real system.
2. The CD will boot using GRUB, not ISOLINUX. This gives you the same flexibility you have with a real system, that it, changing kernel parameters, discovering devices, etc.
3. As a nice side effect of keeping your build source between iterations, the build time will be reduced dramatically. 

Yes, catalyst can use pkg's to save time and I'm sure someone could make it use grub, but the ability to work with a real system in a chroot is appealing, and I've tried catalyst before.  Honestly, I gave up and did a quick remaster of knoppix - the only reason I needed a new LiveCD was really because I needed drivers for my wireless device which weren't included on the LiveCDs (either the native (crappy) rt2500 drivers or the latest version of ndiswrapper or driverloader)).  Still, the issue with the bootstrap remains, and thanks a lot for looking into it. :)

At the very least, if you can't figure this one out, could you like, disable config_protect for that one file? I'm not sure, it sounds like a cheap workaround, but there it is. :)

Thanks a lot,
-Jeff
Comment 8 Rafal Peisert 2004-12-05 15:11:36 UTC
Well this is particularly annoying when your locale isn't amongst the defaults; you get system which doesn't have support for your locales. Maybe somebody could write a patch which would prevent overwriting of the '/etc/locales.build' config if the 'bootstrap' flag is set?
Comment 9 Jeff Davidson 2004-12-06 14:32:51 UTC
Yeah, but a patch is a workaround - glibc gets compiled twice, and thats timeconsuming and pointless.  Hope you all can fix it. :)
Comment 10 Michael Schilling 2004-12-08 09:42:49 UTC
Created attachment 45543 [details]
ebuild for the rt2500-1.4.4.0 driver
Comment 11 Michael Schilling 2004-12-08 09:44:09 UTC
Comment on attachment 45543 [details]
ebuild for the rt2500-1.4.4.0 driver

Sorry, I took the wrong bug...
Comment 12 SpanKY gentoo-dev 2004-12-12 13:48:31 UTC
*** Bug 74219 has been marked as a duplicate of this bug. ***
Comment 13 wiktorw 2004-12-15 16:44:17 UTC
Created attachment 46088 [details]
A list of default locales generated

I would like to confirm that the problem still persist (as of 2004-12-16).
I just did a fresh stage-1 installation from amd64 livecd 2004.3-r1.
After finishing stage-3 I noticed that I don't have desired
en_US and pl_PL locales.
Instead after running 'locale -a' I get a set of default locales (see
attachment).
As the temporary solution I put "-userlocales" in the USE flags and re-emerged
sys-libs/glibc 2.3.4.20040808-r1.

Of course, all the above Should Not Happen and Should Not Be Needed :-)
Comment 14 SpanKY gentoo-dev 2004-12-15 16:47:08 UTC
why not just have bootstrap copy the file out of the way like it does with make.conf/make.conf.build
Comment 15 wiktorw 2004-12-15 17:44:33 UTC
Created attachment 46090 [details]
Unnecessary /etc/locales.build file overwrite after glibc emerge

Okay, maybe this will be useful.

I re-emerged glibc once again. This time I updated /etc/locales.build
beforehand to contain only four locales:

en_US/ISO-8859-1
en_US.UTF-8/UTF-8
pl_PL/ISO-8859-2
pl_PL.UTF-8/UTF-8

After "emerge glibc" I got a surprising message (see attachment).
It looked like after emerging glibc, its ebuild file instructed
emerge to overwrite my carefully crafted /etc/locales.build file.

Indeed, in the 'glibc-2.3.4.20040808-r1.ebuild' file there are many
references to the 'locales.build' file. There are also a couple of
lines near the end:

<quote>
	# This is our new config file for building locales
	insinto /etc
	doins ${FILESDIR}/locales.build
</quote>

I suspect that they may be the source of all the trouble.
Could someone check it once again and confirm?

Regards,
Wiktor Wandchowicz
Comment 16 Jeff Davidson 2004-12-16 05:37:01 UTC
This is expected behavior - someone needs to create the file so that you have something to modify.  That's why it comes in the stage tarball.  Etc-update is just for this reason: This way, when a package comes with a generic configuration file, it doesn't replace what you have automatically.

SpanKY: I like your idea - the problem is, glibc still gets compiled twice, and that's a whole lot of compiling - it takes well over 30 minutes on my p4 2.4 system, and I'm sure there are many with something slower. :)
Comment 17 SpanKY gentoo-dev 2005-05-16 15:18:51 UTC
*** Bug 85675 has been marked as a duplicate of this bug. ***
Comment 18 SpanKY gentoo-dev 2005-05-16 15:18:56 UTC
*** Bug 92729 has been marked as a duplicate of this bug. ***
Comment 19 SpanKY gentoo-dev 2005-05-16 15:23:01 UTC
fixed in portage