Bug 680782 – =sys-libs/glibc-2.28-r5 - pkg_postinst(): run_locale_gen() uses MAKEOPTS from build target

• Product: Gentoo Linux
• Status: RESOLVED FIXED

[Description tt_1 2019-03-17 15:05:13 UTC]

yesterday an update for glibc to 2.28 came in for arm, and so I decided to cross-compile it from an amd64 host with 12 cores available. 

When this binary was emerged on the host with the -k switch of emerge, it ran locale-gen with -j12 as well. In my case it didn't matter much, I have my locales restricted to four anyway and the device has four cores available. 

Nontheless, this shouldn't happen, as it can possibly cause destruction if the discrepance of cores between host and target is way higher, and the amount of locales to be generated much higher.

[Comment 1 Sergei Trofimovich (RETIRED) 2019-03-17 16:56:17 UTC]

locale-gen always runs with '--jobs $(makeopts_jobs)' parameter:
    https://gitweb.gentoo.org/repo/gentoo.git/tree/sys-libs/glibc/glibc-2.28-r5.ebuild#n1113

https://dev.gentoo.org/~ulm/pms/head/pms.html does not say anything special about MAKEOPTS. It means MAKEOPTS is stored in environment as any other variable and is not dynamically changeable in pkg_*() phases.

$ cat b680782-0.ebuild 
    EAPI=6
    inherit multiprocessing
    DESCRIPTION="Reproducer for https://bugs.gentoo.org/680782"
    HOMEPAGE="https://bugs.gentoo.org/680782"
    KEYWORDS="amd64 x86"
    LICENSE="GPL-2"
    SLOT="0"
    S="${WORKDIR}"
    src_install() { einfo "src_install() jobs: $(makeopts_jobs)"; }
    pkg_postinst() { einfo "src_install() jobs: $(makeopts_jobs)"; }

# MAKEOPTS=-j4 emerge --buildpkgonly app-misc/b680782 --quiet-build=n
 * src_install() jobs: 4
# MAKEOPTS=-j2 emerge --usepkg app-misc/b680782 --quiet-build=n
 * src_install() jobs: 4

We could do something for ${MERGE_TYPE} == "binary" 

> Nontheless, this shouldn't happen, as it can possibly cause destruction if the discrepance of cores between host and target is way higher, and the amount of locales to be generated much higher.

Running 400 processes is totally fine on a single core as long as RAM used for locale-generation is reasonable.

[Comment 2 tt_1 2019-03-17 17:02:56 UTC]

restraining to -j1 for ${MERGE_TYPE} == "binary" would be just fine in my opinion.

[Comment 3 Sergei Trofimovich (RETIRED) 2019-03-17 17:30:13 UTC]

(In reply to tt_1 from comment #2)
> restraining to -j1 for ${MERGE_TYPE} == "binary" would be just fine in my
> opinion.

It would slow glibc installation down for everyone who builds binary packages locally just before an upgrade.

[Comment 4 tt_1 2019-03-17 18:06:10 UTC]

Do you mean in case of 

FEATURES="buildpkg" 

?

[Comment 5 Alexander Tsoy 2019-04-10 23:08:44 UTC]

Got the same problem. 6 localedef processes is too much for my arm board:

$ sudo dmesg | grep -i killed
[423527.159682] Killed process 937 (localedef) total-vm:77484kB, anon-rss:74584kB, file-rss:24kB, shmem-rss:0kB

I thinks it's clearly a package manager issue. It should restore MAKEOPTS to its original state after restoring environment at least in pkg_* phases... or doesn't store MAKEOPTS in environment in the first place.

[Comment 6 Sergei Trofimovich (RETIRED) 2019-04-11 09:41:55 UTC]

(In reply to Alexander Tsoy from comment #5)
> Got the same problem. 6 localedef processes is too much for my arm board:
> 
> $ sudo dmesg | grep -i killed
> [423527.159682] Killed process 937 (localedef) total-vm:77484kB,
> anon-rss:74584kB, file-rss:24kB, shmem-rss:0kB
> 
> I thinks it's clearly a package manager issue. It should restore MAKEOPTS to
> its original state after restoring environment at least in pkg_* phases...
> or doesn't store MAKEOPTS in environment in the first place.

As a workaround you can set lower MAKEOPTS on a host machine where you build glibc.

Gentoo's package manager specification does not say anything special about MAKEOPTS. Hence it will be preserved in the environmet as the rest of options.

Thus I would say it's not a problem in package manager but in a glibc ebuild.

[Comment 7 tt_1 2019-07-22 11:59:15 UTC]

So, what is the status of this? Do you deem it to be fixable by dropping to -j1 with some fancy guards for merge type binary, in src_install? If so, I might be able to cook up something and attach a possible patch for the glibc ebuild. 

Thanks

[Comment 8 Sergei Trofimovich (RETIRED) 2019-07-22 19:00:08 UTC]

I'm not aware of any work done in this area thus status is unchanged: we need a tweak in glibc ebuild.

I don't think there is a way to implement it cleanly. Aside from moving locale generation into another package.

'[[ ${MERGE_TYPE} = "binary" ]]' might might be goot enough compromise. It will slow down all binary package users though.

[Comment 9 tt_1 2019-08-11 10:54:55 UTC]

Created attachment 586536 [details, diff]
patch against glibc-2.29-r2 ebuild

Would this attempt work? In case you were willing to slow down locale-gen to -j1 for everyone who uses a binpkg, no matter how strong the hardware? :-)

[Comment 10 Sergei Trofimovich (RETIRED) 2019-08-11 16:40:04 UTC]

(In reply to tt_1 from comment #9)
> Created attachment 586536 [details, diff] [details, diff]
> patch against glibc-2.29-r2 ebuild
> 
> Would this attempt work? In case you were willing to slow down locale-gen to
> -j1 for everyone who uses a binpkg, no matter how strong the hardware? :-)

> if [[ ${MERGE_TYPE} == !"binary" ]] ; then

That is not a valid match syntax.

You might want to look at USE=compile-locales added no recent glibcs. That shifts locale generation from pkg_*() to src_*() phases and makes locale-gen a no-op.

[Comment 11 tt_1 2019-09-18 15:28:17 UTC]

fixed in glibc-2.29-r5! :)