Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 824482 - bootstrap-prefix.sh: sys-libs/glibc-2.34 causes bootstrap to fail (binutils: undefined reference to `__libc_csu_init')
Summary: bootstrap-prefix.sh: sys-libs/glibc-2.34 causes bootstrap to fail (binutils: ...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo/Alt
Classification: Unclassified
Component: Prefix Support (show other bugs)
Hardware: AMD64 Linux
: Normal normal with 1 vote (vote)
Assignee: Gentoo Prefix
URL:
Whiteboard: Workaround applied in bootstrap-prefi...
Keywords: PATCH, PullRequest
: 827164 (view as bug list)
Depends on:
Blocks: glibc-2.34
  Show dependency tree
 
Reported: 2021-11-18 11:03 UTC by Pietro
Modified: 2022-09-03 01:59 UTC (History)
5 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
=sys-devel/binutils-2.37_p1-r1's configure (configure,478.54 KB, text/plain)
2021-11-18 11:17 UTC, Pietro
Details
emerge --info (emerge-info.log,4.63 KB, text/plain)
2021-11-18 11:17 UTC, Pietro
Details
emerge -pqv (emerge-pqv.log,286 bytes, text/plain)
2021-11-18 11:18 UTC, Pietro
Details
build error (error.log,20.43 KB, text/plain)
2021-11-18 11:18 UTC, Pietro
Details
config.log (config.log,22.32 KB, text/plain)
2021-11-18 11:59 UTC, Pietro
Details
patch-bug-824482-mask-glibc-2.34.patch (patch-bug-824482-mask-glibc-2.34.patch,705 bytes, patch)
2021-11-29 07:46 UTC, Benjamin Block
Details | Diff
Fold CPPFLAGS/LDFLAGS in CC/CXX to allow new glibc in bootstrap stage3 (bootstrap-prefix-allow-new-gcc-glibc.diff,2.25 KB, patch)
2022-09-01 17:43 UTC, Bart Oldeman
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Pietro 2021-11-18 11:03:36 UTC
I have been trying to bootstrap Gentoo prefix on a HPC cluster which runs on Ubuntu 18.04 and is part of an AD domain, though I am now running into another issue, this time with binutils-2.37_p1-r1. 

I did look at $EPREFIX/var/tmp/portage/sys-devel/binutils-2.37_p1-r1/work/binutils-2.37/configure for clues, though I can't really figure out where the issue lays. The log errors do mention to look at configure.log for more info, however I was unable to find such log file.

Attached is the build error itself, alongside with configure.log, the output of `emerge --info '=sys-devel/binutils-2.37_p1-r1::gentoo'` and `emerge -pqv '=sys-devel/binutils-2.37_p1-r1::gentoo'.

Any help will be much appreciated. 

Reproducible: Always

Steps to Reproduce:
1. ${EPREFIX}/bootstrap-prefix.sh ${EPREFIX} stage3
Comment 1 Pietro 2021-11-18 11:17:14 UTC
Created attachment 752318 [details]
=sys-devel/binutils-2.37_p1-r1's configure
Comment 2 Pietro 2021-11-18 11:17:41 UTC
Created attachment 752322 [details]
emerge --info
Comment 3 Pietro 2021-11-18 11:18:07 UTC
Created attachment 752326 [details]
emerge -pqv
Comment 4 Pietro 2021-11-18 11:18:21 UTC
Created attachment 752330 [details]
build error
Comment 5 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-11-18 11:19:11 UTC
configure: error: in `/nas/weka/resources/tools/apps/software/devel/gentoo/var/tmp/portage/sys-devel/binutils-2.37_p1-r1/work/build':
configure: error: C compiler cannot create executables
See `config.log' for more details
 * ERROR: sys-devel/binutils-2.37_p1-r1::gentoo failed (configure phase):
 *   (no error message)

Could you include config.log (if you're not already working on attaching that)?
Comment 6 Pietro 2021-11-18 11:34:14 UTC
Hello Sam, I am actually unable to locate it.
(base) pietro@003:~$ find  $EPREFIX/var/tmp/portage/sys-devel/binutils-2.37_p1-r1 |grep configure.log
(base) pietro@003:~$
Comment 7 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-11-18 11:36:59 UTC
K(In reply to Pietro from comment #6)
> Hello Sam, I am actually unable to locate it.
> (base) pietro@003:~$ find 
> $EPREFIX/var/tmp/portage/sys-devel/binutils-2.37_p1-r1 |grep configure.log
> (base) pietro@003:~$

config, not configure :)
Comment 8 Pietro 2021-11-18 11:59:00 UTC
Created attachment 752338 [details]
config.log

ugh, I had misread the filename and thought to be configure.log instead. My bad.
Comment 9 Pietro 2021-11-18 12:55:20 UTC
I could be mistaken but by the looks of it binutils is unable to properly detect gcc?

>>>
gcc version 10.3.0 (Gentoo 10.3.0-r2 p3)
configure:4341: $? = 0
configure:4330: x86_64-pc-linux-gnu-gcc -V >&5
x86_64-pc-linux-gnu-gcc: error: unrecognized command-line option '-V'
x86_64-pc-linux-gnu-gcc: fatal error: no input files
compilation terminated.
configure:4341: $? = 1
configure:4330: x86_64-pc-linux-gnu-gcc -qversion >&5
x86_64-pc-linux-gnu-gcc: error: unrecognized command-line option '-qversion'; did you mean '--version'?
x86_64-pc-linux-gnu-gcc: fatal error: no input files
compilation terminated.
<<<
Comment 10 Pietro 2021-11-18 12:59:13 UTC
Oh, it can actually detect gcc, the -V flag seems to be wrong as -v actually works.

pietro003:~$ /nas/weka/resources/tools/apps/software/devel/gentoo/tmp/usr/x86_64-pc-linux-gnu/gcc-bin/10.3.0/x86_64-pc-linux-gnu-gcc -v
Using built-in specs.
COLLECT_GCC=/nas/weka/resources/tools/apps/software/devel/gentoo/tmp/usr/x86_64-pc-linux-gnu/gcc-bin/10.3.0/x86_64-pc-linux-gnu-gcc
COLLECT_LTO_WRAPPER=/nas/weka/resources/tools/apps/software/devel/gentoo/tmp/usr/libexec/gcc/x86_64-pc-linux-gnu/10.3.0/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: /nas/weka/resources/tools/apps/software/devel/gentoo/tmp/var/tmp/portage/sys-devel/gcc-10.3.0-r2/work/gcc-10.3.0/configure --host=x86_64-pc-linux-gnu --build=x86_64-pc-linux-gnu --prefix=/nas/weka/resources/tools/apps/software/devel/gentoo/tmp/usr --bindir=/nas/weka/resources/tools/apps/software/devel/gentoo/tmp/usr/x86_64-pc-linux-gnu/gcc-bin/10.3.0 --includedir=/nas/weka/resources/tools/apps/software/devel/gentoo/tmp/usr/lib/gcc/x86_64-pc-linux-gnu/10.3.0/include --datadir=/nas/weka/resources/tools/apps/software/devel/gentoo/tmp/usr/share/gcc-data/x86_64-pc-linux-gnu/10.3.0 --mandir=/nas/weka/resources/tools/apps/software/devel/gentoo/tmp/usr/share/gcc-data/x86_64-pc-linux-gnu/10.3.0/man --infodir=/nas/weka/resources/tools/apps/software/devel/gentoo/tmp/usr/share/gcc-data/x86_64-pc-linux-gnu/10.3.0/info --with-gxx-include-dir=/nas/weka/resources/tools/apps/software/devel/gentoo/tmp/usr/lib/gcc/x86_64-pc-linux-gnu/10.3.0/include/g++-v10 --with-python-dir=/share/gcc-data/x86_64-pc-linux-gnu/10.3.0/python --enable-languages=c,c++ --enable-obsolete --enable-secureplt --disable-werror --with-system-zlib --disable-nls --disable-libunwind-exceptions --enable-checking=release --with-bugurl=https://bugs.gentoo.org/ --with-pkgversion='Gentoo 10.3.0-r2 p3' --disable-esp --enable-libstdcxx-time --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --disable-multilib --with-multilib-list=m64 --disable-fixed-point --enable-targets=all --enable-libgomp --disable-libssp --disable-libada --disable-systemtap --disable-vtable-verify --disable-libvtv --without-zstd --enable-lto --without-isl --disable-libsanitizer --enable-default-pie --enable-default-ssp --disable-bootstrap --with-linker-hash-style=both --with-local-prefix=/nas/weka/resources/tools/apps/software/devel/gentoo
Thread model: posix
Supported LTO compression algorithms: zlib
gcc version 10.3.0 (Gentoo 10.3.0-r2 p3)
(base) pietro003:~$
Comment 11 Tee KOBAYASHI 2021-11-18 13:11:55 UTC
From config.log:

configure:4383: x86_64-pc-linux-gnu-gcc -O2 -pipe -O2 -pipe  -L/nas/weka/resources/tools/apps/software/devel/gentoo/usr/lib64 -Wl,--dynamic-linker=/nas/weka/resources/tools/apps/software/devel/gentoo/lib64/ld-linux-x86-64.so.2 conftest.c  >&5
/nas/weka/resources/tools/apps/software/devel/gentoo/tmp/usr/lib/gcc/x86_64-pc-linux-gnu/10.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: /lib/../lib64/Scrt1.o: in function `_start':
(.text+0x12): undefined reference to `__libc_csu_fini'
/nas/weka/resources/tools/apps/software/devel/gentoo/tmp/usr/lib/gcc/x86_64-pc-linux-gnu/10.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: (.text+0x19): undefined reference to `__libc_csu_init'
collect2: error: ld returned 1 exit status
configure:4387: $? = 1

The toolchain fails to link an executable that is simpler than hello world.
Comment 12 Benjamin Block 2021-11-19 11:11:29 UTC
I have the same problem with bootstrapping on a Fedora 34; same errors in the `var/tmp/portage/sys-devel/binutils-2.37_p1-r1/work/build/config.log`:

configure:4321: checking for C compiler version
configure:4330: x86_64-pc-linux-gnu-gcc --version >&5
x86_64-pc-linux-gnu-gcc (Gentoo 10.3.0-r2 p3) 10.3.0
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

configure:4341: $? = 0
configure:4330: x86_64-pc-linux-gnu-gcc -v >&5
Using built-in specs.
COLLECT_GCC=x86_64-pc-linux-gnu-gcc
COLLECT_LTO_WRAPPER=/home/share/gentoo/tmp/usr/libexec/gcc/x86_64-pc-linux-gnu/10.3.0/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: /home/share/gentoo/tmp/var/tmp/portage/sys-devel/gcc-10.3.0-r2/work/gcc-10.3.0/configure --host=x86_64-pc-linux-gnu --build=x86_64-pc-linux-gnu --prefix=/home/share/gentoo/tmp/usr --bindir=/home/share/gentoo/tmp/usr/x86_64-pc-linux-gnu/gcc-bin/10.3.0 --inc
ludedir=/home/share/gentoo/tmp/usr/lib/gcc/x86_64-pc-linux-gnu/10.3.0/include --datadir=/home/share/gentoo/tmp/usr/share/gcc-data/x86_64-pc-linux-gnu/10.3.0 --mandir=/home/share/gentoo/tmp/usr/share/gcc-data/x86_64-pc-linux-gnu/10.3.0/man --infodir=/home/share/gentoo/tmp/u
sr/share/gcc-data/x86_64-pc-linux-gnu/10.3.0/info --with-gxx-include-dir=/home/share/gentoo/tmp/usr/lib/gcc/x86_64-pc-linux-gnu/10.3.0/include/g++-v10 --with-python-dir=/share/gcc-data/x86_64-pc-linux-gnu/10.3.0/python --enable-languages=c,c++ --enable-obsolete --enable-se
cureplt --disable-werror --with-system-zlib --disable-nls --disable-libunwind-exceptions --enable-checking=release --with-bugurl=https://bugs.gentoo.org/ --with-pkgversion='Gentoo 10.3.0-r2 p3' --disable-esp --enable-libstdcxx-time --enable-shared --enable-threads=posix --
enable-__cxa_atexit --enable-clocale=gnu --disable-multilib --with-multilib-list=m64 --disable-fixed-point --enable-targets=all --enable-libgomp --disable-libssp --disable-libada --disable-systemtap --disable-vtable-verify --disable-libvtv --without-zstd --enable-lto --wit
hout-isl --disable-libsanitizer --enable-default-pie --enable-default-ssp --disable-bootstrap --with-linker-hash-style=both --with-local-prefix=/home/share/gentoo
Thread model: posix
Supported LTO compression algorithms: zlib
gcc version 10.3.0 (Gentoo 10.3.0-r2 p3)
configure:4341: $? = 0
configure:4330: x86_64-pc-linux-gnu-gcc -V >&5
x86_64-pc-linux-gnu-gcc: error: unrecognized command-line option '-V'
x86_64-pc-linux-gnu-gcc: fatal error: no input files
compilation terminated.
configure:4341: $? = 1
configure:4330: x86_64-pc-linux-gnu-gcc -qversion >&5
x86_64-pc-linux-gnu-gcc: error: unrecognized command-line option '-qversion'; did you mean '--version'?
x86_64-pc-linux-gnu-gcc: fatal error: no input files
compilation terminated.
configure:4341: $? = 1
configure:4361: checking whether the C compiler works
configure:4383: x86_64-pc-linux-gnu-gcc -O2 -pipe -O2 -pipe  -L/home/share/gentoo/usr/lib64 -Wl,--dynamic-linker=/home/share/gentoo/lib64/ld-linux-x86-64.so.2 conftest.c  >&5
/home/share/gentoo/tmp/usr/lib/gcc/x86_64-pc-linux-gnu/10.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: /lib/../lib64/Scrt1.o: in function `_start':
(.text+0x16): undefined reference to `__libc_csu_fini'
/home/share/gentoo/tmp/usr/lib/gcc/x86_64-pc-linux-gnu/10.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: (.text+0x1d): undefined reference to `__libc_csu_init'
collect2: error: ld returned 1 exit status
configure:4387: $? = 1
configure:4425: result: no
configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME ""
| #define PACKAGE_TARNAME ""
| #define PACKAGE_VERSION ""
| #define PACKAGE_STRING ""
| #define PACKAGE_BUGREPORT ""
| #define PACKAGE_URL ""
| /* end confdefs.h.  */
|
| int
| main ()
| {
|
|   ;
|   return 0;
| }
configure:4430: error: in `/home/share/gentoo/var/tmp/portage/sys-devel/binutils-2.37_p1-r1/work/build':
configure:4432: error: C compiler cannot create executables

Seems like it's missing some symbol `__libc_csu_init`. AFAICT this uses RAP - I didn't chnage anything about the defaults.
Comment 13 Pietro 2021-11-19 11:39:37 UTC
Actually prefix is being bootstrapped from a CentOS 7 host, and realized I was within a Conda environment. I do apologize for the confusion. I am now trying to bootstrap it from a Debian 10 host using the desired $EPREFIX PATH and will rsync it once and if done. Hopefully that's going to work...
Comment 14 Tee KOBAYASHI 2021-11-19 13:21:35 UTC
Reference to __libc_csu_init and __libc_csu_fini should be resolved by linking against ${ROOT}/usr/$(get_libdir)/libc_nonshared.a coming from glibc, which I would expect should be done by default.
Comment 15 Tee KOBAYASHI 2021-11-20 04:11:41 UTC
There seem to be two points.

One is that glibc-2.34 does not have __libc_csu_{init,fini}. They are not defined nor used anywhere inside the prefix system.

Another is that the toolchain is going to use startfiles (*crt*.o) outside the prefix system, as shown in config.log:
 
/nas/weka/resources/tools/apps/software/devel/gentoo/tmp/usr/lib/gcc/x86_64-pc-linux-gnu/10.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: /lib/../lib64/Scrt1.o: in function `_start':

Here I would expect /nas/weka/resources/tools/apps/software/devel/gentoo/tmp/usr/lib64/Scrt1.o (from glibc-2.34, inside prefix) be used instead of /lib64/Scrt1.o (from pre glibc-2.34, outside prefix, I suppose).
Comment 16 Fabian Groffen gentoo-dev 2021-11-20 08:34:05 UTC
(In reply to Tee KOBAYASHI from comment #15)
> There seem to be two points.
> 
> One is that glibc-2.34 does not have __libc_csu_{init,fini}. They are not
> defined nor used anywhere inside the prefix system.
> 
> Another is that the toolchain is going to use startfiles (*crt*.o) outside
> the prefix system, as shown in config.log:
>  
> /nas/weka/resources/tools/apps/software/devel/gentoo/tmp/usr/lib/gcc/x86_64-
> pc-linux-gnu/10.3.0/../../../../x86_64-pc-linux-gnu/bin/ld:
> /lib/../lib64/Scrt1.o: in function `_start':
> 
> Here I would expect
> /nas/weka/resources/tools/apps/software/devel/gentoo/tmp/usr/lib64/Scrt1.o
> (from glibc-2.34, inside prefix) be used instead of /lib64/Scrt1.o (from pre
> glibc-2.34, outside prefix, I suppose).

that makes sense

Just for my understanding, Scrt1.o in the place you would expect, does exist in your case, right?
Comment 17 Tee KOBAYASHI 2021-11-20 09:06:23 UTC
(In reply to Fabian Groffen from comment #16)
> Just for my understanding, Scrt1.o in the place you would expect, does exist
> in your case, right?

Not really. I have no prefix environment with me.

As for crossdev, I have the file /usr/sparc64-unknown-linux-gnu/usr/lib64/Scrt1.o that comes from cross-sparc64-unknown-linux-gnu/glibc-2.34-r2. So I guess it should also exist in prefix.
Comment 18 Kenneth Hoste 2021-11-20 17:13:25 UTC
I'm seeing the same issue when bootstrapping Gentoo Prefix in CentOS 8.4, both on aarch64 (a Graviton2 VM in AWS) and ppc64le (emulated via QEMU).

Anything I can look into or try to help sort this out?

From the config.log on aarch64:

configure:4361: checking whether the C compiler works
configure:4383: aarch64-unknown-linux-gnu-gcc -O2 -O2 -pipe  -L/cvmfs/pilot.eessi-hpc.org/2021.12/compat/linux/aarch64/usr/lib64 -Wl,--dynamic-linker=/cvmfs/pilot.eessi-hpc.org/2021.12/compat/linux/aarch64/lib64/ld-linux-aarch64.so.1 conftest.c  >&5
/cvmfs/pilot.eessi-hpc.org/2021.12/compat/linux/aarch64/tmp/usr/lib/gcc/aarch64-unknown-linux-gnu/10.3.0/../../../../aarch64-unknown-linux-gnu/bin/ld: /lib/../lib64/Scrt1.o: in function `_start':
(.text+0x20): undefined reference to `__libc_csu_init'
/cvmfs/pilot.eessi-hpc.org/2021.12/compat/linux/aarch64/tmp/usr/lib/gcc/aarch64-unknown-linux-gnu/10.3.0/../../../../aarch64-unknown-linux-gnu/bin/ld: (.text+0x24): undefined reference to `__libc_csu_init'
/cvmfs/pilot.eessi-hpc.org/2021.12/compat/linux/aarch64/tmp/usr/lib/gcc/aarch64-unknown-linux-gnu/10.3.0/../../../../aarch64-unknown-linux-gnu/bin/ld: (.text+0x28): undefined reference to `__libc_csu_fini'
/cvmfs/pilot.eessi-hpc.org/2021.12/compat/linux/aarch64/tmp/usr/lib/gcc/aarch64-unknown-linux-gnu/10.3.0/../../../../aarch64-unknown-linux-gnu/bin/ld: (.text+0x2c): undefined reference to `__libc_csu_fini'
collect2: error: ld returned 1 exit status
Comment 19 Kenneth Hoste 2021-11-20 17:32:58 UTC
> Just for my understanding, Scrt1.o in the place you would expect, does exist
> in your case, right?

For me, Scrt1.o is indeed there where expected in the Prefix installation:

/cvmfs/pilot.eessi-hpc.org/2021.12/compat/linux/aarch64/usr/lib64/Scrt1.o
 
So the culprit is that the binutils build is not Prefix-aware, since it picks up  lib64/Scrt1.o, despite the correct -L being specified in the test compiler command:

aarch64-unknown-linux-gnu-gcc -O2 -O2 -pipe  -L/cvmfs/pilot.eessi-hpc.org/2021.12/compat/linux/aarch64/usr/lib64 -Wl,--dynamic-linker=/cvmfs/pilot.eessi-hpc.org/2021.12/compat/linux/aarch64/lib64/ld-linux-aarch64.so.1 conftest.c
Comment 20 Kenneth Hoste 2021-11-20 21:51:03 UTC
Some additional info after playing around with older portage snapshots: this problem does not occur when using the 20211112 snapshot of portage, because then  binutils-2.37_p1 is installed instead. With the 20211113 snapshot, the same problem occurs.

So it seems like this problem got introduced by the changes in https://gitweb.gentoo.org/repo/gentoo.git/commit/sys-devel/binutils?id=90c8ce89b326ff688cbd0f05eef458c530e4e7e8
Comment 21 Larry the Git Cow gentoo-dev 2021-11-20 23:32:19 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=409923c61c08bb0c46bf883af7b4c6c1f9bf859e

commit 409923c61c08bb0c46bf883af7b4c6c1f9bf859e
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2021-11-20 23:31:48 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2021-11-20 23:31:48 +0000

    profiles/features/prefix: mask problematic Binutils version (2.37_p1-r1)
    
    Doesn't seem to respect prefix search paths. Mask pending
    further investigation to fix bootstraps.
    
    Bug: https://bugs.gentoo.org/824482
    Signed-off-by: Sam James <sam@gentoo.org>

 profiles/features/prefix/package.mask | 6 ++++++
 1 file changed, 6 insertions(+)
Comment 22 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-11-21 01:09:45 UTC
(In reply to Kenneth Hoste from comment #18)
> I'm seeing the same issue when bootstrapping Gentoo Prefix in CentOS 8.4,
> both on aarch64 (a Graviton2 VM in AWS) and ppc64le (emulated via QEMU).
> 
> Anything I can look into or try to help sort this out?
> 

Thanks for tracking down the exact Binutils version which broke. The next step would be diffing the 2.37_p1 and 2.37_p1-r1 tarballs (there's new patches from upstream in -r1, grabbed from https://sourceware.org/git/?p=binutils-gdb.git;a=shortlog;h=refs/heads/binutils-2_37-branch) and seeing what broke it, I think..
Comment 23 Kenneth Hoste 2021-11-21 17:24:23 UTC
After taking a closer look and more playing around, it's now clear it wasn't about binutils 2.37_p1 vs 2.37_p1-r1 at all.

The reason I wasn't seeing the problem with the 20211112 snapshot is because glibc 2.34 wasn't in there yet:

>>> Emerging (1 of 1) sys-libs/glibc-2.33-r7::gentoo

So the masking of binutils-2.37_p1-r1 doesn't actually fix the bootstrap of prefix at all, glibc-2.34 should be masked instead.
Comment 24 Bob Dröge 2021-11-21 17:50:31 UTC
(In reply to Kenneth Hoste from comment #23)
> After taking a closer look and more playing around, it's now clear it wasn't
> about binutils 2.37_p1 vs 2.37_p1-r1 at all.
> 
> The reason I wasn't seeing the problem with the 20211112 snapshot is because
> glibc 2.34 wasn't in there yet:
> 
> >>> Emerging (1 of 1) sys-libs/glibc-2.33-r7::gentoo
> 
> So the masking of binutils-2.37_p1-r1 doesn't actually fix the bootstrap of
> prefix at all, glibc-2.34 should be masked instead.


I can confirm this: I saw the same issue on x86_64 with a snapshot of two days ago (and not using the stable x86_64 bootstrap option), and I still saw it with the new snapshot. By masking glibc/2.34 the issue has disappeared for me.
Comment 25 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-11-22 05:10:06 UTC
(In reply to Kenneth Hoste from comment #23)
> After taking a closer look and more playing around, it's now clear it wasn't
> about binutils 2.37_p1 vs 2.37_p1-r1 at all.
> 
> The reason I wasn't seeing the problem with the 20211112 snapshot is because
> glibc 2.34 wasn't in there yet:
> 
> >>> Emerging (1 of 1) sys-libs/glibc-2.33-r7::gentoo
> 
> So the masking of binutils-2.37_p1-r1 doesn't actually fix the bootstrap of
> prefix at all, glibc-2.34 should be masked instead.

This is going to be more tricky to handle given downgrading of glibc isn't something we can do. We can emerge a specific version in the bootstrap script, but if it breaks once it upgrades, that's going to be problematic.

And I don't think we can really add a mask within the script because then it'll sit there forever on bootstrapped prefixes...

Does glibc-2.34 work _once bootstrapped_?
Comment 26 Fabian Groffen gentoo-dev 2021-11-22 07:41:35 UTC
we can start adding the mask for new bootstraps, add a comment for the user with some pointers, then when glibc gets fixed, I'd expect a new revision or version that won't be caught by the mask
Comment 27 Larry the Git Cow gentoo-dev 2021-11-22 07:57:51 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=778b44c821e22fa41cdab710a8c13db024d945e8

commit 778b44c821e22fa41cdab710a8c13db024d945e8
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2021-11-22 07:56:44 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2021-11-22 07:57:44 +0000

    profiles/features/prefix: drop binutils mask
    
    It looks like the issue is glibc-2.34, not binutils.
    
    Bug: https://bugs.gentoo.org/824482
    Signed-off-by: Sam James <sam@gentoo.org>

 profiles/features/prefix/package.mask | 6 ------
 1 file changed, 6 deletions(-)
Comment 28 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-11-24 15:56:40 UTC
*** Bug 827164 has been marked as a duplicate of this bug. ***
Comment 29 Benjamin Block 2021-11-29 07:46:57 UTC
Created attachment 756986 [details, diff]
patch-bug-824482-mask-glibc-2.34.patch

(In reply to Fabian Groffen from comment #26)
> we can start adding the mask for new bootstraps, add a comment for the user
> with some pointers, then when glibc gets fixed, I'd expect a new revision or
> version that won't be caught by the mask

Just FYI, doing that with the attached patch to the bootstrap script eventually allowed me to successfully bootstrap a (non-stable) prefix (it would always fail otherwise: comment #12). It's kinda steamrolling the problem.. but it works for now.
Comment 30 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-12-13 06:26:16 UTC
(In reply to Benjamin Block from comment #29)
> Created attachment 756986 [details, diff] [details, diff]
> patch-bug-824482-mask-glibc-2.34.patch
> 
> (In reply to Fabian Groffen from comment #26)
> > we can start adding the mask for new bootstraps, add a comment for the user
> > with some pointers, then when glibc gets fixed, I'd expect a new revision or
> > version that won't be caught by the mask
> 
> Just FYI, doing that with the attached patch to the bootstrap script
> eventually allowed me to successfully bootstrap a (non-stable) prefix (it
> would always fail otherwise: comment #12). It's kinda steamrolling the
> problem.. but it works for now.

grobian, this ok? Not sure what kind of revision we want to pick instead of using ~ though.
Comment 31 Fabian Groffen gentoo-dev 2021-12-13 07:08:16 UTC
yes that works for me, thanks
Comment 32 Larry the Git Cow gentoo-dev 2021-12-22 23:58:05 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/proj/prefix.git/commit/?id=f1ba8d24da137ddcd0d0ba6c68d1f5e4b37635ef

commit f1ba8d24da137ddcd0d0ba6c68d1f5e4b37635ef
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2021-12-22 23:55:44 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2021-12-22 23:56:39 +0000

    scripts/bootstrap-prefix.sh: mask glibc-2.34 due to bootstrapping issues
    
    Mask <sys-libs/glibc-2.34_p1 for now due to bootstrapping issues. Note that
    _p1 doesn't exist but I wanted to use _something_ to allow us to push
    glibc-2.34 to prefix users in future and _p1 is easier than guessing a revision.
    
    We rarely use _pN in ::gentoo.
    
    Bug: https://bugs.gentoo.org/824482
    Signed-off-by: Sam James <sam@gentoo.org>

 scripts/bootstrap-prefix.sh | 12 ++++++++++++
 1 file changed, 12 insertions(+)
Comment 33 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-12-22 23:58:37 UTC
(feel free to adapt grobian if desired, just wanted to get something in and figured this was a decent choice for now)
Comment 34 Larry the Git Cow gentoo-dev 2022-01-22 18:51:24 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/proj/prefix.git/commit/?id=8a6d831adcc3edc23c751d6fe6f600765bcc37b2

commit 8a6d831adcc3edc23c751d6fe6f600765bcc37b2
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2022-01-22 18:51:00 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2022-01-22 18:51:00 +0000

    scripts/bootstrap-prefix.sh: fix glibc mask/unmask logic
    
    Closes: https://bugs.gentoo.org/824482
    Signed-off-by: Sam James <sam@gentoo.org>

 scripts/bootstrap-prefix.sh | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
Comment 35 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2022-03-07 06:27:47 UTC
Not had any real time/motivation to poke at this. Help very much welcome if anyone has ideas or time to look into it a bit.

glibc-2.34 is about to be stable in ::gentoo and as time goes on, we'll be looking to drop 2.33, so this is becoming more important :(
Comment 36 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2022-03-07 06:29:51 UTC
(In reply to Sam James from comment #35)
> Not had any real time/motivation to poke at this. Help very much welcome if
> anyone has ideas or time to look into it a bit.
> 
> glibc-2.34 is about to be stable in ::gentoo and as time goes on, we'll be
> looking to drop 2.33, so this is becoming more important :(

A good start would be:
1. diffing the last 2.33 ebuild which worked with 2.34
2. figuring out if it was an ebuild change which caused the issue
3. diffing the build system parts of 2.33 and 2.34 (just configure.ac, Makefile.am should be enough)
Comment 37 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2022-03-07 06:38:23 UTC
(In reply to Sam James from comment #36)
> (In reply to Sam James from comment #35)
> > Not had any real time/motivation to poke at this. Help very much welcome if
> > anyone has ideas or time to look into it a bit.
> > 
> > glibc-2.34 is about to be stable in ::gentoo and as time goes on, we'll be
> > looking to drop 2.33, so this is becoming more important :(
> 
> A good start would be:
> 1. diffing the last 2.33 ebuild which worked with 2.34
> 2. figuring out if it was an ebuild change which caused the issue
> 3. diffing the build system parts of 2.33 and 2.34 (just configure.ac,
> Makefile.am should be enough)

I wonder if this is related to https://github.com/NixOS/nixpkgs/issues/158042. Obviously that's about their own wrapper partly but it's interesting in the sense that NixOS has to handle essentially prefixes too, and may be hitting similar problems.

Notably nix is still on 2.33 and they're only seeing this when moving to 2.35, which doesn't rule out 2.34 (and the commit below looks like it's from 2.34):
>My theory is that it was not visible before because until glibc-2.35 crt1.o did not change much. But upstream sourceware.org/git/?p=glibc.git;a=commitdiff;h=035c012e32c11e84d64905efaf55e74f704d3668 change made it visible.

(I'll CC slyfox just in case he's interested too, as if nothing else, it may be useful for him if it turns out nix is hating this same bug.)
Comment 38 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2022-03-07 06:39:18 UTC
(In reply to Sam James from comment #37)

hitting*
Comment 39 Sergei Trofimovich 2022-03-07 08:33:46 UTC
At least for `staging` branch nixpkgs is able to bootstrap into 2.35 with already submitted hack of https://github.com/NixOS/nixpkgs/commit/649ebfbed65189d7d62e4f2fe0e491552308a6f1 (at least that's what I use successfully for my glibc-2.35 system). [Note that nixpkgs bootstraps today from glibc-2.27. It's likely not important for this case but it might not cover a few other breakage scenarios related to removal of libraries from glibc.]

The bootstrap trick is to avoid accidental mixing of crt*.o from new glibc and libc.so/ld.so from old glibc. Or the other way around (your case).

Currently nixpkgs uses careful mix of -L/-B/-Wl,-dynamic-linker options for bootstrap phase. That should be close enough for Gentoo prefix case. [Eventually it will craft /lib/ directory that contains only one of glibc version at any point of time to avoid reliance on directory ordering.]

Looking specifically at the configure failure above:

"""
configure:4383: x86_64-pc-linux-gnu-gcc -O2 -pipe -O2 -pipe  -L/home/share/gentoo/usr/lib64 -Wl,--dynamic-linker=/home/share/gentoo/lib64/ld-linux-x86-64.so.2 conftest.c  >&5
/home/share/gentoo/tmp/usr/lib/gcc/x86_64-pc-linux-gnu/10.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: /lib/../lib64/Scrt1.o: in function `_start':
(.text+0x16): undefined reference to `__libc_csu_fini'
/home/share/gentoo/tmp/usr/lib/gcc/x86_64-pc-linux-gnu/10.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: (.text+0x1d): undefined reference to `__libc_csu_init'
"""

I think the missing option is -B/home/share/gentoo/usr/lib64/ to direct gcc to correct Scrt1.o. A assume x86_64-pc-linux-gnu-gcc is still host gcc and not prefix gcc.

Or mechanically change https://gitweb.gentoo.org/repo/proj/prefix.git/plain/scripts/bootstrap-prefix.sh

from
  export LDFLAGS="-L${ROOT}/usr/$(get_libdir) -Wl,--dynamic-linker=${RAP_DLINKER}"
to
  export LDFLAGS="-B${ROOT}/usr/$(get_libdir) -L${ROOT}/usr/$(get_libdir) -Wl,--dynamic-linker=${RAP_DLINKER}"
Comment 40 Bart Oldeman 2022-08-29 12:38:52 UTC
I opened https://github.com/gentoo/gentoo/pull/27057 so custom-cflags can be used in the bootstrap allowing -B not to be stripped and binutils to be built.

I did a few experiments to see what's going on (some of this is also in Sergei Trofimovich's comment above, it's just confirming that)

the stage2 gcc has two correct modus operandi:
1. plain, no flags, link to system glibc
2. with flags, link to stage3 glibc

The problem is that at present it links to *crt*.o from system glibc but to libc_nonshared.a and libc.so from stage3 glibc. This has worked by accident.

If you pass -B${ROOT}/usr/$(get_libdir) (with or without -L${ROOT}/usr/$(get_libdir) !!) you'll get all files from stage3 glibc, so you get correct situation 2.

I also tried some other alternatives:
a. LIBRARY_PATH=${ROOT}/usr/$(get_libdir). This also picks up *crt*.o from the right place but breaks situation 1, as some configure scripts ignore LDFLAGS in test compilations and some helper utilities.
b. symlinking ${ROOT}/tmp/usr/$(get_libdir)/*crt*.o to ${ROOT}/usr/$(get_libdir)
This causes gcc to pick up stage3 glibc's *crt*.o without special flags. But this also breaks situation 1, as you now combine new *crt*.o with system libc*.
Comment 41 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2022-08-29 20:41:49 UTC
(In reply to Bart Oldeman from comment #40)
> I opened https://github.com/gentoo/gentoo/pull/27057 so custom-cflags can be
> used in the bootstrap allowing -B not to be stripped and binutils to be
> built.
> 
> I did a few experiments to see what's going on (some of this is also in
> Sergei Trofimovich's comment above, it's just confirming that)
> 
> the stage2 gcc has two correct modus operandi:
> 1. plain, no flags, link to system glibc
> 2. with flags, link to stage3 glibc
> 
> The problem is that at present it links to *crt*.o from system glibc but to
> libc_nonshared.a and libc.so from stage3 glibc. This has worked by accident.
> 
> If you pass -B${ROOT}/usr/$(get_libdir) (with or without
> -L${ROOT}/usr/$(get_libdir) !!) you'll get all files from stage3 glibc, so
> you get correct situation 2.
> 
> I also tried some other alternatives:
> a. LIBRARY_PATH=${ROOT}/usr/$(get_libdir). This also picks up *crt*.o from
> the right place but breaks situation 1, as some configure scripts ignore
> LDFLAGS in test compilations and some helper utilities.
> b. symlinking ${ROOT}/tmp/usr/$(get_libdir)/*crt*.o to
> ${ROOT}/usr/$(get_libdir)
> This causes gcc to pick up stage3 glibc's *crt*.o without special flags. But
> this also breaks situation 1, as you now combine new *crt*.o with system
> libc*.

1. Coulld we have the ebuild always pass -B ... and -L ... on prefix safely? Or is it only set temporarily during bootstrap-prefix.sh?

2. COuld/should strip-flags ignore -B and -L?
Comment 42 Bart Oldeman 2022-09-01 17:41:31 UTC
(In reply to Sam James from comment #41)

> 1. Coulld we have the ebuild always pass -B ... and -L ... on prefix safely?
> Or is it only set temporarily during bootstrap-prefix.sh?

it's only a temporary phase during stage3, if I do "grep emerge stage3.log" it's the part between ====

stage3.log:>>> Emerging (1 of 1) sys-apps/baselayout-2.8-r2::gentoo
stage3.log:>>> Emerging (1 of 1) sys-apps/gentoo-functions-0.17::gentoo
stage3.log:>>> Emerging (1 of 1) app-portage/elt-patches-20220831::gentoo
stage3.log:>>> Emerging (1 of 1) sys-kernel/linux-headers-5.19::gentoo
stage3.log:>>> Emerging (1 of 1) sys-libs/glibc-2.35-r8::gentoo
====
stage3.log:>>> Emerging (1 of 1) sys-devel/binutils-config-5.4.1::gentoo
stage3.log:>>> Emerging (1 of 1) sys-libs/zlib-1.2.12-r3::gentoo
stage3.log:>>> Emerging (1 of 1) sys-devel/binutils-2.38-r2::gentoo
stage3.log:>>> Emerging (1 of 1) sys-apps/attr-2.5.1-r2::gentoo
stage3.log:>>> Emerging (1 of 1) sys-libs/libcap-2.65::gentoo
stage3.log:>>> Emerging (1 of 1) sys-libs/libxcrypt-4.4.28-r1::gentoo
stage3.log:>>> Emerging (1 of 1) dev-libs/gmp-6.2.1-r2::gentoo
stage3.log:>>> Emerging (1 of 1) dev-libs/mpfr-4.1.0_p13-r1::gentoo
stage3.log:>>> Emerging (1 of 1) dev-libs/mpc-1.2.1::gentoo
stage3.log:>>> Emerging (1 of 1) sys-devel/gcc-config-2.5-r1::gentoo
stage3.log:>>> Emerging (1 of 1) sys-devel/gcc-12.2.0::gentoo
====
stage3.log:>>> Emerging (1 of 1) sys-apps/gawk-5.1.1-r2::gentoo
stage3.log:>>> Emerging (1 of 103) sys-devel/gnuconfig-20220508::gentoo

once stage3 gcc is built all is fairly safe and we can forget about -B, -L and -Wl,--dynamic-linker (the script does
        unset CXX CPPFLAGS LDFLAGS
there)
 
> 2. COuld/should strip-flags ignore -B and -L?
I'm not sure here.
There's another workaround, which is to fold -B and friends into "$CC" and "$CXX", ie. CC="gcc $CPPFLAGS $LDFLAGS" This also makes sure that *everything* is linked to the new glibc, including configure test programs and helpers that ignore LDFLAGS and CPPFLAGS.

I'm attaching a patch that does that. Bootstrapped successfully with GCC 12.2 and glibc 2.35.
Comment 43 Bart Oldeman 2022-09-01 17:43:03 UTC
Created attachment 802501 [details, diff]
Fold CPPFLAGS/LDFLAGS in CC/CXX to allow new glibc in bootstrap stage3
Comment 44 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2022-09-02 00:45:33 UTC
(In reply to Bart Oldeman from comment #43)
> Created attachment 802501 [details, diff] [details, diff]
> Fold CPPFLAGS/LDFLAGS in CC/CXX to allow new glibc in bootstrap stage3

I think it's cheesy but we've done worse in the bootstrap script. Fine with me but want grobian's ack before committing.

Thanks for trying that!
Comment 45 Fabian Groffen gentoo-dev 2022-09-02 06:21:10 UTC
Since this is in the RAP path (only) this should be OK.  Don't think you'd have to conditionalise for GCC, as it is the only host-compiler we know about for Gentoo Linux at the moment.

Thanks for your hard work to try and figure this out!  Let's apply this patch, Sam, will you do the honours?
Comment 46 Bart Oldeman 2022-09-02 11:11:07 UTC
(In reply to Fabian Groffen from comment #45)
> Since this is in the RAP path (only) this should be OK.  Don't think you'd
> have to conditionalise for GCC, as it is the only host-compiler we know
> about for Gentoo Linux at the moment.

If I read things correctly, if you bootstrap on Mac OS (Darwin), it explicitly sets compiler_type to clang in configure_toolchain().

Since I have no Darwin readily available to test on, and doubt the issue applies there I was cautious here.

Thanks for applying this patch!
Comment 47 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2022-09-03 01:54:10 UTC
(In reply to Fabian Groffen from comment #45)
> Since this is in the RAP path (only) this should be OK.  Don't think you'd
> have to conditionalise for GCC, as it is the only host-compiler we know
> about for Gentoo Linux at the moment.
> 
> Thanks for your hard work to try and figure this out!  Let's apply this
> patch, Sam, will you do the honours?

happily, thanks fabian! :)
Comment 48 Larry the Git Cow gentoo-dev 2022-09-03 01:59:17 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/proj/prefix.git/commit/?id=d15a8504e74114febbbbee439493a282f4450c5c

commit d15a8504e74114febbbbee439493a282f4450c5c
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2022-09-03 01:55:19 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2022-09-03 01:55:19 +0000

    scripts/bootstrap-prefix.sh: fix for >= glibc-2.35 in stage3
    
    slyfox explains it well in the linked bugs, but
    the gist is that we mostly got lucky for a while.
    
    We would mix system crt*.o with just-built libc_nonshared.a/libc.so
    which led to issues like:
    ```
    configure:4383: x86_64-pc-linux-gnu-gcc -O2 -pipe -O2 -pipe  -L/home/share/gentoo/usr/lib64 -Wl,--dynamic-linker=/home/share/gentoo/lib64/ld-linux-x86-64.so.2 conftest.c  >&5
    /home/share/gentoo/tmp/usr/lib/gcc/x86_64-pc-linux-gnu/10.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: /lib/../lib64/Scrt1.o: in function `_start':
    (.text+0x16): undefined reference to `__libc_csu_fini'
    /home/share/gentoo/tmp/usr/lib/gcc/x86_64-pc-linux-gnu/10.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: (.text+0x1d): undefined reference to `__libc_csu_init'
    ```
    
    We need to force GCC to use the Prefix version
    of glibc we just built.
    
    Bug: https://github.com/NixOS/nixpkgs/issues/158042
    Closes: https://bugs.gentoo.org/824482
    Thanks-to: Bart Oldeman <bartoldeman@gmail.com>
    Thanks-to: Sergei Trofimovich <slyich@gmail.com>
    Signed-off-by: Sam James <sam@gentoo.org>

 scripts/bootstrap-prefix.sh | 39 ++++++++-------------------------------
 1 file changed, 8 insertions(+), 31 deletions(-)