This was discovered in #gentoo-haskell while working on ghc builds. In Gentoo, sys-devel/gcc:11 is missing arm64 _sync atomics in libgcc, but they are correctly present in sys-devel/gcc:12 (tested with 11.3.0 and 12.2.0 respectively). The problem can be observed in the arm64 stage3's: $ nm stage3-arm64-openrc-20220828T233145Z/usr/lib/gcc/aarch64-unknown-linux-gnu/11.3.0/libgcc.a 2>/dev/null | grep "_sync" U __aarch64_sync_cache_range 0000000000000000 T __aarch64_sync_cache_range But here's what it looks like on a system running gcc 12: $ nm /usr/lib/gcc/aarch64-unknown-linux-gnu/12.2.0/libgcc.a 2>/dev/null | grep "_sync" 0000000000000000 T __aarch64_cas1_sync 0000000000000000 T __aarch64_cas2_sync 0000000000000000 T __aarch64_cas4_sync 0000000000000000 T __aarch64_cas8_sync 0000000000000000 T __aarch64_cas16_sync 0000000000000000 T __aarch64_swp1_sync 0000000000000000 T __aarch64_ldadd1_sync 0000000000000000 T __aarch64_ldclr1_sync 0000000000000000 T __aarch64_ldeor1_sync 0000000000000000 T __aarch64_ldset1_sync 0000000000000000 T __aarch64_swp2_sync 0000000000000000 T __aarch64_ldadd2_sync 0000000000000000 T __aarch64_ldclr2_sync 0000000000000000 T __aarch64_ldeor2_sync 0000000000000000 T __aarch64_ldset2_sync 0000000000000000 T __aarch64_swp4_sync 0000000000000000 T __aarch64_ldadd4_sync 0000000000000000 T __aarch64_ldclr4_sync 0000000000000000 T __aarch64_ldeor4_sync 0000000000000000 T __aarch64_ldset4_sync 0000000000000000 T __aarch64_swp8_sync 0000000000000000 T __aarch64_ldadd8_sync 0000000000000000 T __aarch64_ldclr8_sync 0000000000000000 T __aarch64_ldeor8_sync 0000000000000000 T __aarch64_ldset8_sync U __aarch64_sync_cache_range 0000000000000000 T __aarch64_sync_cache_range These are specifically defined in files matching *_5.o in the archive, for example: cas_1_5.o: 0000000000000000 T __aarch64_cas1_sync U __aarch64_have_lse_atomics But the gcc 11 build is missing these files, and only these files, from its archive: $ ar t stage3-arm64-openrc-20220828T233145Z/usr/lib/gcc/aarch64-unknown-linux-gnu/11.3.0/libgcc.a > 1.txt $ ar t /usr/lib/gcc/aarch64-unknown-linux-gnu/12.2.0/libgcc.a > 2.txt $ diff 1.txt 2.txt 20a21,25 > cas_1_5.o > cas_2_5.o > cas_4_5.o > cas_8_5.o > cas_16_5.o 100a106,125 > swp_1_5.o > ldadd_1_5.o > ldclr_1_5.o > ldeor_1_5.o > ldset_1_5.o > swp_2_5.o > ldadd_2_5.o > ldclr_2_5.o > ldeor_2_5.o > ldset_2_5.o > swp_4_5.o > ldadd_4_5.o > ldclr_4_5.o > ldeor_4_5.o > ldset_4_5.o > swp_8_5.o > ldadd_8_5.o > ldclr_8_5.o > ldeor_8_5.o > ldset_8_5.o These external library functions are needed for the -moutline-atomics option, which is enabled by default: -moutline-atomics -mno-outline-atomics Enable or disable calls to out-of-line helpers to implement atomic operations. These helpers will, at runtime, determine if the LSE instructions from ARMv8.1-A can be used; if not, they will use the load/store-exclusive instructions that are present in the base ARMv8.0 ISA. This option is only applicable when compiling for the base ARMv8.0 instruction set. If using a later revision, e.g. -march=armv8.1-a or -march=armv8-a+lse, the ARMv8.1-Atomics instructions will be used directly. The same applies when using -mcpu= when the selected cpu supports the lse feature. This option is on by default. This means that code generating calls to these functions will fail to link unless it's explicitly disabled (with -mno-outline-atomics or an appropriately recent -march=). I asked the affected user to test rebuilding gcc:11, and this did NOT generate a correct libgcc.a. However building gcc:12 DID generate a correct one that included the functions. To check whether or not this was a Gentoo-specific issue, I downloaded Debian's gcc-11 release and found that it DOES have the correct symbols: $ nm libgcc-11-dev_11.3.0-5_arm64/usr/lib/gcc/aarch64-linux-gnu/11/libgcc.a 2>/dev/null | grep "_sync" 0000000000000000 T __aarch64_cas1_sync 0000000000000000 T __aarch64_cas2_sync 0000000000000000 T __aarch64_cas4_sync 0000000000000000 T __aarch64_cas8_sync 0000000000000000 T __aarch64_cas16_sync 0000000000000000 T __aarch64_swp1_sync 0000000000000000 T __aarch64_ldadd1_sync 0000000000000000 T __aarch64_ldclr1_sync 0000000000000000 T __aarch64_ldeor1_sync 0000000000000000 T __aarch64_ldset1_sync 0000000000000000 T __aarch64_swp2_sync 0000000000000000 T __aarch64_ldadd2_sync 0000000000000000 T __aarch64_ldclr2_sync 0000000000000000 T __aarch64_ldeor2_sync 0000000000000000 T __aarch64_ldset2_sync 0000000000000000 T __aarch64_swp4_sync 0000000000000000 T __aarch64_ldadd4_sync 0000000000000000 T __aarch64_ldclr4_sync 0000000000000000 T __aarch64_ldeor4_sync 0000000000000000 T __aarch64_ldset4_sync 0000000000000000 T __aarch64_swp8_sync 0000000000000000 T __aarch64_ldadd8_sync 0000000000000000 T __aarch64_ldclr8_sync 0000000000000000 T __aarch64_ldeor8_sync 0000000000000000 T __aarch64_ldset8_sync U __aarch64_sync_cache_range 0000000000000000 T __aarch64_sync_cache_range This shows that this is an issue specific to the Gentoo toolchain. Reproducible: Always Portage 3.0.35 (python 3.10.6-final-0, default/linux/arm64/17.0, gcc-12.2.0, glibc-2.35-r8, 5.18.18-gentoo-dist aarch64) ================================================================= System uname: Linux-5.18.18-gentoo-dist-aarch64-with-glibc2.35 KiB Mem: 15767544 total, 3944528 free KiB Swap: 0 total, 0 free Timestamp of repository gentoo: Fri, 02 Sep 2022 02:46:58 +0000 Head commit of repository gentoo: b07498352f9ead901c8dcab33d28e2fa8b9508e6 sh bash 5.1_p16-r2 ld GNU ld (Gentoo 2.38 p4) 2.38 app-misc/pax-utils: 1.3.5::gentoo app-shells/bash: 5.1_p16-r2::gentoo dev-java/java-config: 2.3.1::gentoo dev-lang/perl: 5.36.0::gentoo dev-lang/python: 3.10.6_p2::gentoo, 3.11.0_rc1::gentoo dev-lang/rust: 1.62.1::gentoo dev-util/cmake: 3.24.1::gentoo dev-util/meson: 0.63.1::gentoo sys-apps/baselayout: 2.8-r2::gentoo sys-apps/openrc: 0.45.2::gentoo sys-apps/sandbox: 2.29::gentoo sys-devel/autoconf: 2.71-r1::gentoo sys-devel/automake: 1.16.5::gentoo sys-devel/binutils: 2.38-r2::gentoo sys-devel/binutils-config: 5.4.1::gentoo sys-devel/gcc: 12.2.0::gentoo sys-devel/gcc-config: 2.5-r1::gentoo sys-devel/libtool: 2.4.7::gentoo sys-devel/llvm: 14.0.6-r2::gentoo sys-devel/make: 4.3::gentoo sys-kernel/linux-headers: 5.19::gentoo (virtual/os-headers) sys-libs/glibc: 2.35-r8::gentoo Repositories: gentoo location: /var/db/repos/gentoo sync-type: git sync-uri: https://github.com/gentoo-mirror/gentoo sync-user: portage:portage priority: -1000 sync-git-verify-commit-signature: yes local location: /var/db/repos/local masters: gentoo guru location: /var/lib/layman/guru sync-type: laymansync sync-uri: https://anongit.gentoo.org/git/repo/proj/guru.git masters: gentoo priority: 50 lto-overlay location: /var/lib/layman/lto-overlay sync-type: laymansync sync-uri: https://github.com/InBetweenNames/gentooLTO.git masters: gentoo mv priority: 50 matoro location: /var/lib/layman/matoro sync-type: laymansync sync-uri: https://github.com/matoro/overlay.git masters: gentoo priority: 50 mv location: /var/lib/layman/mv sync-type: laymansync sync-uri: https://anongit.gentoo.org/git/user/mv.git masters: gentoo priority: 50 ACCEPT_KEYWORDS="arm64 ~arm64" ACCEPT_LICENSE="@FREE" CBUILD="aarch64-unknown-linux-gnu" CFLAGS="-O3 -march=native -mtune=native -O3 -fgraphite-identity -floop-nest-optimize -fdevirtualize-at-ltrans -fipa-pta -fno-semantic-interposition -flto=16 -fuse-linker-plugin -pipe" CHOST="aarch64-unknown-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O3 -march=native -mtune=native -O3 -fgraphite-identity -floop-nest-optimize -fdevirtualize-at-ltrans -fipa-pta -fno-semantic-interposition -flto=16 -fuse-linker-plugin -pipe" DISTDIR="/var/cache/distfiles" EMERGE_DEFAULT_OPTS="--usepkg --autounmask=n --complete-graph --with-bdeps=y --keep-going" ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR" FCFLAGS="-O2" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg buildpkg-live compress-build-logs compressdebug config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-install pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2" GENTOO_MIRRORS="https://gentoo.osuosl.org/ https://mirror.leaseweb.com/gentoo/ https://mirror.rackspace.com/gentoo/" LANG="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j10" PKGDIR="/var/cache/binpkgs" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" SHELL="/bin/bash" USE="acl arm64 bash-completion bzip2 caps cli crypt dist-kernel dri elogind fortran gdbm gentoo-vm graphite headless-awt iconv ipv6 libbsd libglvnd libtirpc lm-sensors lto ncurses nls nptl openmp pam pcre pgo readline seccomp split-usr ssl system-bootstrap system-llvm threads udev unicode verify-sig vim-syntax xattr zlib" ADA_TARGET="gnat_2020" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_ARM="aes crc32 edsp sha1 sha2 thumb thumb2 v4 v5 v6 v7 v8 vfp vfp-d32 vfpv3 vfpv4" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64" INPUT_DEVICES="libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-4 php8-0" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_10" PYTHON_TARGETS="python3_10" RUBY_TARGETS="ruby27" USERLAND="GNU" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LEX, LFLAGS, LIBTOOL, LINGUAS, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS
Created attachment 802747 [details] example build log showing link failure
Asked matoro to try GCC 11 w/ USE=vanilla and possibly from source. Debian's being okay is alarming! It looks like they're not shipping 11 anymore, but we can see their patches at https://salsa.debian.org/toolchain-team/gcc/-/tree/gcc-11-debian/debian/patches.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=956418 has >[...] > * debian/control.in/main: build-depends on gcc-10 (>= 10-20200431) on arm64 > to ensure that -moutline-atomics is enabled by default. Closes: #956418. I can't see which commit does that though.
This whole thread is interesting: https://www.mail-archive.com/gcc-patches@gcc.gnu.org/msg235020.html.
USE=vanilla does NOT fix the problem. But, =sys-devel/gcc-10.4.0 DOES have the correct functions.
*** Bug 883219 has been marked as a duplicate of this bug. ***
(In reply to matoro from comment #5) > USE=vanilla does NOT fix the problem. But, =sys-devel/gcc-10.4.0 DOES have > the correct functions. This makes it pretty hard to deduce what we're doing that's special. We don't really do much interesting there? I assume a manual build is good? For testing purposes, disabling bootstrapping with EXTRA_ECONF="--disable-bootstrap" may help.
(In reply to Sam James from comment #7) > (In reply to matoro from comment #5) > > USE=vanilla does NOT fix the problem. But, =sys-devel/gcc-10.4.0 DOES have > > the correct functions. > > This makes it pretty hard to deduce what we're doing that's special. We > don't really do much interesting there? > > I assume a manual build is good? For testing purposes, disabling > bootstrapping with EXTRA_ECONF="--disable-bootstrap" may help. Interestingly, just tried a manual build (checkout tag releases/gcc-11.3.0, ./configure, make) and it is NOT good: $ nm ./aarch64-unknown-linux-gnu/libgcc/libgcc.a 2>/dev/null | grep "_sync" U __aarch64_sync_cache_range 0000000000000000 T __aarch64_sync_cache_range Have no idea what this means or how Debian is getting it to work.
(In reply to matoro from comment #8) > (In reply to Sam James from comment #7) > > (In reply to matoro from comment #5) > > > USE=vanilla does NOT fix the problem. But, =sys-devel/gcc-10.4.0 DOES have > > > the correct functions. > > > > This makes it pretty hard to deduce what we're doing that's special. We > > don't really do much interesting there? > > > > I assume a manual build is good? For testing purposes, disabling > > bootstrapping with EXTRA_ECONF="--disable-bootstrap" may help. > > Interestingly, just tried a manual build (checkout tag releases/gcc-11.3.0, > ./configure, make) and it is NOT good: > > $ nm ./aarch64-unknown-linux-gnu/libgcc/libgcc.a 2>/dev/null | grep "_sync" > U __aarch64_sync_cache_range > 0000000000000000 T __aarch64_sync_cache_range > > Have no idea what this means or how Debian is getting it to work. Very interesting. Possibly related, probably not, but: https://trofi.github.io/posts/275-nixpkgs-bootstrap-deep-dive.html.
> To check whether or not this was a Gentoo-specific issue, I downloaded > Debian's gcc-11 release and found that it DOES have the correct symbols: > > $ nm libgcc-11-dev_11.3.0-5_arm64/usr/lib/gcc/aarch64-linux-gnu/11/libgcc.a > 2>/dev/null | grep "_sync" > 0000000000000000 T __aarch64_cas1_sync > 0000000000000000 T __aarch64_cas2_sync > 0000000000000000 T __aarch64_cas4_sync > 0000000000000000 T __aarch64_cas8_sync > 0000000000000000 T __aarch64_cas16_sync > 0000000000000000 T __aarch64_swp1_sync > 0000000000000000 T __aarch64_ldadd1_sync > 0000000000000000 T __aarch64_ldclr1_sync > 0000000000000000 T __aarch64_ldeor1_sync > 0000000000000000 T __aarch64_ldset1_sync > 0000000000000000 T __aarch64_swp2_sync > 0000000000000000 T __aarch64_ldadd2_sync > 0000000000000000 T __aarch64_ldclr2_sync > 0000000000000000 T __aarch64_ldeor2_sync > 0000000000000000 T __aarch64_ldset2_sync > 0000000000000000 T __aarch64_swp4_sync > 0000000000000000 T __aarch64_ldadd4_sync > 0000000000000000 T __aarch64_ldclr4_sync > 0000000000000000 T __aarch64_ldeor4_sync > 0000000000000000 T __aarch64_ldset4_sync > 0000000000000000 T __aarch64_swp8_sync > 0000000000000000 T __aarch64_ldadd8_sync > 0000000000000000 T __aarch64_ldclr8_sync > 0000000000000000 T __aarch64_ldeor8_sync > 0000000000000000 T __aarch64_ldset8_sync > U __aarch64_sync_cache_range > 0000000000000000 T __aarch64_sync_cache_range > > This shows that this is an issue specific to the Gentoo toolchain. not really :P.. I'll build an aarch64 toolchain tomorrow to see what happens. out of interest, can you build the latest gcc11 snapshot off of gcc.gnu.org? see what it contains (either install to an alt prefix or don't install at all)
oh, my bad, sam did do that, and it's indeed also wrong in that case. apologies
