Bug 454200 – sys-libs/glibc-2.17 breaks on arm hardfp systems w/mixed tagged libs - error while loading shared libraries: libgcc_s.so.1: cannot open shared object file: No such file or directory

• Status: RESOLVED FIXED
• Product: Gentoo Linux

[Description William Throwe 2013-01-26 18:09:24 UTC]

Created attachment 336934 [details]
build.log.gz

Looks like fixing bug #453760 was a bad idea.

As soon as the merge phase completed, every dynamically linked binary on the system started failing with "error while loading shared libraries: libgcc_s.so.1: cannot open shared object file: No such file or directory"

After using sln to link libgcc_s.so.1 into /lib, most binaries work again, although things still can't find libstdc++.

$ emerge --info
Portage 2.1.11.31 (default/linux/arm/10.0, gcc-4.5.3, glibc-2.17, 2.6.31.14.22-efikamx armv7l)
=================================================================
System uname: Linux-2.6.31.14.22-efikamx-armv7l-ARMv7_Processor_rev_5_-v7l-with-gentoo-2.1
Timestamp of tree: Sat, 26 Jan 2013 07:45:01 +0000
ld GNU ld (GNU Binutils) 2.22
distcc 3.1 armv7a-hardfloat-linux-gnueabi [enabled]
app-shells/bash:          4.2_p37
dev-lang/python:          2.7.3-r2, 3.2.3
dev-util/cmake:           2.8.7-r5
dev-util/pkgconfig:       0.27.1
sys-apps/baselayout:      2.1-r1
sys-apps/openrc:          0.9.8.4
sys-apps/sandbox:         2.5
sys-devel/autoconf:       2.13, 2.68
sys-devel/automake:       1.11.6
sys-devel/binutils:       2.22-r1
sys-devel/gcc:            4.5.3-r2
sys-devel/gcc-config:     1.7.3
sys-devel/libtool:        2.4-r1
sys-devel/make:           3.82-r3
sys-kernel/linux-headers: 3.4-r2 (virtual/os-headers)
sys-libs/glibc:           2.17
Repositories: gentoo
ACCEPT_KEYWORDS="arm"
ACCEPT_LICENSE="@FREE"
CBUILD="armv7a-hardfloat-linux-gnueabi"
CFLAGS="-O2 -pipe -mcpu=cortex-a8 -mfpu=vfpv3 -mfloat-abi=hard"
CHOST="armv7a-hardfloat-linux-gnueabi"
CONFIG_PROTECT="/etc"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-O2 -pipe -mcpu=cortex-a8 -mfpu=vfpv3 -mfloat-abi=hard"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--ask"
FCFLAGS="-O2"
FEATURES="assume-digests binpkg-logs clean-logs compress-build-logs config-protect-if-modified distcc ebuild-locks fixlafiles merge-sync news protect-owned sandbox sfperms skiprocheck strict unknown-features-warn unmerge-orphans userfetch"
FFLAGS="-O2"
GENTOO_MIRRORS="http://gentoo.netnitco.net http://mirror.mcs.anl.gov/pub/gentoo/ http://mirror.csclub.uwaterloo.ca/gentoo-distfiles/ http://mirror.datapipe.net/gentoo ftp://mirror.datapipe.net/gentoo"
LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu -Wl,--no-keep-memory,--reduce-memory-overheads"
MAKEOPTS="-j5 -l2"
PKGDIR="/mnt/build/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY=""
SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage"
USE="X acl arm berkdb bindist bzip2 cairo cli cracklib crypt cxx emacs fortran gdbm gpm gtk iconv jpeg latex modules mudflap ncurses nptl openmp pam pcre png readline session ssl system-sqlite tcpd truetype unicode zlib" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" PHP_TARGETS="php5-3" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_2" RUBY_TARGETS="ruby18 ruby19" USERLAND="GNU" VIDEO_CARDS="exynos fbdev omap omapfb dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, INSTALL_MASK, LANG, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON

[Comment 1 SpanKY 2013-01-27 16:35:08 UTC]

what does `gcc-config -l` show ?  is your installed toolchain set to active ?

what does /etc/ld.so.conf.d/05gcc-armv7a-hardfloat-linux-gnueabi.conf have ?

what about `ldconfig -p` ?

[Comment 2 William Throwe 2013-01-28 19:05:06 UTC]

Created attachment 337136 [details]
ldconfig output

Looks fine as far as I can tell.

mim ~ # gcc-config -l
 [1] armv7a-hardfloat-linux-gnueabi-4.5.3 *
mim ~ # cat /etc/ld.so.conf.d/05gcc-armv7a-hardfloat-linux-gnueabi.conf 
/usr/lib/gcc/armv7a-hardfloat-linux-gnueabi/4.5.3
mim ~ # ldconfig -p >ldconfig
mim ~ # grep libgcc_s ldconfig 
        libgcc_s.so.1 (libc6) => /usr/lib/gcc/armv7a-hardfloat-linux-gnueabi/4.5.3/libgcc_s.so.1

[Comment 3 SpanKY 2013-01-28 20:20:11 UTC]

so if you delete /lib/libgcc_s.so.1 now, everything runs fine ?

[Comment 4 William Throwe 2013-01-28 20:29:37 UTC]

(In reply to comment #3)
> so if you delete /lib/libgcc_s.so.1 now, everything runs fine ?

Nope.  Everything breaks again if I delete the symlink.

I don't know if it is interesting or not, but the ldconfig -p output is the same either with or without the /lib/libgcc_s.so.1 symlink.

[Comment 5 SpanKY 2013-01-28 20:43:21 UTC]

(In reply to comment #4)

if you delete it, then run `ldconfig`, does things work ?  glibc should be able to locate libgcc_s.so.1 in gcc's internal dir just fine ...

i'm assuming you're just running `ls` and such and not trying to reboot with a sep /usr partition

[Comment 6 William Throwe 2013-01-28 21:01:35 UTC]

(In reply to comment #5)
> (In reply to comment #4)
> 
> if you delete it, then run `ldconfig`, does things work ?  glibc should be
> able to locate libgcc_s.so.1 in gcc's internal dir just fine ...
> 

Removing the symlink and then running ldconfig does not help.  Still can't find libgcc_s.so.1 .

> i'm assuming you're just running `ls` and such and not trying to reboot with
> a sep /usr partition

Yeah, just running ls and similar.  I haven't tried to reboot this machine yet since installing glibc-2.17, and it doesn't have a separate /usr anyway.

[Comment 7 SpanKY 2013-01-28 21:27:04 UTC]

(In reply to comment #6)

hrmph.  this really really should not be the case.

if you run `ldd /bin/ls`, does it show the /lib/ one ?  and if you rm it, it shows not found ?

can you run `LD_DEBUG=all ls` after deleting the /lib/ one (and running `ldconfig -p`) and post the output as an attachment ?

[Comment 8 William Throwe 2013-01-28 21:53:39 UTC]

Created attachment 337154 [details]
LD_DEBUG=all ls

(In reply to comment #7)
> (In reply to comment #6)
> 
> hrmph.  this really really should not be the case.
> 
> if you run `ldd /bin/ls`, does it show the /lib/ one ?  and if you rm it, it
> shows not found ?
> 
> can you run `LD_DEBUG=all ls` after deleting the /lib/ one (and running
> `ldconfig -p`) and post the output as an attachment ?

Correct, except that ldd doesn't work without the symlink so I can't do that test.

mim ~ # ldd /bin/ls
        librt.so.1 => /lib/librt.so.1 (0x2aadc000)
        libacl.so.1 => /lib/libacl.so.1 (0x2aaeb000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2aafa000)
        libc.so.6 => /lib/libc.so.6 (0x2ab0d000)
        libpthread.so.0 => /lib/libpthread.so.0 (0x2ac45000)
        libattr.so.1 => /lib/libattr.so.1 (0x2ac65000)
        /lib/ld-linux.so.3 (0x2aaab000)
mim ~ # rm /lib/libgcc_s.so.1 
mim ~ # ldd /bin/ls
/bin/bash: error while loading shared libraries: libgcc_s.so.1: cannot open shared object file: No such file or directory
mim ~ # ldconfig -p>/dev/null
mim ~ # LD_DEBUG=all ls 2>ld_debug

[Comment 9 SpanKY 2013-01-28 22:55:31 UTC]

(In reply to comment #8)

i wonder if libgcc_s.so.1 or ls was built with some ELF tags that the runtime ldso uses to reject the cache entries, and then falls back to a path search and says "screw it i'll use this".

earlier passes in that file indicate a bug too.  for example, it should have found librt.so.1 via the cache, not via the fallback path search.

could you run `readelf -Ae` on /bin/ls and /lib/librt.so.1 and /lib/libgcc.s.so.1 and post the output as an attachment ?

also, run `ldconfig -pvvv` and post that as an attachment.  that way it might include tag information too.

[Comment 10 William Throwe 2013-01-28 23:32:55 UTC]

Created attachment 337160 [details]
readelf output

readelf -Ae for the requested files attached.

`ldconfig -pvvv` gave the same output as `ldconfig -p`.  The only difference from the previously attached ldconfig output is that it now lists ld-linux-armhf.so.3 instead of ld-linux.so.3 on the last line.  I don't know why that changed.  (The latter is a symlink to the former.)

[Comment 11 SpanKY 2013-01-29 00:12:59 UTC]

(In reply to comment #10)

hrm, i'm guessing this is an older system that transitioned from the old arm ldso name to the new one.

if you compile a dummy app like so:
 echo 'main(){}' | gcc -x c - -lgcc_s -o a.out

and then run `./a.out`, can it locate libgcc_s.so.1 in the internal gcc dir (i.e. rm it from /lib/ and try running it) ?

if so, can you re-emerge coreutils and see if `ls` starts working ?

if not, does `ldd` show a.out using ld-linux.so.3 or ld-linux-armhf.so.3 (the latter is the correct one) ?  you might have to upgrade/rebuild your gcc to one that uses the new ldso name -- gcc-4.5+ in the tree should be patched now to use the new ldso name.

[Comment 12 William Throwe 2013-01-29 00:49:04 UTC]

(In reply to comment #11)
> (In reply to comment #10)
> 
> hrm, i'm guessing this is an older system that transitioned from the old arm
> ldso name to the new one.
> 

In fact, it mostly hasn't transitioned because I hit bug 434856 at the same time as the transition and everything has been broken since.

> if you compile a dummy app like so:
>  echo 'main(){}' | gcc -x c - -lgcc_s -o a.out
> 
> and then run `./a.out`, can it locate libgcc_s.so.1 in the internal gcc dir
> (i.e. rm it from /lib/ and try running it) ?
> 

No.  It fails.

> if so, can you re-emerge coreutils and see if `ls` starts working ?
> 
> if not, does `ldd` show a.out using ld-linux.so.3 or ld-linux-armhf.so.3
> (the latter is the correct one) ?  you might have to upgrade/rebuild your
> gcc to one that uses the new ldso name -- gcc-4.5+ in the tree should be
> patched now to use the new ldso name.

It shows ld-linux.so.3 .  I will try updating to gcc-4.6.3.

[Comment 13 William Throwe 2013-01-29 18:29:54 UTC]

No better with 4.6.3, even though it uses the correct ld.so name:

mim ~ # echo 'main(){}' | gcc-4.5.3 -x c - -lgcc_s -o 4.5.3.out
mim ~ # echo 'main(){}' | gcc-4.6.3 -x c - -lgcc_s -o 4.6.3.out
mim ~ # rm /lib/libgcc_s.so.1 
mim ~ # ./4.5.3.out 
./4.5.3.out: error while loading shared libraries: libgcc_s.so.1: cannot open shared object file: No such file or directory
mim ~ # ./4.6.3.out 
./4.6.3.out: error while loading shared libraries: libgcc_s.so.1: cannot open shared object file: No such file or directory
mim ~ # sln /usr/lib/gcc/armv7a-hardfloat-linux-gnueabi/4.5.3/libgcc_s.so.1 /lib/libgcc_s.so.1
mim ~ # ldd 4.5.3.out 
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2aadd000)
        libc.so.6 => /lib/libc.so.6 (0x2aaf0000)
        /lib/ld-linux.so.3 (0x2aaab000)
mim ~ # ldd 4.6.3.out 
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2aadd000)
        libc.so.6 => /lib/libc.so.6 (0x2aaf0000)
        /lib/ld-linux-armhf.so.3 (0x2aaab000)

Also, I can't change the system compiler to 4.6.3 because gcc-config removes the symlink and crashes before completing the update.  (The install of 4.6.3 also removed the symlink and failed after the merge, I think because it ran gcc-config.)

[Comment 14 William Throwe 2013-02-02 19:28:51 UTC]

I recompiled with USE=vanilla, no change.

However, I found this upstream bug: http://sourceware.org/bugzilla/show_bug.cgi?id=15006

If I'm reading it correctly, glibc is assuming everything on the system has been built with a binutils feature that is so new that it hasn't been released yet.

[Comment 15 SpanKY 2013-02-02 23:24:48 UTC]

yes, that looks like the same issue

[Comment 16 SpanKY 2013-04-26 15:48:24 UTC]

Created attachment 346650 [details, diff]
upstream patch

can you try this patch and see if it fixes things ?  you can put it in:
  /etc/portage/patches/sys-libs/glibc/
and it should get automatically applied

[Comment 17 Matt Whitlock 2013-04-26 22:24:23 UTC]

(In reply to comment #16)
> Created attachment 346650 [details, diff] [details, diff]
> upstream patch
> 
> can you try this patch and see if it fixes things ?  you can put it in:
>   /etc/portage/patches/sys-libs/glibc/
> and it should get automatically applied

Confirmed. This fixes it. After deleting my symlinks in /usr/lib and running `LD_LIBRARY_PATH=/usr/lib/gcc/armv6j-hardfloat-linux-gnueabi/4.7.2 ldconfig`, my system is working as it should.

[Comment 18 SpanKY 2013-04-28 03:59:11 UTC]

(In reply to comment #17)

great ... thanks for testing

http://sources.gentoo.org/gentoo/src/patchsets/glibc/2.17/0090_all_glibc-2.17-arm-ldso.cache.patch?rev=1.1

[Comment 19 Philipp Riegger 2013-04-29 09:29:26 UTC]

I was hit by that bug, too. What's the easiest way to fix this? Insert the SD card into some other machine and unpack a binpkg (built on the Raspberry Pi, of course) from an older glibc version that worked?

[Comment 20 Matt Whitlock 2013-04-29 18:17:53 UTC]

(In reply to comment #19)
> I was hit by that bug, too. What's the easiest way to fix this? Insert the
> SD card into some other machine and unpack a binpkg (built on the Raspberry
> Pi, of course) from an older glibc version that worked?

You only need to make a few symlinks:

busybox ln -s /usr/lib/gcc/armv6j-hardfloat-linux-gnueabi/*/lib*.so* /usr/lib/

Then re-emerge glibc-2.17.

If you've shut down your system and can't get a prompt on it now, you'll have to put the SD card in another Linux system and make those symlinks there. Then you should be able to boot into it on the RPi and re-emerge glibc.

After the glibc build has completed, you can remove the symlinks:

cd /usr/lib
rm $(cd /usr/lib/gcc/armv6j-hardfloat-linux-gnueabi/* ; command ls lib*.so*)

Then you should re-run ldconfig to fix your cache:

LD_LIBRARY_PATH=$(echo /usr/lib/gcc/armv6j-hardfloat-linux-gnueabi/*) ldconfig

[Comment 21 gentooer 2013-04-30 09:39:10 UTC]

Will be fixed in the new 2.17 point release.

http://sourceware.org/bugzilla/show_bug.cgi?id=15122

[Comment 22 Philipp Riegger 2013-05-07 14:39:38 UTC]

This did not work for me. I did the following:

- created the symlinks. System works normally again.
- re-emerged glibc (*)
- removed the symlinks. I get the error about libgcc_s.so.1 again

(*) I re-emerged glibc multiple times, updated gcc from 4.6 to 4.7, removed and re-added the symlinks several times... nothing worked. The installed ebuild is

$Header: /var/cvsroot/gentoo-x86/sys-libs/glibc/glibc-2.17.ebuild,v 1.13 2013/04/28 04:00:36 vapier Exp $

so that should be the latest one with vapier's fix, correct?

[Comment 23 Matt Whitlock 2013-05-07 14:55:34 UTC]

(In reply to comment #22)
> This did not work for me. I did the following:
> 
> - created the symlinks. System works normally again.
> - re-emerged glibc (*)
> - removed the symlinks. I get the error about libgcc_s.so.1 again
> 
> (*) I re-emerged glibc multiple times, updated gcc from 4.6 to 4.7, removed
> and re-added the symlinks several times... nothing worked. The installed
> ebuild is
> 
> $Header: /var/cvsroot/gentoo-x86/sys-libs/glibc/glibc-2.17.ebuild,v 1.13
> 2013/04/28 04:00:36 vapier Exp $
> 
> so that should be the latest one with vapier's fix, correct?

Did you rebuild your ld.so.cache after deleting the symlinks? Run ldconfig (with LD_LIBRARY_PATH set to the path containing libgcc_s.so).

[Comment 24 Philipp Riegger 2013-05-07 15:12:37 UTC]

Seems to work, thanks a lot.