When attempting to install glibc 2.19-r1 for stablization on my hardened ARM box, I get a failure during @system rebuild. Specifically, glib 2.38.2-r1 has the following in src_prepare: # Prevent build failure in stage3 where pkgconfig is not available, bug #481056 mv -f "${WORKDIR}"/pkg-config-*/pkg.m4 "${S}"/m4macros/ || die This causes a failure, due to the -* in the pkg-config path. Running the commands manually, you see the following output: xu work # mv -f pkg-config-*/pkg.m4 glib-2.38.2/m4macros/ mv: cannot stat ‘pkg-config-*/pkg.m4’: No such file or directory xu work # mv -f pkg-config-0.28/pkg.m4 glib-2.38.2/m4macros/ xu work # Reproducible: Always
xu breakdown # emerge --info Portage 2.2.8-r1 (hardened/linux/arm/armv7a, gcc-4.8.3, glibc-2.19-r1, 3.4.98 armv7l) ================================================================= System uname: Linux-3.4.98-armv7l-ARMv7_Processor_rev_3_-v7l-with-gentoo-2.2 KiB Mem: 1788124 total, 89052 free KiB Swap: 2097148 total, 2081560 free Timestamp of tree: Wed, 30 Jul 2014 17:00:01 +0000 ld GNU ld (Gentoo 2.23.2 p1.0) 2.23.2 distcc 3.1 armv7a-hardfloat-linux-gnueabi [disabled] app-shells/bash: 4.2_p45 dev-java/java-config: 2.2.0 dev-lang/python: 2.7.6, 3.3.3 dev-util/cmake: 2.8.12.2-r1 dev-util/pkgconfig: 0.28-r1 sys-apps/baselayout: 2.2 sys-apps/openrc: 0.12.4 sys-apps/sandbox: 2.6-r1 sys-devel/autoconf: 2.69 sys-devel/automake: 1.13.4 sys-devel/binutils: 2.23.2 sys-devel/gcc: 4.7.3-r1, 4.8.3 sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4.2-r1 sys-devel/make: 3.82-r4 sys-kernel/linux-headers: 3.13 (virtual/os-headers) sys-libs/glibc: 2.19-r1 Repositories: gentoo steev-meta ACCEPT_KEYWORDS="arm" ACCEPT_LICENSE="* -@EULA Oracle-BCLA-JavaSE" CBUILD="armv7a-hardfloat-linux-gnueabi" CFLAGS="-O2 -pipe -march=armv7-a -mtune=cortex-a15 -mfpu=neon -mfloat-abi=hard" CHOST="armv7a-hardfloat-linux-gnueabi" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -pipe -march=armv7-a -mtune=cortex-a15 -mfpu=neon -mfloat-abi=hard" DISTDIR="/usr/portage/distfiles" EMERGE_DEFAULT_OPTS="-j9 --load-average=9" FCFLAGS="-O2 -pipe -march=armv7-a" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe -march=armv7-a" GENTOO_MIRRORS="http://distfiles.gentoo.org" LANG="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j9 -l9" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/var/overlays/temp" USE="X acl arm auto-hinter berkdb bindist bzip2 cli cracklib crypt cxx dri gdbm gles1 gles2 hardened iconv ipv6 jpeg modules ncurses nls nptl openmp pam pax_kernel pcre pic png readline session ssl tcpd truetype unicode urandom xcb xlib-xcb xtpax zlib" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_3" RUBY_TARGETS="ruby19 ruby20" USERLAND="GNU" VIDEO_CARDS="fbdev" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, SYNC, USE_PYTHON
builds fine on my non-hardened armv7 w/gcc-4.7. going to guess one of those two factors is in play. i don't fully trust hardening on arm.
(In reply to SpanKY from comment #2) > builds fine on my non-hardened armv7 w/gcc-4.7. going to guess one of those > two factors is in play. i don't fully trust hardening on arm. That's fine and dandy. But some of us do, as we haven't had any issues with it prior to this stablization.
(In reply to Steev Klimaszewski from comment #3) not noticing bugs is not proof that they do not exist. i'm not sure hardened is a blocker at this point for stabilization on arm. i don't have a hardened setup myself for arm, so someone will have to debug it.
(In reply to SpanKY from comment #4) > (In reply to Steev Klimaszewski from comment #3) > > not noticing bugs is not proof that they do not exist. i'm not sure > hardened is a blocker at this point for stabilization on arm. > > i don't have a hardened setup myself for arm, so someone will have to debug > it. Most of the arm team seems to have hardened setups. If we provide you access, will you help debug? I can say my skills are no up to the task.
can't reproduce on hardened either - unpack stage3-armv7a_hardfp-hardened-20140627.tar.bz2 - install glibc-2.19-r1 - build glib-2.40.0-r1 - works
Aye, it would appear to be an issue with gcc 4.8.3, I tried on a gcc 4.7.3, and couldn't reproduce either.
Confirming this issue with hardened gcc-4.8.3 on armv6j: >>> Preparing source in /var/tmp/portage/dev-libs/glib-2.40.0-r1/work/glib-2.40.0 ... mv: cannot stat ‘/var/tmp/portage/dev-libs/glib-2.40.0-r1/work/pkg-config-*/pkg.m4’: No such file or directory However when I run command manually, it works fine: build ~ # mv -f /var/tmp/portage/dev-libs/glib-2.40.0-r1/work/pkg-config-*/pkg.m4 /var/tmp/portage/dev-libs/glib-2.40.0-r1/work/glib-2.40.0/m4macros/ build ~ # echo $? 0
Hmm.. This is likely a NFS-related issue in my case. I applied the following patch on glib ebuild: @@ -78,7 +78,11 @@ src_prepare() { # Prevent build failure in stage3 where pkgconfig is not available, bug #481056 - mv -f "${WORKDIR}"/pkg-config-*/pkg.m4 "${S}"/m4macros/ || die + rc=1 + while [[ $rc != 0 ]] ; do + mv -f "${WORKDIR}"/pkg-config-*/pkg.m4 "${S}"/m4macros/ + rc=$? + done # Fix gmodule issues on fbsd; bug #184301, upstream bug #107626 epatch "${FILESDIR}"/${PN}-2.12.12-fbsd.patch And it works after several retries: < snip > >>> Preparing source in /var/tmp/portage/dev-libs/glib-2.40.0-r1/work/glib-2.40.0 ... mv: cannot stat ‘/var/tmp/portage/dev-libs/glib-2.40.0-r1/work/pkg-config-*/pkg.m4’: No such file or directory mv: cannot stat ‘/var/tmp/portage/dev-libs/glib-2.40.0-r1/work/pkg-config-*/pkg.m4’: No such file or directory * Applying glib-2.12.12-fbsd.patch ... [ ok ] * Applying glib-2.40.0-external-gdbus-codegen.patch ... [ ok ] * Applying glib-2.36.4-znodelete.patch ... [ ok ] < snip > I never had this problem before with older glibc and gcc. The only problem I had is the following error after emerging of any package (inability to remove temp directory): >>> sys-apps/portage-2.2.8-r2 merged. rm: cannot remove ‘/var/tmp/portage/sys-apps/portage-2.2.8-r2/temp’: Directory not empty
(In reply to Alexander Tsoy from comment #9) > Hmm.. This is likely a NFS-related issue in my case. I retested this on a local flash storage with the same result. So NFS is not a culprit. Also confirming that there are no problems with non-hardened profile.
*** Bug 532986 has been marked as a duplicate of this bug. ***
(In reply to Steev Klimaszewski from comment #7) > Aye, it would appear to be an issue with gcc 4.8.3, I tried on a gcc 4.7.3, > and couldn't reproduce either. I'm reproducing this with armv7a gcc-4.8.3 hardened musl. Again 4.7.3 works.
*** Bug 542872 has been marked as a duplicate of this bug. ***
I can easily reproduce with a qemu-user chroot of stage3-armv7a_hardfp-hardened-20141211.tar.bz2. (gcc-4.8.3, bash-4.2_p53) A small test case is: mkdir p-aaaa touch p-aaaa/a bash -c 'echo ./p-*/a' prints './p-*/a' although the strace output shows that bash has found out that ./p-aaaa exists.
Some further debugging reveals: Of the compilers [1] armv7a-hardfloat-linux-gnueabi-4.8.3 [2] armv7a-hardfloat-linux-gnueabi-4.8.3-hardenednopie [3] armv7a-hardfloat-linux-gnueabi-4.8.3-hardenednopiessp [4] armv7a-hardfloat-linux-gnueabi-4.8.3-hardenednossp [5] armv7a-hardfloat-linux-gnueabi-4.8.3-vanilla only 1 & 2 lead to a bad bash executable. Changing for the compilation of bash.../lib/glob/glob.c between the compilers makes a difference. So lib/glob/glob.c might be miscompiled.
(In reply to Felix Janda from comment #15) > Some further debugging reveals: > > Of the compilers > > [1] armv7a-hardfloat-linux-gnueabi-4.8.3 > [2] armv7a-hardfloat-linux-gnueabi-4.8.3-hardenednopie > [3] armv7a-hardfloat-linux-gnueabi-4.8.3-hardenednopiessp > [4] armv7a-hardfloat-linux-gnueabi-4.8.3-hardenednossp > [5] armv7a-hardfloat-linux-gnueabi-4.8.3-vanilla > > only 1 & 2 lead to a bad bash executable. > > Changing for the compilation of bash.../lib/glob/glob.c between the > compilers makes a difference. So lib/glob/glob.c might be miscompiled. can you pass -fstack-check=no to cflags to see if that fix it on 1?
Created attachment 402248 [details, diff] Patch for bash-4.2 Seems to fix the problem. Please test. bash's glob_vector() has something like 1: char *p; 2: if (whatever) { 3: int i; 4: p = alloca(10); 5: } Is this valid usage of alloca? Naively I would expect that at line 3, (on arm) 4 bytes are allocated on the stack, on the next line another 10 and finally 4 bytes are freed because i is out of scope. However the wrong 4 bytes are freed...
can you pass -fstack-check=no to cflags to see if that fix it on 1? Yes, then bash + gcc fixing, no?
(In reply to Felix Janda from comment #17) > Created attachment 402248 [details, diff] [details, diff] > Patch for bash-4.2 > > Seems to fix the problem. Please test. > > bash's glob_vector() has something like > > 1: char *p; > 2: if (whatever) { > 3: int i; > 4: p = alloca(10); > 5: } > > Is this valid usage of alloca? alloca() is supposed to allocate in the stack frame of the function and then cleaned up when the stack frame is destroyed on return. That's what supposed to happen but ... > > Naively I would expect that at line 3, (on arm) 4 bytes are allocated > on the stack, on the next line another 10 and finally 4 bytes are > freed because i is out of scope. However the wrong 4 bytes are freed... This is a good point. I don't know and why I've never liked alloca. So, why not replace that with malloc() and then just before return free(p) and see if that fixes the issue.
(In reply to Felix Janda from comment #17) that code looks valid to me. (In reply to Anthony Basile from comment #19) the man page says alloca frees on function return, not on local scope. to be clear, malloc/free is a useful test, but it is not the right fix.
(In reply to Magnus Granberg from comment #16) building just glob.c w/-fstack-check=no produces a working bash more specifically, applying __attribute__((optimize("-fstack-check=no"))) to the glob_vector function in glob.c alone also produces a working bash
Replacing alloca by malloc also works. (Though in order to call free() properly the code would need to be analyzed carefully. In my test I just replaced alloca by malloc and did not care about the memleak.) I have a minimal test case: #include <stdlib.h> #include <stdio.h> #include <string.h> #include <sys/stat.h> int main(int argc, char **argp) { char *p; if (1) { struct stat i; p = alloca(8); strcpy(p, "1234567"); printf("&i: %p p: %p\n", &i, p); } if (1) { struct stat i, i2; printf("&i: %p &i2: %p\n", &i, &i2); } printf("p: %s\n", p); return 0; } Compile with 'gcc -O0'. The bad compiler lets i2 overlap with p and garbage gets written to p. struct stat serves here as a ready to use big struct.
Why not fix gcc and the assembly generated? For the long term I wish keep -fstack-check=yes
(In reply to Felix Janda from comment #22) ah, great. should be able to iterate w/this. i also tried setting --param ssp-buffer-size=8 (since our 4.8 patches changes that to 4), but it didn't help (In reply to BRULE Herman from comment #23) please read the bug -- that's what we're talking about. it's why the bug is assigned to toolchain@.
I've just checked that vanilla arm gcc-4.9.2 is also bad when being given the -fstack-check option. The git HEAD is likely also bad. Could maybe someone with a strong build machine bisect this? A program which returns 1 when compiled with the bad compiler and 0 else: #include <string.h> #include <stdlib.h> #include <sys/stat.h> int main(void) { char *p; if(1) { struct stat i; p = alloca(8); strcpy(p, "123456\n"); } if(1) { struct stat i, j; memset(&j, 0, sizeof j); } return (p[0] == 0); } An arm machine is not required. A cross-compiler and qemu-arm suffices.
(In reply to Felix Janda from comment #25) > I've just checked that vanilla arm gcc-4.9.2 is also bad when being > given the -fstack-check option. The git HEAD is likely also bad. > > Could maybe someone with a strong build machine bisect this? > A program which returns 1 when compiled with the bad compiler and > 0 else: Gentoo added --stack-check to the hardened spec file in 4.8, but gcc has supported it for a lot longer. So we'd have to see if worked in versions even earlier than 4.7. It might have always broken alloca() on arm.
(In reply to Anthony Basile from comment #26) > (In reply to Felix Janda from comment #25) > > I've just checked that vanilla arm gcc-4.9.2 is also bad when being > > given the -fstack-check option. The git HEAD is likely also bad. > > > > Could maybe someone with a strong build machine bisect this? > > A program which returns 1 when compiled with the bad compiler and > > 0 else: > > Gentoo added --stack-check to the hardened spec file in 4.8, but gcc has > supported it for a lot longer. So we'd have to see if worked in versions > even earlier than 4.7. It might have always broken alloca() on arm. I see. So let's just check git HEAD and then report upstream?
Reported upstream as: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65958
*** Bug 547566 has been marked as a duplicate of this bug. ***
i guess we'll need to update the ssp default patches to disable -fstack-check by default, and add a check to throw an error if the user tries to utilize the flag at all ...
(In reply to SpanKY from comment #30) > i guess we'll need to update the ssp default patches to disable > -fstack-check by default, and add a check to throw an error if the user > tries to utilize the flag at all ... on the affected arches, or all arches?
(In reply to Anthony Basile from comment #31) just the affected arches of course. assuming we can easily determine that info. i haven't gone through the code yet upstream alluded to to see if that's feasible.
I've added the following mask to the hardened arm profiles: # Anthony G. Basile <blueness@gentoo.org> (08 May 2015) # Mask gcc 4.8 and above pending the fix of bug #518598 =sys-devel/gcc-4.8* =sys-devel/gcc-4.9* =sys-devel/gcc-5.1* Specifically this will affect the following as listed by `eselect profile list` [33] hardened/linux/arm/armv7a * [34] hardened/linux/arm/armv6j [35] hardened/linux/musl/arm/armv7a [37] hardened/linux/uclibc/arm/armv7a I will be rebuilding stage3-armv7a_hardfp-hardened-20141211.tar.bz2 currently in <mirror>/experimental/arm/hardened/ using gcc-4.7.4.
But only gcc 4.8+ on ARM fix other problem and performance issues...
(In reply to BRULE Herman from comment #34) > But only gcc 4.8+ on ARM fix other problem and performance issues... alloca() is used in many places. hardened + arm + >=gcc-4.8 is broken. If you want performance + brokenness, unmask on your local system.
I yanked the older stage3-armv7a_hardfp-hardened-20141211.tar.bz2 built using gcc-4.8.3 from the mirrors and added stage3-armv7a_hardfp-hardened-20150509.tar.bz2 built with gcc-4.7.4. I also had to use glibc-2.19-r1 because 2.20 and above requires 4.8. This dependancy should probably be in the glibc ebuild. (I'll open a separate bug.)
(In reply to Anthony Basile from comment #36) > I also had to use glibc-2.19-r1 because 2.20 and > above requires 4.8. This dependancy should probably be in the glibc ebuild. > (I'll open a separate bug.) I triggered bug #547420. I'll keep these stages at gcc-4.7.4 and glibc-2.19-r1 until this bug is fixed.
upstream has posted a patch, but it looks stalled in the meantime, i've disabled -fstack-check enabling by default in the arm/esp: http://sources.gentoo.org/gentoo/src/patchsets/gcc/4.8.5/pie/35_all_gcc48_config_arm.patch?r1=1.1&r2=1.2 http://sources.gentoo.org/gentoo/src/patchsets/gcc/4.8.5/pie/40_all_gcc48_config_esp.patch?r1=1.1&r2=1.2 http://sources.gentoo.org/gentoo/src/patchsets/gcc/4.9.3/pie/35_all_gcc48_config_arm.patch?r1=1.1&r2=1.2 http://sources.gentoo.org/gentoo/src/patchsets/gcc/4.9.3/pie/40_all_gcc49_config_esp.patch?r1=1.1&r2=1.2 http://sources.gentoo.org/gentoo/src/patchsets/gcc/5.1.0/pie/35_all_gcc51_config_arm.patch?r1=1.1&r2=1.2 http://sources.gentoo.org/gentoo/src/patchsets/gcc/5.1.0/pie/40_all_gcc49_config_esp.patch?r1=1.1&r2=1.2 http://sources.gentoo.org/gentoo/src/patchsets/gcc/5.2.0/pie/35_all_gcc51_config_arm.patch?r1=1.1&r2=1.2 http://sources.gentoo.org/gentoo/src/patchsets/gcc/5.2.0/pie/40_all_gcc49_config_esp.patch?r1=1.1&r2=1.2 pushed out in 4.8.5/4.9.3/5.1.0/5.2.0: http://gitweb.gentoo.org/repo/gentoo.git/commit/?id=66b03a0b75d103598b4cd2d37271d67a89a3a8be
(In reply to SpanKY from comment #38) > upstream has posted a patch, but it looks stalled > > pushed out in 4.8.5/4.9.3/5.1.0/5.2.0: > http://gitweb.gentoo.org/repo/gentoo.git/commit/ > ?id=66b03a0b75d103598b4cd2d37271d67a89a3a8be thanks mike. this allows me to lift the above masks. also, it avoids a related issue in uclibc for which I didn't open a bug but I mentioned on the uclibc list. A bad sed in libpthread/nptl/sysdeps/pthread/Makefile.in clips a couple of labels in the arm asm and causes breakage. This only happens with stack-check on. See http://lists.uclibc.org/pipermail/uclibc/2014-December/048738.html When the upstream patch gets pushed through, I'll revisit the issue there.
(In reply to Anthony Basile from comment #39) > > thanks mike. this allows me to lift the above masks. > all the arm stages which i push out through the mirrors under experimental (hardened glibc, uclibc and musl) have been updated to gcc-4.8.5.
this has been fixed in gcc-6, but i don't think they'll backport to gcc-5 since it's not a regression. we've disabled the flag in the hardened builds, so this in practice is handled for the most part. i've backported the common fix as that looks simple, but the arch-specific bits are non-trivial and don't apply cleanly.
i've included this in gcc-5.3 http://sources.gentoo.org/gentoo/src/patchsets/gcc/5.3.0/gentoo/77_all_gcc-5-pr65958.patch