As far as I can tell, dev-util/ninja-1.7.1 and 1.7.2 (not in tree yet) will fail to build on an x86_64 system using the hardened profile if the compiler is gcc-6.x. The build will fail early in ninja's bootstrapping stage, after it generates the build bootstrap 'ninja' binary and then runs it. I tested building ninja on a mips64 platform with gcc-6.2, and it builds fine there. On x86_64, I tried removing LTO and graphite support, but no-go. So this looks like something to do with hardened support and ninja. Error (after ./configure.py --bootstrap --verbose --debug in the build dir): ]# ./ninja -j1 -v [1/27] "./src/inline.sh" kBrowsePy < src/browse.py > build/browse_py.h FAILED: build/browse_py.h "./src/inline.sh" kBrowsePy < src/browse.py > build/browse_py.h ninja: build stopped: subcommand failed. This leaves behind this line in dmesg: [157501.096792] ninja[23950]: segfault at 39e7ba73e40 ip 0000039e7acbf4f4 sp 0000039e7ba73e40 error 6 in libc-2.24.so[39e7abe0000+196000] strace and gdb aren't of much help, even when bootstrapping ninja with '--debug': # gdb ./ninja GNU gdb (Gentoo 7.12 vanilla) 7.12 Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://bugs.gentoo.org/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from ./ninja...done. (gdb) set follow-fork-mode child (gdb) set args -j1 -v (gdb) run Starting program: /ramfs/ninja-1.7.2/ninja -j1 -v warning: Cannot call inferior functions, Linux kernel PaX protection forbids return to non-executable pages! [New process 23967] Thread 2.1 "ninja" received signal SIGSEGV, Segmentation fault. [Switching to process 23967] 0x0000034aa363a4f4 in __spawni_child () from /lib64/libc.so.6 (gdb) bt #0 0x0000034aa363a4f4 in __spawni_child () from /lib64/libc.so.6 #1 0x0000034aa364964f in clone () from /lib64/libc.so.6 (gdb) 24024 stat("src/subprocess-posix.cc", {st_mode=S_IFREG|0664, st_size=9278, ...}) = 0 24024 rt_sigprocmask(SIG_BLOCK, [HUP INT TERM], [], 8) = 0 24024 rt_sigaction(SIGINT, {sa_handler=0x49364c, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x34c53490d70}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 24024 rt_sigaction(SIGTERM, {sa_handler=0x49364c, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x34c53490d70}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 24024 rt_sigaction(SIGHUP, {sa_handler=0x49364c, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x34c53490d70}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 24024 stat("build", {st_mode=S_IFDIR|0755, st_size=780, ...}) = 0 24024 pipe([5, 6]) = 0 24024 fcntl(5, F_GETFD) = 0 24024 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 24024 getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=4*1024}) = 0 24024 getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=4*1024}) = 0 24024 getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=4*1024}) = 0 24024 getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=4*1024}) = 0 24024 getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=4*1024}) = 0 24024 getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=4*1024}) = 0 24024 pipe2([7, 8], O_CLOEXEC) = 0 24024 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x34c542d1000 24024 rt_sigprocmask(SIG_BLOCK, ~[], [HUP INT TERM], 8) = 0 24024 clone( <unfinished ...> 24025 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x34c542d0e40} --- 24024 <... clone resumed> child_stack=0x34c542d1ff0, flags=CLONE_VM|CLONE_VFORK|SIGCHLD) = 24025 24025 +++ killed by SIGSEGV +++ 24024 close(8) = 0
# emerge --info Portage 2.3.3 (python 2.7.12-final-0, hardened/linux/amd64, gcc-6.3.0, glibc-2.24, 4.8.14 x86_64) ================================================================= System uname: Linux-4.8.14-x86_64-Intel-R-_Core-TM-_i7_CPU_970_@_3.20GHz-with-gentoo-2.3 KiB Mem: 16351384 total, 10293896 free KiB Swap: 4209024 total, 4209024 free Timestamp of repository gentoo: Fri, 30 Dec 2016 15:45:01 +0000 sh bash 4.4_p5-r1 ld GNU ld (Gentoo 2.27 p1.0) 2.27 distcc 3.2rc1 x86_64-pc-linux-gnu [disabled] ccache version 3.3.3 [disabled] app-shells/bash: 4.4_p5-r1::gentoo dev-java/java-config: 2.2.0-r3::gentoo dev-lang/perl: 5.24.1_rc4::gentoo dev-lang/python: 2.7.12::gentoo, 3.4.5::gentoo, 3.5.2::gentoo dev-util/ccache: 3.3.3::gentoo dev-util/cmake: 3.7.1::gentoo dev-util/pkgconfig: 0.29.1::gentoo sys-apps/baselayout: 2.3::gentoo sys-apps/openrc: 0.23::gentoo sys-apps/sandbox: 2.11-r3::gentoo sys-devel/autoconf: 2.13::gentoo, 2.69-r2::gentoo sys-devel/automake: 1.11.6-r2::gentoo, 1.12.6-r1::gentoo, 1.13.4-r1::gentoo, 1.14.1-r1::gentoo, 1.15-r2::gentoo sys-devel/binutils: 2.26.1::gentoo, 2.27::gentoo sys-devel/gcc: 6.2.0-r1::gentoo, 6.3.0::gentoo sys-devel/gcc-config: 1.8-r1::gentoo sys-devel/libtool: 2.4.6-r2::gentoo sys-devel/make: 4.2.1::gentoo sys-kernel/linux-headers: 4.9::gentoo (virtual/os-headers) sys-libs/glibc: 2.24::gentoo Repositories: gentoo location: /usr/portage sync-type: rsync sync-uri: rsync://gentoo.cs.uni.edu/gentoo-portage priority: -1000 local location: /usr/portage/local masters: gentoo priority: 0 ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -march=corei7 -mtune=corei7 -mfpmath=sse -msse -msse2 -mssse3 -msse4.1 -msse4.2 -mcx16 -msahf -mcrc32 -maccumulate-outgoing-args -fforce-addr -fmodulo-sched -fivopts -ftree-loop-im -ftree-loop-linear -ftree-loop-ivcanon -fgcse-after-reload -fgcse-lm -fgcse-sm -fgcse-las -floop-interchange -ftree-loop-distribution -floop-strip-mine -floop-block -ftree-vectorize -flto=8 -fuse-linker-plugin" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/init.d /etc/pam.d /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-O2 -march=corei7 -mtune=corei7 -mfpmath=sse -msse -msse2 -mssse3 -msse4.1 -msse4.2 -mcx16 -msahf -mcrc32 -maccumulate-outgoing-args -fforce-addr -fmodulo-sched -fivopts -ftree-loop-im -ftree-loop-linear -ftree-loop-ivcanon -fgcse-after-reload -fgcse-lm -fgcse-sm -fgcse-las -floop-interchange -ftree-loop-distribution -floop-strip-mine -floop-block -ftree-vectorize -flto=8 -fuse-linker-plugin" DISTDIR="/usr/portage/distfiles" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs candy cgroup config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms sign splitdebug unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS=" http://gentoo.osuosl.org/ http://mirror.usu.edu/mirrors/gentoo/ http://mirror.csclub.uwaterloo.ca/gentoo-distfiles/ http://gentoo.arcticnetwork.ca/ " INSTALL_MASK="/usr/lib/systemd /etc/systemd" LANG="en_US.iso885915" LDFLAGS="-O2 -march=corei7 -mtune=corei7 -mfpmath=sse -msse -msse2 -mssse3 -msse4.1 -msse4.2 -mcx16 -msahf -mcrc32 -maccumulate-outgoing-args -fforce-addr -fmodulo-sched -fivopts -ftree-loop-im -ftree-loop-linear -ftree-loop-ivcanon -fgcse-after-reload -fgcse-lm -fgcse-sm -fgcse-las -floop-interchange -ftree-loop-distribution -floop-strip-mine -floop-block -ftree-vectorize -flto=8 -fuse-linker-plugin -Wl,-z,now -Wl,-z,relro" MAKEOPTS="-j14" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/ramfs" USE="amd64 apache2 berkdb bzip2 cli cracklib curl cxx dbus dri encode gd gdbm graphite hardened inotify ipv6 justify lto mmx mmxext modules multilib ncurses nptl nptlonly offensive openmp pam pax_kernel pcre pic pie readline real rtc seccomp session sqlite sse sse2 ssl ssp tcpd unicode urandom xattr xml xtpax zlib" ABI_X86="64 32" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_4 python3_5" QEMU_SOFTMMU_TARGETS="mips mips64 mipsel mips64el" RUBY_TARGETS="ruby21" USERLAND="GNU" VIDEO_CARDS="vesa vga fbdev" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
Have you tried using simpler CXXFLAGS?
(In reply to Mike Gilbert from comment #2) > Have you tried using simpler CXXFLAGS? The issue is reproducible outside of Portage: tar zfx /usr/portage/distfiles/ninja-1.7.2.tar.gz cd ninja-1.7.2/ ./configure.py --bootstrap --verbose --debug <boom after several seconds>
Ah. Well, I cannot reproduce this on a non-hardened profile.
could you try to build glibc with disable-bind-now? or use a older glibc 2.23? we have seen some segfault and build probs with 2.24 and pie
(In reply to Magnus Granberg from comment #5) > could you try to build glibc with disable-bind-now? > or use a older glibc 2.23? > we have seen some segfault and build probs with 2.24 and pie Is "disable-bind-now" a USE flag? quse doesn't indicate that exists. I'm in the process of rebuilding my userland after a forced hardware upgrade. Somehow, I incidentally fixed GDB to actually emit a more complete backtrace, which is revealing some additional information: # gdb ninja GNU gdb (Gentoo 7.12.1 vanilla) 7.12.1 Copyright (C) 2017 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://bugs.gentoo.org/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from ninja...done. (gdb) run Starting program: /usr/src/ninja-1.7.2/ninja warning: Cannot call inferior functions, Linux kernel PaX protection forbids return to non-executable pages! [0/27] INLINE build/browse_py.hninja: ../sysdeps/unix/sysv/linux/spawni.c:360: __spawnix: Assertion `ec >= 0' failed. Program received signal SIGABRT, Aborted. 0x000003d52ecaff0f in raise () from /lib64/libc.so.6 (gdb) bt #0 0x000003d52ecaff0f in raise () from /lib64/libc.so.6 #1 0x000003d52ecb140a in abort () from /lib64/libc.so.6 #2 0x000003d52eca8936 in __assert_fail_base () from /lib64/libc.so.6 #3 0x000003d52eca89f1 in __assert_fail () from /lib64/libc.so.6 #4 0x000003d52ed5b7ea in __spawnix () from /lib64/libc.so.6 #5 0x000003d52ed5bd5a in __spawni () from /lib64/libc.so.6 #6 0x000003d52ed5b49a in posix_spawn () from /lib64/libc.so.6 #7 0x0000002862990c95 in Subprocess::Start (this=0x2862df3e20, set=0x2862df3350, command="\"./src/inline.sh\" kBrowsePy < src/browse.py > build/browse_py.h") at ./src/subprocess-posix.cc:101 #8 0x00000028629914d9 in SubprocessSet::Add (this=0x2862df3350, command="\"./src/inline.sh\" kBrowsePy < src/browse.py > build/browse_py.h", use_console=false) at ./src/subprocess-posix.cc:210 #9 0x0000002862931d7a in RealCommandRunner::StartCommand (this=0x2862df3340, edge=0x2862dedf70) at ./src/build.cc:540 #10 0x0000002862932ebe in Builder::StartEdge (this=0x3ef385453f0, edge=0x2862dedf70, err=0x3ef385454e0) at ./src/build.cc:753 #11 0x00000028629328ed in Builder::Build (this=0x3ef385453f0, err=0x3ef385454e0) at ./src/build.cc:665 #12 0x0000002862917041 in (anonymous namespace)::NinjaMain::RunBuild (this=0x3ef38545630, argc=0, argv=0x3ef38545a50) at ./src/ninja.cc:951 #13 0x0000002862917c95 in (anonymous namespace)::real_main (argc=0, argv=0x3ef38545a50) at ./src/ninja.cc:1143 #14 0x0000002862917d7a in main (argc=1, argv=0x3ef38545a48) at ./src/ninja.cc:1172 (gdb) This is after manually compiling ninja per my notes in Comment #3. Still leaves segfault messages in dmesg, though: [22296.267615] ninja[18737]: segfault at 3a62b775e50 ip 000003a62a9b2804 sp 000003a62b775e50 error 6 in libc-2.25.so[3a62a8d3000+197000] [22331.247998] ninja[18743]: segfault at 358a9b60e50 ip 00000358a8dae804 sp 00000358a9b60e50 error 6 in libc-2.25.so[358a8ccf000+197000]
Looks like "ec" means "error code from child. Source patch that added this assert is: https://sourceware.org/ml/libc-alpha/2016-09/msg00360.html One of the follow-up messages suggests the more meaningful variable name that doesn't appear to have been taken into context. So it looks like on a hardened system, ninja is tripping up this assert in glibc. Not sure now if that means it's really a glibc bug, ninja bug, or hardened bug.
Can you rebuild glibc and put -fstack-check=no in the cflags?
(In reply to Joshua Kinard from comment #7) > Looks like "ec" means "error code from child. It would be nice to have more detailed info from the child process, which is failing early in __spawni_child().
(In reply to Magnus Granberg from comment #8) > Can you rebuild glibc and put -fstack-check=no in the cflags? I was able to reproduce the issue in a container with a current amd64-hardened stage3 tarball + gcc-6.3.0 + glibc-2.24-r1. Rebuilding glibc-2.24-r1 with -fstack-check=no seems to make the problem go away.
Thanks for the triage help. Sounds like this is a bug in glibc or gcc.
(In reply to Mike Gilbert from comment #10) > (In reply to Magnus Granberg from comment #8) > > Can you rebuild glibc and put -fstack-check=no in the cflags? > > I was able to reproduce the issue in a container with a current > amd64-hardened stage3 tarball + gcc-6.3.0 + glibc-2.24-r1. > > Rebuilding glibc-2.24-r1 with -fstack-check=no seems to make the problem go > away. Stack checking caused an issue on one of the kernels for my SGI machines by causing it to panic in hardcoded assembler for a spin lock. I had a funny feeling it might be tied to this, but didn't think to rebuild glibc with -fno-stack-check.
(In reply to Mike Gilbert from comment #10) > (In reply to Magnus Granberg from comment #8) > > Can you rebuild glibc and put -fstack-check=no in the cflags? > > I was able to reproduce the issue in a container with a current > amd64-hardened stage3 tarball + gcc-6.3.0 + glibc-2.24-r1. > > Rebuilding glibc-2.24-r1 with -fstack-check=no seems to make the problem go > away. I was about to state that I cannot reproduce this success, as I was using -fno-stack-check. but, on a whim, switched to -fstack-check=no and that works. So it looks like flags-o-matic needs a small update to handle the -fno version of this flag, as both -fno-stack-check and -fstack-check=no are equivalent.
*** This bug has been marked as a duplicate of bug 608788 ***
*** Bug 620944 has been marked as a duplicate of this bug. ***
