Bug report based on my thread http://forums.gentoo.org/viewtopic-p-7175950.html When I compile hardened kernel with CONFIG_PAX_RANDMMAP option turned on, system reboots while emerging glibc. Always in the same line. All (or I missed something) options to reboot system after panic/lock were disabled, but nothing shows up on remote terminal (via ttyS0) and still server reboots instantly. Just like after pressing reset button. I've also tested newest unstable hardened (3.6.4) - problem still exists. We can rule out hardware problem, machine was thoroughly tested. Also this doesn't happen without CONFIG_PAX_RANDMMAP or on gentoo-sources. There's no enough place in "additional information" to paste both emerge info and glibc log, so you can find second one in url above. Reboot happens always during last command. I've marked it "Critical" because it did crash and corrupted my filesystem a bit for the first time (next time I was prepared), but if you'd like please change it to "Major". Reproducible: Always Steps to Reproduce: 1. while on hardened-sources 2. emerge -1 glibc Actual Results: reboot Expected Results: no reboot ;) # emerge --info Portage 2.1.11.9 (hardened/linux/amd64, gcc-4.5.4, glibc-2.15-r3, 3.5.4-hardened-r1-tyrael-v1 x86_64) ================================================================= System uname: Linux-3.5.4-hardened-r1-tyrael-v1-x86_64-AMD_Opteron-TM-_Processor_6212-with-gentoo-2.1 Timestamp of tree: Sat, 03 Nov 2012 02:00:01 +0000 app-shells/bash: 4.2_p37 dev-lang/python: 3.2.3 dev-util/pkgconfig: 0.27.1 sys-apps/baselayout: 2.1-r1 sys-apps/openrc: 0.9.8.4 sys-apps/sandbox: 2.5 sys-devel/autoconf: 2.68 sys-devel/automake: 1.11.6 sys-devel/binutils: 2.22-r1 sys-devel/gcc: 4.5.4 sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4-r1 sys-devel/make: 3.82-r3 sys-kernel/linux-headers: 3.4-r2 (virtual/os-headers) sys-libs/glibc: 2.15-r3 Repositories: gentoo location: /usr/portage sync: rsync://rsync.gentoo.org/gentoo-portage priority: -1000 ABI="amd64" ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="*" ACCEPT_PROPERTIES="*" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias" ARCH="amd64" AUTOCLEAN="yes" BOOTSTRAP_USE="cxx unicode multilib hardened pax_kernel pic -jit -orc" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="ptp2" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=k8 -O2 -pipe" CFLAGS_amd64="-m64" CFLAGS_x32="-mx32" CFLAGS_x86="-m32" CHOST="x86_64-pc-linux-gnu" CHOST_amd64="x86_64-pc-linux-gnu" CHOST_x32="x86_64-pc-linux-gnux32" CHOST_x86="i686-pc-linux-gnu" CLEAN_DELAY="5" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" COLLISION_IGNORE="/lib/modules/* *.py[co]" CONFIG_PROTECT="/etc" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CURL_SSL="gnutls" CXXFLAGS="-O2 -pipe" DEFAULT_ABI="amd64" DISTDIR="/usr/portage/distfiles" EDITOR="/usr/bin/vim" ELIBC="glibc" EMERGE_DEFAULT_OPTS="--verbose --ask --tree --keep-going --usepkg --with-bdeps y --jobs 2" EMERGE_WARNING_DELAY="10" EPREFIX="" EROOT="/" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles news parallel-fetch parse-eapi-ebuild-head protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch" FETCHCOMMAND="wget -t 3 -T 60 --passive-ftp -O "${DISTDIR}/${FILE}" "${URI}"" FETCHCOMMAND_RSYNC="rsync -avP "${URI}" "${DISTDIR}/${FILE}"" FETCHCOMMAND_SFTP="bash -c "x=\${2#sftp://} ; host=\${x%%/*} ; port=\${host##*:} ; host=\${host%:*} ; [[ \${host} = \${port} ]] && port=22 ; exec sftp -P \${port} \"\${host}:/\${x#*/}\" \"\$1\"" sftp "${DISTDIR}/${FILE}" "${URI}"" FETCHCOMMAND_SSH="bash -c "x=\${2#ssh://} ; host=\${x%%/*} ; port=\${host##*:} ; host=\${host%:*} ; [[ \${host} = \${port} ]] && port=22 ; exec rsync --rsh=\"ssh -p\${port}\" -avP \"\${host}:/\${x#*/}\" \"\$1\"" rsync "${DISTDIR}/${FILE}" "${URI}"" FFLAGS="-O2 -pipe" GCC_SPECS="" GENTOO_MIRRORS="http://gentoo.po.opole.pl http://gentoo.prz.rzeszow.pl/ http://src.gentoo.pl/ http://dev.gentoo.org/~vapier/dist" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" GRUB_PLATFORMS="" HOME="/root" INFOPATH="/usr/share/info:/usr/share/gcc-data/x86_64-pc-linux-gnu/4.5.4/info:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.22/info" INPUT_DEVICES="keyboard mouse evdev" IUSE_IMPLICIT="prefix" KERNEL="linux" LANG="pl_PL" LANGUAGE="48" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LC_ALL="pl_PL" LDFLAGS="-Wl,--as-needed,-O1" LDFLAGS_amd64="-m elf_x86_64" LDFLAGS_x32="-m elf32_x86_64" LDFLAGS_x86="-m elf_i386" LESS="-R -M --shift 5" LESSOPEN="|lesspipe %s" LIBDIR_amd64="lib64" LIBDIR_amd64_fbsd="lib64" LIBDIR_n32="lib32" LIBDIR_n64="lib64" LIBDIR_o32="lib" LIBDIR_ppc="lib32" LIBDIR_ppc64="lib64" LIBDIR_s390="lib32" LIBDIR_s390x="lib64" LIBDIR_sparc32="lib32" LIBDIR_sparc64="lib64" LIBDIR_x32="libx32" LIBDIR_x86="lib32" LIBDIR_x86_fbsd="lib32" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="pl" LOGNAME="root" MAIL="/var/mail/root" MAKEOPTS="-j6" MANPATH="/usr/local/share/man:/usr/share/man:/usr/share/gcc-data/x86_64-pc-linux-gnu/4.5.4/man:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.22/man" MULTILIB_ABIS="amd64 x86" MULTILIB_STRICT_DENY="64-bit.*shared object" MULTILIB_STRICT_DIRS="/lib32 /lib /usr/lib32 /usr/lib /usr/kde/*/lib32 /usr/kde/*/lib /usr/qt/*/lib32 /usr/qt/*/lib /usr/X11R6/lib32 /usr/X11R6/lib" MULTILIB_STRICT_EXEMPT="(perl5|gcc|gcc-lib|binutils|eclipse-3|debug|portage|udev)" NETBEANS="apisupport cnd groovy gsf harness ide identity j2ee java mobility nb php profiler soa visualweb webcommon websvccommon xml" PAGER="/usr/bin/less" PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.5.4:/opt/hp/hpacucli" PHP_TARGETS="php5-3" PKGDIR="/usr/portage/packages" PORTAGE_ARCHLIST="ppc sparc64-freebsd ppc-openbsd x86-openbsd ppc64 x86-winnt x86-fbsd ppc-aix alpha arm x86-freebsd s390 amd64 arm-linux x86-macos x64-openbsd ia64-hpux hppa x86-netbsd ppc64-linux x86-cygwin amd64-linux ia64-linux x86 sparc-solaris x64-freebsd sparc64-solaris x86-linux x64-macos sparc m68k-mint ia64 mips ppc-macos x86-interix hppa-hpux amd64-fbsd x64-solaris m68k sh x86-solaris sparc-fbsd" PORTAGE_BINHOST_CHUNKSIZE="3000" PORTAGE_BIN_PATH="/usr/lib64/portage/bin" PORTAGE_COMPRESS_EXCLUDE_SUFFIXES="css gif htm[l]? jp[e]?g js pdf png" PORTAGE_CONFIGROOT="/" PORTAGE_DEBUG="0" PORTAGE_DEPCACHEDIR="/var/cache/edb/dep" PORTAGE_ECLASS_WARNING_ENABLE="0" PORTAGE_ELOG_CLASSES="log warn error" PORTAGE_ELOG_MAILFROM="portage@localhost" PORTAGE_ELOG_MAILSUBJECT="[portage] ebuild log for ${PACKAGE} on ${HOST}" PORTAGE_ELOG_MAILURI="root" PORTAGE_ELOG_SYSTEM="save_summary echo" PORTAGE_FETCH_CHECKSUM_TRY_MIRRORS="5" PORTAGE_FETCH_RESUME_MIN_SIZE="350K" PORTAGE_GID="250" PORTAGE_GPG_SIGNING_COMMAND="gpg --sign --digest-algo SHA256 --clearsign --yes --default-key "${PORTAGE_GPG_KEY}" --homedir "${PORTAGE_GPG_DIR}" "${FILE}"" PORTAGE_INST_GID="0" PORTAGE_INST_UID="0" PORTAGE_NICENESS="19" PORTAGE_OVERRIDE_EPREFIX="" PORTAGE_PYM_PATH="/usr/lib64/portage/pym" PORTAGE_RSYNC_INITIAL_TIMEOUT="500" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_RSYNC_RETRIES="-1" PORTAGE_SYNC_STALE="30" PORTAGE_TMPDIR="/var/tmp" PORTAGE_VERBOSE="1" PORTAGE_WORKDIR_MODE="0700" PORTDIR="/usr/portage" PORTDIR_OVERLAY="" PORT_LOGDIR="/var/log/portage" PORT_LOGDIR_CLEAN="find "${PORT_LOGDIR}" -type f ! -name "summary.log*" -mtime +7 -delete" PROFILE_ONLY_VARIABLES="ARCH ELIBC IUSE_IMPLICIT KERNEL USERLAND USE_EXPAND_IMPLICIT USE_EXPAND_UNPREFIXED USE_EXPAND_VALUES_ARCH USE_EXPAND_VALUES_ELIBC USE_EXPAND_VALUES_KERNEL USE_EXPAND_VALUES_USERLAND" PS1="\[\e[38;5;118m\]`Q=$?; [ "$Q" != 0 ] && echo "($Q) "`\[\033[0m\]\[\e[38;5;196m\][[ \[\e[38;5;39m\]\h \[\e[38;5;196m\]]] \[\e[38;5;226m\]:: \[\e[00;37m\]\w \[\e[00;37m\]\$ \[\033[0m\]" PWD="/root" PYTHONDONTWRITEBYTECODE="1" PYTHON_TARGETS="python3_2" RESUMECOMMAND="wget -c -t 3 -T 60 --passive-ftp -O "${DISTDIR}/${FILE}" "${URI}"" RESUMECOMMAND_RSYNC="rsync -avP "${URI}" "${DISTDIR}/${FILE}"" RESUMECOMMAND_SSH="bash -c "x=\${2#ssh://} ; host=\${x%%/*} ; port=\${host##*:} ; host=\${host%:*} ; [[ \${host} = \${port} ]] && port=22 ; exec rsync --rsh=\"ssh -p\${port}\" -avP \"\${host}:/\${x#*/}\" \"\$1\"" rsync "${DISTDIR}/${FILE}" "${URI}"" ROOT="/" ROOTPATH="/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.5.4:/opt/hp/hpacucli" RPMDIR="/usr/portage/rpm" RUBY_TARGETS="ruby19" SHELL="/bin/bash" SHLVL="1" SSH_CLIENT="89.68.40.128 38714 22" SSH_CONNECTION="89.68.40.128 38714 188.128.253.130 22" SSH_TTY="/dev/pts/0" SYMLINK_LIB="yes" SYNC="rsync://rsync.gentoo.org/gentoo-portage" TERM="rxvt" UNINSTALL_IGNORE="/lib/modules/*" USE="acl amd64 berkdb bzip2 cli cracklib crypt cups cxx dri gdbm gpm hardened iconv ipv6 justify mmx modules mudflap multilib ncurses nls nptl openmp pam pax_kernel pcre pppd readline session sse sse2 ssl tcpd unicode urandom zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CURL_SSL="gnutls" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="pl" PHP_TARGETS="php5-3" PYTHON_TARGETS="python3_2" RUBY_TARGETS="ruby19" USERLAND="GNU" VIDEO_CARDS="fbdev glint intel mach64 mga nouveau nv r128 radeon savage sis tdfx trident vesa via vmware dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" USER="root" USERLAND="GNU" USE_EXPAND="ALSA_CARDS ALSA_PCM_PLUGINS APACHE2_MODULES APACHE2_MPMS CALLIGRA_FEATURES CAMERAS COLLECTD_PLUGINS CROSSCOMPILE_OPTS CURL_SSL DRACUT_MODULES DVB_CARDS ELIBC FCDSL_CARDS FFTOOLS FOO2ZJS_DEVICES FRITZCAPI_CARDS GPSD_PROTOCOLS GRUB_PLATFORMS INPUT_DEVICES KERNEL LCD_DEVICES LIBREOFFICE_EXTENSIONS LINGUAS LIRC_DEVICES MISDN_CARDS NETBEANS_MODULES NGINX_MODULES_HTTP NGINX_MODULES_MAIL OFED_DRIVERS OPENMPI_FABRICS OPENMPI_OFED_FEATURES OPENMPI_RM PHP_TARGETS PYTHON_TARGETS QEMU_SOFTMMU_TARGETS QEMU_USER_TARGETS RUBY_TARGETS SANE_BACKENDS USERLAND VIDEO_CARDS VOICEMAIL_STORAGE XFCE_PLUGINS XTABLES_ADDONS" USE_EXPAND_HIDDEN="CROSSCOMPILE_OPTS ELIBC KERNEL USERLAND" USE_EXPAND_IMPLICIT="ARCH ELIBC KERNEL USERLAND" USE_EXPAND_UNPREFIXED="ARCH" USE_EXPAND_VALUES_ARCH="alpha amd64 amd64-fbsd amd64-linux arm arm-linux hppa hppa-hpux ia64 ia64-hpux ia64-linux m68k m68k-mint mips ppc ppc64 ppc64-linux ppc-aix ppc-macos ppc-openbsd s390 sh sparc sparc64-freebsd sparc64-solaris sparc-fbsd sparc-solaris x64-freebsd x64-macos x64-openbsd x64-solaris x86 x86-cygwin x86-fbsd x86-freebsd x86-interix x86-linux x86-macos x86-netbsd x86-openbsd x86-solaris x86-winnt" USE_EXPAND_VALUES_ELIBC="AIX Cygwin Darwin FreeBSD glibc HPUX Interix mintlib NetBSD OpenBSD SunOS uclibc Winnt" USE_EXPAND_VALUES_KERNEL="AIX Cygwin Darwin FreeBSD freemint HPUX Interix linux NetBSD OpenBSD SunOS Winnt" USE_EXPAND_VALUES_USERLAND="BSD GNU" USE_ORDER="env:pkg:conf:defaults:pkginternal:repo:env.d" VIDEO_CARDS="fbdev glint intel mach64 mga nouveau nv r128 radeon savage sis tdfx trident vesa via vmware dummy v4l" WANT_MP="true" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Could you try with hardened-sources-3.5.4-r2? I had filesystem related problems with 3.5.4-r1 that was fixed with the -r1 and currently I am compiling glibc-2.15-r3 on that system, with CONFIG_PAX_RANDMMAP=y The paste seems to be cut-off at the last line too so important information might have been left out from that too.
(In reply to comment #1) > Could you try with hardened-sources-3.5.4-r2? > > I had filesystem related problems with 3.5.4-r1 that was fixed with the -r1 > and currently I am compiling glibc-2.15-r3 on that system, with > CONFIG_PAX_RANDMMAP=y > > The paste seems to be cut-off at the last line too so important information > might have been left out from that too. With -r2 I mean, sorry for typo.
Just to start some differential issue analysis, can you try with MAKEOPTS="-j1" ?
Like I've said, I also tried newest hardened 3.6.4 with the same result, so if this was wixed in 3.5.4-r2 it also should work with 3.6.x, right? Log from emerging glibc isn't cut off, this is all I see before reboot. Every time. I'll try -j1 in hour or two.
(In reply to comment #4) > Like I've said, I also tried newest hardened 3.6.4 with the same result, so > if this was wixed in 3.5.4-r2 it also should work with 3.6.x, right? > > Log from emerging glibc isn't cut off, this is all I see before reboot. > Every time. > > I'll try -j1 in hour or two. I emerged glibc-2.15-r3 on a 3.5.4-r2 kernel just now and worked without any hitch. Will be interesting to see if -j1 solved it for you, I have 2 6128 opteron and run -j16 and seeing as it's a HP Proliant server hardware issues can most likely be ruled out as you stated in the initial report.
Have you tride older kernels?
Could you give us the output of gcc-config -l, I can't find -fno-stack-protector in the CFLAGS part of your paste. It should be appended by the fully hardened compiler.
I'm lost. I had few more files [for smaller packets] corrupted in /var/db/pkg [after first reboot] so I removed those folders, did emerge -1 for them and emerge -e world. Just to be sure, not sorry. And now, when I tried to recreate this reboot for you, I can't. Before I had no problem with it, happened all the time. Now the same config [http://bpaste.net/show/55081/] seems to be working fine. Event with -j6. Looks like we've lost it :/ # gcc-config -l [1] x86_64-pc-linux-gnu-4.5.4 * [2] x86_64-pc-linux-gnu-4.5.4-hardenednopie [3] x86_64-pc-linux-gnu-4.5.4-hardenednopiessp [4] x86_64-pc-linux-gnu-4.5.4-hardenednossp [5] x86_64-pc-linux-gnu-4.5.4-vanilla
Well for your own sanity I suggest you do a memcheck and SMART check to rule out faulty hardware for reals. What filesystem you use?
I did test both memory and cpu for at least few hours. RAID was checked by local people in server house, I don't have too much of a experience with those HP Arrays. Filesystem for / is [for now] ext3. I've tried mrproper/config/make one more time, and it's still working. No reboots. Problem was somewhere, but looks like it wasn't entirely kernel's fault, since emerge -e world fixed it. But I'll keep an eye on everything. Sorry for taking your time.
No problem, good that you aren't seeing the issue anymore. Might have been a gremlin that was removed by the reboot or such. Ext3 shouldn't have issues with the recently found bug either, and that is also in newer kernels than you were using.
