Commit "e5fdfd9 toolchain.eclass: automatically select latest gcc-config profile when unmerging active version #529608" in the portage tree started choosing the last gcc profile when none was previously selected. For hardened stages, this is always the *-vanilla profile (since it is latest alphabetically). This change exposed a bug in catalyst: the setup_gcc line in stage1-preclean-chroot.sh has no effect because it runs in the extracted stage3, not the stage1root. This means it appears to change the gcc profile back to the first available (which would be the expected hardened one), but actually does nothing. This can be fixed by exporting ROOT=/tmp/stage1root so setup_gcc operates on the stage1 as expected. Alternatively, toolchain.eclass could be changed to select the first appropriate gcc profile instead of the last. There is also a bug related to choosing a gcc profile: clst_CHOST is always empty, so "ls ${clst_CHOST}-*" in chroot-functions.sh interprets -* as an option. This, however, does not affect which profile gets chosen. Reproducible: Always Steps to Reproduce: 1. Use catalyst to create any hardened stage1 2. grep GCC_SPECS /etc/profile.env in the stage1root Actual Results: export GCC_SPECS='x86_64-gentoo-linux-musl-4.9.3-vanilla' Expected Results: export GCC_SPECS='' Portage 2.2.28 (python 2.7.10-final-0, hardened/linux/musl/amd64, gcc-4.9.3, musl-1.1.14, 4.4.8-hardened-r1 x86_64) ================================================================= System Settings ================================================================= System uname: Linux-4.4.8-hardened-r1-x86_64-Intel-R-_Xeon-R-_CPU_X5675_@_3.07GHz-with-gentoo-2.2 KiB Mem: 49513572 total, 18773092 free KiB Swap: 30916604 total, 30916604 free sh bash 4.3_p42-r1 ld GNU ld (Gentoo 2.25.1 p1.1) 2.25.1 app-shells/bash: 4.3_p42-r1::gentoo dev-lang/perl: 5.20.2::gentoo dev-lang/python: 2.7.10-r1::gentoo, 3.4.3-r1::gentoo sys-apps/baselayout: 2.2::gentoo sys-apps/sandbox: 2.10-r99::musl sys-devel/autoconf: 2.69::gentoo sys-devel/automake: 1.15::gentoo sys-devel/binutils: 2.25.1-r1::gentoo sys-devel/gcc: 4.9.3-r99::musl sys-devel/gcc-config: 1.8-r1::gentoo sys-devel/libtool: 2.4.6::gentoo sys-devel/make: 4.1-r1::gentoo sys-kernel/linux-headers: 4.3-r99::musl (virtual/os-headers) sys-libs/musl: 1.1.14::gentoo Repositories: gentoo location: /usr/portage sync-type: rsync sync-uri: rsync://rsync.gentoo.org/gentoo-portage priority: -1000 ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-gentoo-linux-musl" CFLAGS="-fomit-frame-pointer -march=westmere -mtune=westmere -O2 -pipe" CHOST="x86_64-gentoo-linux-musl" CONFIG_PROTECT="/etc /etc/grs/systems.conf" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/sandbox.d /etc/terminfo" CXXFLAGS="-fomit-frame-pointer -march=westmere -mtune=westmere -O2 -pipe" DISTDIR="/usr/portage/distfiles" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://distfiles.gentoo.org" INSTALL_MASK="charset.alias" LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,--hash-style=both" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" USE="amd64 cli cracklib crypt cxx dri fortran hardened iconv ipv6 modules ncurses nls nptl openmp pax_kernel pcre pic readline seccomp session ssl tcpd unicode xattr zlib" ABI_X86="64" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdirusertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="musl" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt g psclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-consolepresenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_4" RUBY_TARGETS="ruby20 ruby21" USERLAND="GNU" VIDEO_CARDS="dummy fbdev v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, LANG, LC_ALL, MAKEOPTS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON ================================================================= Package Settings ================================================================= dev-util/catalyst-3.0_rc1::gentoo was built with the following: USE="-ccache -doc" PYTHON_TARGETS="python2_7 -python3_4 -python3_5"
(In reply to Samuel Holland from comment #0) > Commit "e5fdfd9 toolchain.eclass: automatically select latest gcc-config > profile when unmerging active version #529608" in the portage tree started > choosing the last gcc profile when none was previously selected. For > hardened stages, this is always the *-vanilla profile (since it is latest > alphabetically). > > This change exposed a bug in catalyst: the setup_gcc line in > stage1-preclean-chroot.sh has no effect because it runs in the extracted > stage3, not the stage1root. This means it appears to change the gcc profile > back to the first available (which would be the expected hardened one), but > actually does nothing. These needs to be fixed. I'm looking at it now. > > This can be fixed by exporting ROOT=/tmp/stage1root so setup_gcc operates on > the stage1 as expected. Alternatively, toolchain.eclass could be changed to > select the first appropriate gcc profile instead of the last. Both need to be fixed because the toolchain bug will come up in other contexts. > > There is also a bug related to choosing a gcc profile: clst_CHOST is always > empty, so "ls ${clst_CHOST}-*" in chroot-functions.sh interprets -* as an > option. This, however, does not affect which profile gets chosen. >
Add something like this after the target=$(.... line in the eclass: target=$(gcc-config -S $target | awk '{print $1 "-" $2}')
(In reply to Jason Zaman from comment #2) that seems like a reasonable default. it's still not entirely correct though. for that, we'd have to update gcc-config to return the old config, extract the subprofile bits out, and then see if the current version supports it. that way if you had hardenednopie selected, we'd continue using that.
(In reply to SpanKY from comment #3) > (In reply to Jason Zaman from comment #2) > > that seems like a reasonable default. it's still not entirely correct > though. for that, we'd have to update gcc-config to return the old config, > extract the subprofile bits out, and then see if the current version > supports it. that way if you had hardenednopie selected, we'd continue > using that. i need this pretty asap so even if its not complete, it works. i'm testing now with a couple of catalyst runs. if it works, i'll prepare a patch and send it to gentoo-dev@. if all is okay, i'll commit in a few days.
(In reply to Anthony Basile from comment #4) > (In reply to SpanKY from comment #3) > > (In reply to Jason Zaman from comment #2) > > > > that seems like a reasonable default. it's still not entirely correct > > though. for that, we'd have to update gcc-config to return the old config, > > extract the subprofile bits out, and then see if the current version > > supports it. that way if you had hardenednopie selected, we'd continue > > using that. > > i need this pretty asap so even if its not complete, it works. i'm testing > now with a couple of catalyst runs. if it works, i'll prepare a patch and > send it to gentoo-dev@. if all is okay, i'll commit in a few days. this did not fix it for me. i'm still getting that the specs switched to vanilla. i'm testing reverting commit e5fdfd9. this is pretty disastrous for me as its broken a lot of automated stuff.
(In reply to Anthony Basile from comment #5) > this did not fix it for me. i'm still getting that the specs switched to > vanilla. > > i'm testing reverting commit e5fdfd9. this is pretty disastrous for me as > its broken a lot of automated stuff. The easy short term fix for catalyst is below. diff -ur a/usr/share/catalyst/targets/stage1/stage1-preclean-chroot.sh b/usr/share/catalyst/targets/stage1/stage1-preclean-chroot.sh --- a/usr/share/catalyst/targets/stage1/stage1-preclean-chroot.sh 2016-05-19 18:21:11.084413623 -0500 +++ b/usr/share/catalyst/targets/stage1/stage1-preclean-chroot.sh 2016-05-22 18:08:28.273952913 -0500 @@ -1,5 +1,6 @@ #!/bin/bash +export ROOT=/tmp/stage1root export RUN_DEFAULT_FUNCS="no" source /tmp/chroot-functions.sh @@ -8,8 +12,6 @@ show_debug # Now, some finishing touches to initialize gcc-config.... -unset ROOT - setup_gcc setup_binutils
(In reply to Samuel Holland from comment #6) > (In reply to Anthony Basile from comment #5) > > this did not fix it for me. i'm still getting that the specs switched to > > vanilla. > > > > i'm testing reverting commit e5fdfd9. this is pretty disastrous for me as > > its broken a lot of automated stuff. > > The easy short term fix for catalyst is below. > > diff -ur a/usr/share/catalyst/targets/stage1/stage1-preclean-chroot.sh > b/usr/share/catalyst/targets/stage1/stage1-preclean-chroot.sh > --- a/usr/share/catalyst/targets/stage1/stage1-preclean-chroot.sh > 2016-05-19 18:21:11.084413623 -0500 > +++ b/usr/share/catalyst/targets/stage1/stage1-preclean-chroot.sh > 2016-05-22 18:08:28.273952913 -0500 > @@ -1,5 +1,6 @@ > #!/bin/bash > > +export ROOT=/tmp/stage1root > export RUN_DEFAULT_FUNCS="no" > > source /tmp/chroot-functions.sh > @@ -8,8 +12,6 @@ > show_debug > > # Now, some finishing touches to initialize gcc-config.... > -unset ROOT > - > setup_gcc > setup_binutils have you tested this? all i did was add the line from comment #2 to toolchain.eclass.
(In reply to Anthony Basile from comment #7) > (In reply to Samuel Holland from comment #6) > > (In reply to Anthony Basile from comment #5) > > > this did not fix it for me. i'm still getting that the specs switched to > > > vanilla. > > > > > > i'm testing reverting commit e5fdfd9. this is pretty disastrous for me as > > > its broken a lot of automated stuff. > > > > The easy short term fix for catalyst is below. > > > > diff -ur a/usr/share/catalyst/targets/stage1/stage1-preclean-chroot.sh > > b/usr/share/catalyst/targets/stage1/stage1-preclean-chroot.sh > > --- a/usr/share/catalyst/targets/stage1/stage1-preclean-chroot.sh > > 2016-05-19 18:21:11.084413623 -0500 > > +++ b/usr/share/catalyst/targets/stage1/stage1-preclean-chroot.sh > > 2016-05-22 18:08:28.273952913 -0500 > > @@ -1,5 +1,6 @@ > > #!/bin/bash > > > > +export ROOT=/tmp/stage1root > > export RUN_DEFAULT_FUNCS="no" > > > > source /tmp/chroot-functions.sh > > @@ -8,8 +12,6 @@ > > show_debug > > > > # Now, some finishing touches to initialize gcc-config.... > > -unset ROOT > > - > > setup_gcc > > setup_binutils > > have you tested this? all i did was add the line from comment #2 to > toolchain.eclass. Yes, I've been doing daily catalyst stage1->stage2->stage3->stage1 runs for almost a week now (since I first mentioned it in IRC). This is with dev-util/catalyst-3.0_rc1, and I made no changes to toolchain.eclass (this diff is all).
(In reply to Samuel Holland from comment #8) > (In reply to Anthony Basile from comment #7) > > > > have you tested this? all i did was add the line from comment #2 to > > toolchain.eclass. > > Yes, I've been doing daily catalyst stage1->stage2->stage3->stage1 runs for > almost a week now (since I first mentioned it in IRC). This is with > dev-util/catalyst-3.0_rc1, and I made no changes to toolchain.eclass (this > diff is all). this can't be all there is to it because i hit the problem between stage2 and stage3.
(In reply to Anthony Basile from comment #9) > (In reply to Samuel Holland from comment #8) > > (In reply to Anthony Basile from comment #7) > > > > > > have you tested this? all i did was add the line from comment #2 to > > > toolchain.eclass. > > > > Yes, I've been doing daily catalyst stage1->stage2->stage3->stage1 runs for > > almost a week now (since I first mentioned it in IRC). This is with > > dev-util/catalyst-3.0_rc1, and I made no changes to toolchain.eclass (this > > diff is all). > > this can't be all there is to it because i hit the problem between stage2 > and stage3. Oh, I have in package.accept_keywords "~sys-devel/gcc-config-1.8 ~amd64". You need that version to use /lib/gentoo/functions.sh instead of /etc/init.d/functions.sh because stage2 doesn't have openrc (which provides /etc/init.d/functions.sh). (So gcc-config in stage2 is producing no output, toolchain.eclass is seeing 'no current profile' and picking the wrong one again). Since I don't use openrc, I already had that unmasked. That's the only thing I can think of that's relevant.
(In reply to Samuel Holland from comment #10) > (In reply to Anthony Basile from comment #9) > > (In reply to Samuel Holland from comment #8) > > > (In reply to Anthony Basile from comment #7) > > > > > > > > have you tested this? all i did was add the line from comment #2 to > > > > toolchain.eclass. > > > > > > Yes, I've been doing daily catalyst stage1->stage2->stage3->stage1 runs for > > > almost a week now (since I first mentioned it in IRC). This is with > > > dev-util/catalyst-3.0_rc1, and I made no changes to toolchain.eclass (this > > > diff is all). > > > > this can't be all there is to it because i hit the problem between stage2 > > and stage3. > > Oh, I have in package.accept_keywords "~sys-devel/gcc-config-1.8 ~amd64". > You need that version to use /lib/gentoo/functions.sh instead of > /etc/init.d/functions.sh because stage2 doesn't have openrc (which provides > /etc/init.d/functions.sh). (So gcc-config in stage2 is producing no output, > toolchain.eclass is seeing 'no current profile' and picking the wrong one > again). > > Since I don't use openrc, I already had that unmasked. That's the only thing > I can think of that's relevant. That makes sense, I'll test tomorrow.
(In reply to Anthony Basile from comment #11) > (In reply to Samuel Holland from comment #10) > > > > Since I don't use openrc, I already had that unmasked. That's the only thing > > I can think of that's relevant. > > > That makes sense, I'll test tomorrow. Bumping to =sys-devel/gcc-config-1.8-r1 fixed the issue between stage2->stage3. Because gcc-config was broken at stage2 (because functions.sh was moved) the entire do_gcc_config() code block was messed up, and possibly more. I did not have to hack the toolchain.eclass and I did not have to touch catalyst. I'm still not 100% convinced about the need to export ROOT=/tmp/stage1root. I have to look over that code more carefully, but it seems to me that it would be wrong to export it and not just unset it as we are currently doing. Anyhow, I'm going to continue testing today with another set of stages (the uclibc amd64/x86 builds) and if that works with gcc-config-1.8-r1 then i'm going to rapid stabilize it. @vapier any reason we should not move ahead with 1.8-r1 ???
(In reply to Anthony Basile from comment #12) > I'm still not 100% convinced about the need to > export ROOT=/tmp/stage1root. Actually this is correct. I looked at the code and it sits outside of stage1root, so in order for gcc-config to get the right value of ROOT, you need to set it. So to summarize: 1) you need the bump to gcc-config-1.8-r1, 2) you need the fix to stage1-preclean-chroot.sh and 3) you can leave commit e5fdfd9 to the toolchain.eclass alone.
(In reply to Anthony Basile from comment #13) > (In reply to Anthony Basile from comment #12) > > I'm still not 100% convinced about the need to > > export ROOT=/tmp/stage1root. > > Actually this is correct. I looked at the code and it sits outside of > stage1root, so in order for gcc-config to get the right value of ROOT, you > need to set it. > > So to summarize: 1) you need the bump to gcc-config-1.8-r1, 2) you need the > fix to stage1-preclean-chroot.sh and 3) you can leave commit e5fdfd9 to the > toolchain.eclass alone. okay part of the way there. i've pushed the above fix to stage1-preclean-chroot.sh to the 2.X branch of catalyst. https://gitweb.gentoo.org/proj/catalyst.git/commit/?h=2.X&id=5fd2d5edd3c4c1e99687beb9acc130bab162866b it maybe need to be migrated to the other branches. however, i must say, as i look at these catalyst scripts, i get increasingly worried. there's other pieces of that script and others that look like they're going to run in the parent root while they should be running in the stage1root chroot.
(In reply to Anthony Basile from comment #13) > (In reply to Anthony Basile from comment #12) > > I'm still not 100% convinced about the need to > > export ROOT=/tmp/stage1root. > > Actually this is correct. I looked at the code and it sits outside of > stage1root, so in order for gcc-config to get the right value of ROOT, you > need to set it. > > So to summarize: 1) you need the bump to gcc-config-1.8-r1, 2) you need the > fix to stage1-preclean-chroot.sh and 3) you can leave commit e5fdfd9 to the > toolchain.eclass alone. the bump to gcc-config is not needed since gcc-config is run in the unpacked stage3 which does have openrc installed. it would break on systemd stages though.
the revert to -vanilla and the fix in #2 are also relevant in everyday use. Without a patched toolchain.eclass, systems revert to -vanilla every time gcc is updated or re-emerged. It seems this bug should be impacting everyone on a hardened profile at the very least right now.
@hardened: this should be obsolete now that hardened has also only one profile per compiler... correct?
Is only one profile now with newer gcc.
