Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 149649

Summary: develop independent gcc versions containing advanced security technology
Product: Gentoo Linux Reporter: Alexander Gabert (RETIRED) <pappy>
Component: [OLD] UnspecifiedAssignee: Gentoo Linux bug wranglers <bug-wranglers>
Status: RESOLVED CANTFIX    
Severity: major CC: atoth, basic, bugs+gentoo, colding, data, gengor, jeff.rollin, joris.vandenbogerd, kanelxake, l.mierzwa, lkml_ccc, natanael.copa, pageexec, ps.m, samuel, togge.gentoo, vslavik, zorry
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
URL: http://bugs.gentoo.org/show_bug.cgi?id=149292
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on: 94325    
Bug Blocks:    
Attachments: portage-hardened-glibc-gcc.patch
portage-hardened-glibc-gcc.patch with needed files and ppc tryout
gcc-hardened-callouts.patch
correction for i386 endfile handling, goes to upstream dir
gcc-4.1.1 correction for i386 endfile handling, goes to nondef dir
toolchain diff
some more bits for toolchain.eclass
updated set of specs
gcc-hardened-framework.patch
gcc -DHAVE_LD_PIE -DUSE_UCLIBC -o test test.c; ./test
piepatch 9.0.5 for gcc-4.1.1
piepatch 9.0.5 for gcc-3.4.6
sparc failure building glibc-2.3.6-r5 with hardened kevquinn
hake build failure
luna build failure
failure on bender (latest kevquinn overlay)
current build failures
another build failure on bender
piepatch 9.0.6 for gcc-4.1.1
msecure-plt support for ppc32

Description Alexander Gabert (RETIRED) gentoo-dev 2006-09-30 13:32:55 UTC
This is a reference bug concerning a TODO item, please assign it to the hardened developers for further reference and documentation purposes.

This bug describes the process of the migration and the paperwork for comment 103 of the following bug:

http://bugs.gentoo.org/show_bug.cgi?id=94325#c103

My first inspections show this:

gcc/gcc.c contains a list of existing defined specs sections.

For each feature of comment 103, there needs to be a new definition in this list:

#define INIT_STATIC_SPEC(NAME,PTR) \
{ NAME, NULL, PTR, (struct spec_list *) 0, sizeof (NAME) - 1, 0 }

/* List of statically defined specs.  */
static struct spec_list static_specs[] =
{
  INIT_STATIC_SPEC ("asm",                      &asm_spec),
  INIT_STATIC_SPEC ("asm_debug",                &asm_debug),
  INIT_STATIC_SPEC ("asm_final",                &asm_final_spec),
  INIT_STATIC_SPEC ("asm_options",              &asm_options),
  INIT_STATIC_SPEC ("invoke_as",                &invoke_as),

...


Alex
Comment 1 Alexander Gabert (RETIRED) gentoo-dev 2006-09-30 13:37:26 UTC
This is from Kevin Quinn on the old bug.  It's quoted here for reference.

[Quote]
We (in particular psm) have a range of ideas for what to do with the gcc specs
themselves; including:

1) Doing bind_now/relro in ld (binutils), rather than gcc-specs (or in addition
to, for transition since they don't conflict)

2) Factoring the toolchain mods in to easy-to see/maintain chunks, so that you
would see for example:

*cc1:
%(cc1_cpu) %{profile:-p} %(cc1_ssp) %(cc1_pie)

*cc1_ssp:
%{D__KERNEL__|nostdlib|nodefaultlibs|fno-stack-protector|fstack-protector|fno-stack-protector-all:;:-fstack-protector-all}

*cc1_pie:
%{D__KERNEL__|static|nostdlib|nostartfiles|fPIC|fpic|fno-PIC|fno-pic|fPIE|fpie|fno-PIE|fno-pie|nopie:;:-fPIE}

*link_command:
%{!fsyntax-only:%{!c:%{!M:%{!MM:%{!E:%{!S:    %(linker) %l
%{pie:-pie;:%(link_command_pie)} %X %{o*} %{A} %{d} %{e*} %{m} %{N} %{n} %{r}  
 %{s} %{t} %{u*} %{x} %{z} %{Z} %{!A:%{!nostdlib:%{!nostartfiles:%S}}}   
%{static:} %{L*} %(mfwrap) %(link_libgcc) %o %(mflib)   
%{fprofile-arcs|fprofile-generate|coverage:-lgcov}   
%{!nostdlib:%{!nodefaultlibs:%(link_ssp) %(link_gcc_c_sequence)}}   
%{!A:%{!nostdlib:%{!nostartfiles:%E}}} %{T*} }}}}}}

*link_command_pie:
%{D__KERNEL__|static|A|nostdlib|nostartfiles|fPIC|fpic|fno-PIC|fno-pic|fPIE|fpie|fno-PIE|fno-pie|nopie:;:-pie}

(gcc specs still did bind-now/relro it would be done as new
"link_command_zrelro" and "link_command_znow" definitions, referenced from the
appropriate place in the link_command definition)

This means that the switched specs files no longer have to be full sets - they
can be just the modifications necessary; i.e. they just need to define cc1_ssp,
cc1_pie and link_command_pie.

Note that the GCC_SPECS variable can now contain more than one entry; e.g. one
could do:

GCC_SPECS="default_ssp.specs:default_pie.specs" gcc ...

where default_ssp.specs would contain the cc1_ssp definition and
default_pie.specs would be the cc1_pie definition.

There are pros and cons to having the builtin specs themselves hardened, still
undecided on that.
[/Quote]
Comment 2 Peter S. Mazinger 2006-10-01 12:00:33 UTC
My open/undecided issues are (I have all of this already implemented for gcc-3.4.6/4.1.1 - well, for 4.0.x too, but I wouldnt support it due to additionally needed ssp patches):
a. what do we do with fno-stack-protector-all, do we patch gcc to support it or leave it as it is now in gcc-4.1, last option applies, so if -fstack-protector is last, after an -fstack-protector-all, the non-all version wins. Independently of which path we go, I would like to keep gcc-3.x and 4.x in sync, so the hardened profiles work the same way (easier to handle in toolchain.eclass that way)
b. should gcc itself be built as "hardened", or we leave it vanilla in any case, adding only the new specs
c. do we go with GCC_SPECS supporting more specs files and provide mini-specs for each functionality (pie/ssp/ssp_all/relro/now), adding all the needed ones to the list of GCC_SPECS (havent looked to eselect if supported) or we provide combined "maxi"-specs to have the same profiles as earlier (the latter works probably with eselect)
d. the most disturbing change in gcc-4.1 is, that we lost the info about where the stack smashing occured. Are we willing to stay like that, or we patch gcc to provide the function failing as well (that would make gcc-vanilla not quite vanilla)

Before I provide the patches, I would like to have the answers to the questions above.

Note:
due to the changes I did pie/ssp detection used in ebuilds does not work anymore, needs adaption as well
Comment 3 Kevin F. Quinn (RETIRED) gentoo-dev 2006-10-01 14:36:41 UTC
(In reply to comment #2)
> a. what do we do with fno-stack-protector-all

My preference is to leave gcc-4.1 unchanged (i.e. don't add -fno-stack-protector-all, leave the last-setting-wins in place), just tweak the filter-flags() function in flag-o-matic() to do the right thing according to gcc version (and possibly also append-flags() ).

My arguments are:

1) it's unlikely that anyone actually uses '-fno-stack-protector-all', except when trying to undo the hardened specs - which should be done via filter-flags() anyway (any ebuilds that don't use flag-o-matic for this can be fixed).

2) as a general aim, I think it's a good idea to try to keep gcc as close to upstream as possible.  Ideally there should be no impact on the vanilla compiler from hardened (but see (b) below).

> Independently of which path we go, I would like to keep gcc-3.x and 4.x in
> sync, so the hardened profiles work the same way (easier to handle in
> toolchain.eclass that way)

I'm not worried about that.  We probably have to leave the current 3.x management in toolchain.eclass anyway, due to the long-standing issue of never removing functionality from an eclass.

My approach to differences between the two major versions would be to handle that in eclasses (i.e. abstract it from ebuilds).  If we leave the 3.x stuff as it stands, it makes it easier to detect which type of handling is present - just a check on gcc major version - whereas if we go back and modify the older toolchains, we need to do more run-time detection to work out which way the older toolchain was built.

> b. should gcc itself be built as "hardened", or we leave it vanilla in any
> case, adding only the new specs

The case for having the compiler itself built PIE/SSP is relatively weak, I think (unlike glibc, where a stack overflow could be exploited by any application that links to it, including suid apps).  However there are the support libraries, which I think should be SSP and (for the .a's) PIE.

Ah; if you're talking about whether the builtin specs should be hardened or not - I would say no (i.e. leave the builtin specs vanilla, apart from the minispec callouts).  However others make a case for having the built-in specs hardened on a hardened machine.  Not building in the hardened minispecs means the patch to gcc itself becomes much smaller (just adds the call-outs to the minispecs, and default vanilla minispecs for some items), and we no longer have to build xgcc once for each spec combination (see toolchain.eclass).

> c. do we go with GCC_SPECS supporting more specs files and provide mini-specs
> for each functionality (pie/ssp/ssp_all/relro/now), adding all the needed ones
> to the list of GCC_SPECS (havent looked to eselect if supported) or we provide
> combined "maxi"-specs to have the same profiles as earlier (the latter works
> probably with eselect)

eselect-compiler supports multiple specs in GCC_SPECS (that was added a long while back).  It's gcc-config that still needs to be updated (bug #125805 - waiting for vapier to get around to it).

I prefer the mini-specs approach (that's what I have implemented locally).

Note also that for bind-now, relro, I'd like to consider doing that in binutils (by changing the defaults in ld itself) rather than gcc specs.  In the end both can co-exist, especially easily with minispecs, so maybe that's something we can leave for later.

> d. the most disturbing change in gcc-4.1 is, that we lost the info about where
> the stack smashing occured. Are we willing to stay like that, or we patch gcc
> to provide the function failing as well (that would make gcc-vanilla not quite
> vanilla)

My opinion here is that we follow upstream.  As pageexec said elsewhere, once a stack smash occurs, it can be tracked down more effectively via gdb (which reminds me - need to rig the stack smash handler to cause a core dump, which it doesn't at the moment).  On the (slight) plus side, without the extra parameters to the stack smash handler, there is less further modification of the stack between the smash detection and the abort.

> Before I provide the patches, I would like to have the answers to
> the questions above.
> 
> Note:
> due to the changes I did pie/ssp detection used in ebuilds does not work
> anymore, needs adaption as well

Yes; I'm expecting that (tweaks to toolchain-funcs.eclass, and possibly ebuilds that try to do things directly although there shouldn't be many of those).
Comment 4 Alexander Gabert (RETIRED) gentoo-dev 2006-10-05 11:01:24 UTC
i am following Kevins thoughts on the source building of gcc and glibc.

My thoughts on the modularized/single specs:

The specs file is the guts and gore of gcc.
Nobody likes to mess with it, and most users don't even know about it.

And if they know about it, i don't think it would make a big difference to have ONE big file or MANY small files.

My opinion: if you are into specs editing, you don't care.

For a source building distribution like Gentoo, i would estimate that 20% to 33% of all tasks in the distribution is about compiling new applications and updates for security advisories and feature bugs.

This situation makes the compiler an important part of the acceptance of the distribution as a whole thing.

Now, i am thinking: for runtime performance of the gcc compiling these applications my concerns are that opening and parsing several small files especially with MAKEOPTS=-j20 and thus having shitloads of gcc instances running in with small shifts of workloads, the modularized specs effort is taking more disk i/o and runtime performance than just sucking in one big file and parsing it internally.  This could be totally neutralized by hard disk performance and caches, but it's just my thoughts and i hope i'm not absolutely wrong.

Having the PATCHES modularized eases maintenance and is good for devs.
Having the specs modularized is unnecessary and can lead to increase the compilers internal overhead and thus have an impact on the workload of the compiler itself in my eyes.

Peter: it would be nice if you could attach the modularized gcc-4.1.1 patches here for our users to be able to start testing (nixnut was asking for PPC specifically today and i am refusing to repeat your already finished work by mocking up my preliminary "shots" into the 4.1.1 direction)


Thank you and have fun!


Alex
Comment 5 Peter S. Mazinger 2006-10-05 11:18:10 UTC
my bad, I have attached them to #149292
Comment 6 Kevin F. Quinn (RETIRED) gentoo-dev 2006-10-05 12:05:56 UTC
I don't think the cost of opening a few specs files is significant, even for a machine that did nothing but compilation all day.  Either way, merging the minispecs into one file is trivial, and can be done by the ebuild (well, toolchain.eclass).  The main reason for the minispecs is not so much the minispecs themselves, but the idea that the specs modifications are implemented with call-outs to new spec names, which makes it easier to override, and to see what the modifications are.


For reference/comparison, you can find my toolchain mods in my overlay at http://overlays.gentoo.org/dev/kevquinn.  They're not finished so I won't attach them here.  Currently gcc itself doesn't get built hardened, but that I'm planning on doing by setting GCC_SPECS during the build.  I also haven't split out the startfile/endfile stuff yet.

The core difference between what mine (will eventually) do, and what Peter's patches do, is that with my approach the compiler is built with (effectively) vanilla specs built-in for everyone (so hardened users always have GCC_SPECS set to the hardened minispecs), whereas Peter continues the approach of having different builtin specs for hardened and vanilla users.

FWIW my patching of gcc itself should end up being somewhat smaller, as it won't need to do the EFAULT_PIE/SSP etc stuff.
Comment 7 Peter S. Mazinger 2006-10-05 22:32:23 UTC
kevquinn: I have reread and reread your posting:
You can achieve what you want (having vanilla specs builtin by doing it in
toolchain.eclass only, without touching the piepatch tarballs, less the needed
change to be compliant with the gentoo used uclibc patch in 4.1.1). I had in mind
keeping compatibility with current behaviour as much as possible (sspall.specs
has to have %{!fno-stack-protector-all:-fstack-protector-all} for gcc-3.4.x), you
definitely go another route. If you change the logic of the piepatches, I am
asking you politely to change the patch tarballs name and versioning style,
document in changelog the fact that it is your version and not mine.
Comment 8 Kevin F. Quinn (RETIRED) gentoo-dev 2006-10-06 02:16:30 UTC
Peter; it's just a discussion point at the moment - whether to integrate the hardened specs into the gcc driver, or whether to have them outside.  There are pros and cons to both approaches, but I feel that if we have both actually implemented we can make an informed choice (same goes for building libc with ssp-all throughout, bug #94325, which affects whether we have the _LIBC/_LIBC_REENTRANT guard for ssp-all).


I've installed your version into a new directory in my dev overlays area, so it's easier for interested parties to try it out:

svn co http://overlays.gentoo.org/kevquinn/hardened/toolchain-psm

The only differences are the patch tweak I mentioned before, and I've moved the specs files into a specs subdirectory, to keep the sys-devel/gcc/files directory relatively clean, and obviously I've added the files needed for a gcc-4.1.1-r1 ebuild.
Comment 9 Peter S. Mazinger 2006-10-06 02:39:32 UTC
_LIBC* check was added because glibc failed to build, uClibc has an own SSP
config option to allow compile all of it at least with -fssp (-all is used for
all lib*.so, less libc.so), you can remove those anytime after you get glibc
fixed. There is also a patch for kernel, so the kernel check could go too (though
it is not proven that it is useful there at all).I personally want to keep
building gcc itself fully hardened, because it is a good test to see if something
is breaking on the way, before you installed a broken gcc. As an example: I know
that something broke between gcc-4.1.1 and -r1, because I do generation of the
specs (as it is done in gcc-3.4) SPLIT_SPECS is enabled. I know I do not need to
do it anymore, due to the minispecs, but doing it early enough, I know that this
version of gcc is expected to misbehave.
Comment 10 Peter S. Mazinger 2006-10-06 02:48:14 UTC
forgotten to mention, I should have rather said: not gcc-4.1.1 itself is broken,
some change in toolchain.eclass around that time did it
Comment 11 Alexander Gabert (RETIRED) gentoo-dev 2006-10-06 09:51:01 UTC
> I've installed your version into a new directory in my dev overlays area, so
> it's easier for interested parties to try it out:
> 
> svn co http://overlays.gentoo.org/kevquinn/hardened/toolchain-psm

 # svn co http://overlays.gentoo.org/kevquinn/hardened/toolchain-psm
svn: PROPFIND request failed on '/kevquinn/hardened/toolchain-psm'
svn: PROPFIND of '/kevquinn/hardened/toolchain-psm': 405 Method Not Allowed (http://overlays.gentoo.org)


Kevin, i will try to fetch your work via http, hope it goes well.

Alex
Comment 12 Alexander Gabert (RETIRED) gentoo-dev 2006-10-06 10:07:40 UTC
   19  svn checkout http://overlays.gentoo.org/svn/dev/kevquinn/hardened/toolchain
   20  svn checkout http://overlays.gentoo.org/svn/dev/kevquinn/hardened/toolchain-psm


this one did the trick for both overlays

a good introduction how to add and use overlays: 

http://overlays.gentoo.org/dev/wschlich
-> Getting started

There you can learn about using layman, but svn is cool for me too...


Alex
Comment 13 Peter S. Mazinger 2006-10-06 18:05:30 UTC
the move of -z now and relro to binutils is fine with me, it is more logical
and will do it all the time (not only if invoked by gcc). binutils-2.16.1-r3
has already the -z lazy patch, norelro was already there, so this could be
a first step, updating 2.16.1-r3/2.17[.50.0.5], I wouldnt care about others.
The eclass needs the handling of -z lazy.
kevquinn: your 2.17 is wrong s/bfd_dma/bfd_vma/ and you missed including the
needed header (as in 2.16)
Comment 14 Alexander Gabert (RETIRED) gentoo-dev 2006-10-07 04:42:04 UTC
 * The current gcc config appears valid, so it will not be
 * automatically switched for you.  If you would like to
 * switch to the newly installed gcc version, do the
 * following:

 * gcc-config x86_64-pc-linux-gnu-4.1.1
 * source /etc/profile

 # gcc-config x86_64-pc-linux-gnu-4.1.1
 * /usr/bin/gcc-config: Could not locate 'x86_64-pc-linux-gnu-4.1.1' in '/etc/env.d/gcc/' !
 13:31:01 [/space/chroots/chroot001:19137.pts-5.miranda]miranda /master/svn/overlays.gentoo.org/svn/dev/kevquinn/hardened/toolchain/sys-devel/gcc
 # env-update
>>> Regenerating /etc/ld.so.cache...
 13:31:07 [/space/chroots/chroot001:19137.pts-5.miranda]miranda /master/svn/overlays.gentoo.org/svn/dev/kevquinn/hardened/toolchain/sys-devel/gcc
 # gcc-config x86_64-pc-linux-gnu-4.1.1
 * /usr/bin/gcc-config: Could not locate 'x86_64-pc-linux-gnu-4.1.1' in '/etc/env.d/gcc/' !
 13:31:09 [/space/chroots/chroot001:19137.pts-5.miranda]miranda /master/svn/overlays.gentoo.org/svn/dev/kevquinn/hardened/toolchain/sys-devel/gcc
 # source /etc/profile
miranda gcc # gcc-config x86_64-pc-linux-gnu-4.1.1
 * /usr/bin/gcc-config: Could not locate 'x86_64-pc-linux-gnu-4.1.1' in '/etc/env.d/gcc/' !
miranda gcc # cd /etc/env.d/gcc
miranda gcc # ls
config                                      x86_64-pc-linux-gnu-4.1.1-hardened
x86_64-pc-linux-gnu-3.4.4                   x86_64-pc-linux-gnu-4.1.1-hardenednopie
x86_64-pc-linux-gnu-3.4.4-hardenednopie     x86_64-pc-linux-gnu-4.1.1-hardenednopiessp
x86_64-pc-linux-gnu-3.4.4-hardenednopiessp  x86_64-pc-linux-gnu-4.1.1-hardenednossp
x86_64-pc-linux-gnu-3.4.4-hardenednossp     x86_64-pc-linux-gnu-4.1.1-vanilla
x86_64-pc-linux-gnu-3.4.4-vanilla
miranda gcc # gcc-config x86_64-pc-linux-gnu-4.1.1-hardened
 * Switching native-compiler to x86_64-pc-linux-gnu-4.1.1-hardened ...                                    [ ok ]


The message at the end of the ebuild is a bit "erroneously" because it points to a profile which does not exist...

Maybe someone can change this.


Alex
Comment 15 Alexander Gabert (RETIRED) gentoo-dev 2006-10-07 05:21:50 UTC
Hello again,

specs modularization is working with Kevins ebuilds and patches.

Using built-in specs.
Reading specs from /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/hardened.specs
Reading specs from /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/pie.specs
Reading specs from /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/ssp.specs
Reading specs from /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/zrelro.specs
Reading specs from /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/znow.specs


 /usr/libexec/gcc/x86_64-pc-linux-gnu/4.1.1/cc1 -quiet -v vuln-stack.c -fstack-protector-all -fPIE -quiet -dumpbase vuln-stack.c -mtune=k8 -auxbase vuln-stack -version -o /tmp/ccPYgID3.s


 /usr/libexec/gcc/x86_64-pc-linux-gnu/4.1.1/collect2 --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -pie -z now -o vuln-stack /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../../../lib64/Scrt1.o /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../../../lib64/crti.o /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/crtbeginS.o -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../../../x86_64-pc-linux-gnu/lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../.. /tmp/ccalyi1V.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/crtendS.o /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../../../lib64/crtn.o

The assembler step has been left out and some other debug info too.

But there is one "probleM" i have with the current setup.

gcc -dumpspecs shows only the internal specs.

A good thing however is: when GCC_SPECS is unset, the gcc uses internal specs and skips the undefined portions.


Nonetheless the problem with gcc -dumpspecs i vote YES to the modularized specs and hope we can introduce the glibc changes to glibc-2.5 (vapiers mail to gentoo-dev and his private message we should not work on glibc-2.4 any more) and the specs patches to gcc-4.1.1.

Peter and Kevin: thanks for your work!


Alex
Comment 16 Alexander Gabert (RETIRED) gentoo-dev 2006-10-07 05:52:43 UTC
  /var/tmp/portage/binutils-2.17/work/binutils-2.17/ld/ldmain.c
/var/tmp/portage/binutils-2.17/work/binutils-2.17/ld/ldmain.c: In function 'main':
/var/tmp/portage/binutils-2.17/work/binutils-2.17/ld/ldmain.c:314: error: 'bfd_dma' undeclared (first use in this function)
/var/tmp/portage/binutils-2.17/work/binutils-2.17/ld/ldmain.c:314: error: (Each undeclared identifier is reported only once
/var/tmp/portage/binutils-2.17/work/binutils-2.17/ld/ldmain.c:314: error: for each function it appears in.)
/var/tmp/portage/binutils-2.17/work/binutils-2.17/ld/ldmain.c:314: error: expected ';' before 'DF_BIND_NOW'
/var/tmp/portage/binutils-2.17/work/binutils-2.17/ld/ldmain.c:315: error: expected ';' before 'DF_1_NOW'
make[4]: *** [ldmain.o] Error 1
make[4]: Leaving directory `/var/tmp/portage/binutils-2.17/work/build/ld'
make[3]: *** [all-recursive] Error 1
make[3]: Leaving directory `/var/tmp/portage/binutils-2.17/work/build/ld'
make[2]: *** [all] Error 2
make[2]: Leaving directory `/var/tmp/portage/binutils-2.17/work/build/ld'
make[1]: *** [all-ld] Error 2
make[1]: Leaving directory `/var/tmp/portage/binutils-2.17/work/build'
make: *** [all] Error 2

!!! ERROR: sys-devel/binutils-2.17 failed.
Call stack:
  ebuild.sh, line 1546:   Called dyn_compile
  ebuild.sh, line 937:   Called src_compile
  ebuild.sh, line 1255:   Called toolchain-binutils_src_compile
  toolchain-binutils.eclass, line 215:   Called die

!!! emake failed
!!! If you need support, post the topmost build error, and the call stack if relevant.

!!! This ebuild is from an overlay: '/master/svn/overlays.gentoo.org/svn/dev/kevquinn/hardened/toolchain'


binutils-2.17 broken as Peter Mazinger already told in a previous comment.


-Alex
Comment 17 Alexander Gabert (RETIRED) gentoo-dev 2006-10-07 07:32:03 UTC
Created attachment 99045 [details, diff]
portage-hardened-glibc-gcc.patch

emerge sync; cd /usr/portage; patch -p1 --dry-run < portage-hardened-glibc-gcc.patch


-Alex
Comment 18 Alexander Gabert (RETIRED) gentoo-dev 2006-10-07 08:21:12 UTC
Comment on attachment 99045 [details, diff]
portage-hardened-glibc-gcc.patch

this patch is missing some files from Kevins overlay.
Comment 19 Alexander Gabert (RETIRED) gentoo-dev 2006-10-07 09:50:39 UTC
Created attachment 99055 [details, diff]
portage-hardened-glibc-gcc.patch with needed files and ppc tryout
Comment 20 Alexander Gabert (RETIRED) gentoo-dev 2006-10-07 09:57:47 UTC
Kevin, is it possible to move the work of your patch from gcc/gcc.c (where the *_SPEC definitions are overriden) completely to the configure logic?

As far as i understood it, the "tm_file" in config.gcc merges the architecture specific pieces of specs with the global specs in gcc/gcc.c.

So maybe we can get rid of editing the direct gcc files altogether and add some stuff to the configure scripts.

Thanks,

Alex
Comment 21 Kevin F. Quinn (RETIRED) gentoo-dev 2006-10-07 10:54:05 UTC
Alex - I updated the binutils-2.17 earlier today, after psm noted the errors, although I haven't built it yet.  Obviously I put the wrong stuff into svn before...

Re. gcc -dumpspecs - it is defined (see gcc info) to show the built-in specs only, and changing that would be a bad idea, IMHO.  In toolchain-funcs.eclass you can find a simple bit of shell+awk that can parse specs.

I've just updated svn for my gcc style, which now builds hardened as it goes by setting GCC_SPECS for the build process.  It also creates the file "specs", which is unconditionally read by the compiler if present, and replaces the built-in specs.

At this point the difference for the user between my way and Peter's becomes ever smaller; the only real difference now (besides integration of the uclibc stuff, which I haven't put into my toolchain.eclass) is that Peter's has callouts for startfile/endfile, whereas I just supply completely new ones in my minispecs.  My intention there was to supply target-specific versions as necessary in the ebuild's files directory.

Lastly, I have not been able to reproduce the stack smash in ix86_split_to_parts() (bug #149292) - the patching to inhibit the stack protector on the code that smashes is commented out.
Comment 22 Kevin F. Quinn (RETIRED) gentoo-dev 2006-10-07 11:04:54 UTC
(In reply to comment #20)
> Kevin, is it possible to move the work of your patch from gcc/gcc.c (where the
> *_SPEC definitions are overriden) completely to the configure logic?

I'm not sure; it would be nice if we could.  Assuming we could just create (say) config/gentoo-hardened.h and add it to tm_file, so that it is parsed last in the list of target macro files, it would be possible provided we can append to existing macros.

You'll notice in my patch I define a new macro CC1_SPEC_2 to be CC1_SPEC appended with the callouts, and use that on the cc1 definition.  I originally tried to do this by doing things like:

#define PREV_CC1_SPEC CC1_SPEC
#undef CC1_SPEC
#define CC1_SPEC PREV_CC1_SPEC " %(cc1_ssp) %(cc1_pie) "

but macros don't work like that.  I couldn't find a way to do it.
Comment 23 Alexander Gabert (RETIRED) gentoo-dev 2006-10-07 13:48:27 UTC
yeah, bad news:

it would theoretically possible to manage the configure.in with --enable-autopie and --enable-autossp

and then check for it in config.gcc and add the config/hardened/hardened.h to the tm_file macro expansion list.

But this would mean a lot of code rework in the gcc source.  which results in a larger patch.

I will have to think about the whole thing for a bit.

But in the meantime i am happy with the patch as it is at the moment.


-Alex
Comment 24 Peter S. Mazinger 2006-10-07 16:42:25 UTC
the syntax %{pie:-pie;:%(anything)} is not valid on all gcc's, you have to
%{pie:-pie} %{!pie:%(anything)}
Comment 25 Peter S. Mazinger 2006-10-07 16:43:50 UTC
it is possible, but you will have to work harder to support uclibc as well
Comment 26 Peter S. Mazinger 2006-10-07 16:53:54 UTC
Alex, gcc -dumpspecs did return the builtin specs even in your times, it was
never returning the really used specs
Kevin, I have intentionally created startfile_pie/crtfile_pie/endfile_pie/asm_pie to get
rid of the fact that each arch has its own version of these sections, not speaking
of the fact, that some archs do not use the generic section names, they have
specific linux versions you have to overwrite too. The problem does not occur with
ssp, there we can append to CC1_SPEC.
Note: On x86 and x86_64 for gcc-4.1.1 my patch is broken (correct only for 4.1.0)
I have already created a new one, but I would update 9.0.4 by also removing
znow/zrelro from specs. Opinions?
Comment 27 Peter S. Mazinger 2006-10-07 16:58:33 UTC
to reproduce the ssp bug, use gcc-3.4.6 hardened profile and build a vanilla or
hardened gcc-4.1 (where DEFAULT_SSP & DEFAULT_PIE are set)
Comment 28 Alexander Gabert (RETIRED) gentoo-dev 2006-10-07 17:22:10 UTC
Peter: yeah, the ix86_split_to_parts error was also reproduced by me.

Tomorrow i will try to put together some "best parts of both worlds" patch which is orientated at Kevins approach of "minispecs" containing the emitters for the commandlines but at the same time taking care of Peters work for uclibc compatibility and multiple platform support (read: sparc and ppc).

Pijn, nixnut and Weedy (my favourite guinea pigs for test requests, along with the chroots on tinder and miranda) have been working the last two days testing a preliminary adapted version of Kevins glibc patches and gcc minispecs on x86, amd64 and today ppc, and so far things are looking good.

The versions of the ebuilds used for integration and testing are glibc-2.5 and gcc-4.1.1-r1.

There are some minor problems with the toolchain eclass regarding the message about switching to the new gcc profile and on ppc the ebuild is not copying over the specs files to the target directory of the emerged gcc-4.1.1-r1.

Also, the warning message with the aliased function appears on ppc, so maybe there is an #ifdef ppc missing in the code for the glibc gcc-3.* SSP compatibility patch.

Good job and good luck with the next steps of the migration,


Alex
Comment 29 Kevin F. Quinn (RETIRED) gentoo-dev 2006-10-08 03:04:29 UTC
(In reply to comment #24)
> the syntax %{pie:-pie;:%(anything)} is not valid on all gcc's, you have to
> %{pie:-pie} %{!pie:%(anything)}

I figured it would be valid for all versions 3.4.6 and above - is that not the case?
Comment 30 Peter S. Mazinger 2006-10-08 03:12:05 UTC
I would have used that syntax in my patches as well, cant recall now when it failed
Comment 31 Kevin F. Quinn (RETIRED) gentoo-dev 2006-10-08 04:43:57 UTC
(In reply to comment #26)
> Kevin, I have intentionally created
> startfile_pie/crtfile_pie/endfile_pie/asm_pie to get
> rid of the fact that each arch has its own version of these sections, not
> speaking
> of the fact, that some archs do not use the generic section names, they have
> specific linux versions you have to overwrite too. The problem does not occur
> with
> ssp, there we can append to CC1_SPEC.

Yeah; I see that.  I was going to deal with that by having different minispecs for each arch, as necessary, either held in ${FILESDIR}, or as a separate tarball, laid out so:

${FILESDIR}/specs/*.specs
 - most common minispecs
${FILESDIR}/specs/${PV}/*.specs
 - common minispecs for a specific GCC version
${FILESDIR}/specs/${ARCH}/*.specs
 - minispecs for a specific arch
${FILESDIR}/specs/${ARCH}/${PV}/*.specs
 - minispecs for a specific arch and GCC version

Then the installation of specs is simply to copy from each location in order, ensuring that the more specific versions overwrite the more generic versions.

The intention was to minimise as much as possible the need to patch gcc itself.  Thinking about it, it could all be done this way, and so have no patching of gcc itself at all - however it becomes clear that it would be easier to maintain the patches to gcc as Peter has proposed (not least to support uclibc as well).

> Note: On x86 and x86_64 for gcc-4.1.1 my patch is broken (correct only for
> 4.1.0)
> I have already created a new one, but I would update 9.0.4 by also removing
> znow/zrelro from specs. Opinions?

I need to think carefully about that.  We need to ensure the upgrade path works properly; I think it can be done by having DEPENDs conditional on USE=hardened, but it would involve creating stable rev-bumps of binutils, taking into consideration the various supported hardened arches.  My feeling at the moment is that it will be simpler to run with both simultaneously for a while, then drop them from gcc perhaps for 4.2 (by which time binutils with the changed defaults should be stable where necessary).
Comment 32 Peter S. Mazinger 2006-10-08 05:14:35 UTC
I am afraid, that if we do not patch gcc, a careless update of gcc won't catch
all the needed changes (patching gcc would fail at least maybe, telling that
something has to be adapted). Copying minispecs will always work, but won't be
compliant with each version, anyone updating gcc will have to carefully read
what was updated and we have to keep minispecs for different versions and archs
, my approach eliminates this need, we need only one set of minispecs for all
(even so 4.1.0->4.1.1 added ENDFILE_SPEC to i386/linux.h and linux64
 - this is the failure I have encountered - for 4.1.0 the ENDFILE_SPEC for these
2 configs came from the generic one, as result I ended up using crtend.o instead
of crtendS.o for PIE, resulting in fake TEXTREL
Kevin: have you tested if providing GCC_SPECS for the stage builds results in
building all the stages in gcc with pie/ssp enabled. If that is working (as I
stated I want to keep the possibility to compile gcc throughout with hardened)
then DEFAULT_PIE & SSP can be removed from the patches.
Comment 33 Kevin F. Quinn (RETIRED) gentoo-dev 2006-10-08 06:07:03 UTC
(In reply to comment #32)
> Copying minispecs will always work, but won't be
> compliant with each version, anyone updating gcc will have to carefully read
> what was updated and we have to keep minispecs for different versions and archs
> , my approach eliminates this need, we need only one set of minispecs for all

That's a good point. I was already coming to the conclusion that maintaining minispecs for an unpatched gcc would be more work than maintaining your patches - I guess that settles it :)

> Kevin: have you tested if providing GCC_SPECS for the stage builds results in
> building all the stages in gcc with pie/ssp enabled. If that is working (as I
> stated I want to keep the possibility to compile gcc throughout with hardened)
> then DEFAULT_PIE & SSP can be removed from the patches.


Yes, everything got built with ssp.  A sample:

# readelf -a ./gcc/stage1/tree-ssa-phiopt.o | grep stack

  [11] .note.GNU-stack   PROGBITS        00000000 0028f8 000000 00      0   0  1
0000048d  00002e04 R_386_PLT32       00000000   __stack_smash_handler
000009d3  00002e04 R_386_PLT32       00000000   __stack_smash_handler
00000aed  00002e04 R_386_PLT32       00000000   __stack_smash_handler
00001122  00002e04 R_386_PLT32       00000000   __stack_smash_handler
00001659  00002e04 R_386_PLT32       00000000   __stack_smash_handler
0000187f  00002e04 R_386_PLT32       00000000   __stack_smash_handler
000021e2  00002e04 R_386_PLT32       00000000   __stack_smash_handler
00002591  00002e04 R_386_PLT32       00000000   __stack_smash_handler
000025d8  00002e04 R_386_PLT32       00000000   __stack_smash_handler
    46: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND __stack_smash_handler

# readelf -a ./gcc/stageprofile/tree-ssa-phiopt.o | grep stack
  [30] .note.GNU-stack   PROGBITS        00000000 014dbf 000000 00      0   0  1
00000043  00001e02 R_386_PC32        00000000   __stack_chk_fail_local
00000185  00001e02 R_386_PC32        00000000   __stack_chk_fail_local
000007db  00001e02 R_386_PC32        00000000   __stack_chk_fail_local
00001461  00001e02 R_386_PC32        00000000   __stack_chk_fail_local
00002d41  00001e02 R_386_PC32        00000000   __stack_chk_fail_local
    30: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND __stack_chk_fail_local

# readelf -a ./gcc/tree-ssa-phiopt.o | grep stack
  [30] .note.GNU-stack   PROGBITS        00000000 0126c6 000000 00      0   0  1
00000209  00001a02 R_386_PC32        00000000   __stack_chk_fail_local
00000001  00001a02 R_386_PC32        00000000   __stack_chk_fail_local
00000036  00001a02 R_386_PC32        00000000   __stack_chk_fail_local
00000109  00001a02 R_386_PC32        00000000   __stack_chk_fail_local
    26: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND __stack_chk_fail_local


(the different number of calls is down to differences in the way the optimisation varied; there are fewer functions in the final 4.1 object - I've checked the object disassembly and all functions are protected, in all cases).

In toolchain.eclass, I installed the minispecs into the build area, concatenated them to create "${WORKDIR}"/build.specs which I set in GCC_SPECS:

concat_minispecs() {
    local entry spec
    while read entry; do
        spec=${entry/\%include <}
        if [[ ${spec} == ${entry} ]]; then
            printf "%s\n" "$entry" >> $2
        else
            spec=${spec/>}
            cat "${FILESDIR}"/specs/${spec} >> $2
        fi
    done < ${FILESDIR}/specs/${1}.specs
}
...
gcc_src_compile() {
    gcc_do_filter_flags
    einfo "CFLAGS=\"${CFLAGS}\""
    einfo "CXXFLAGS=\"${CXXFLAGS}\""

    if use hardened ; then
        if hardened_gcc_works ; then
            concat_minispecs hardened "${WORKDIR}"/build.specs
        elif hardened_gcc_works pie ; then
            concat_minispecs hardenednossp "${WORKDIR}"/build.specs
        elif hardened_gcc_works ssp ; then
            concat_minispecs hardenednopie "${WORKDIR}"/build.specs
        else
            concat_minispecs hardenednopiessp "${WORKDIR}"/build.specs
        fi
        export GCC_SPECS="${WORKDIR}"/build.specs
    fi
...

This was where 'hardened.specs' etc contain just '%include <xxx>' references to the other minispecs.  Note; %include is not permitted in built-in specs, nor in the file /usr/lib/gcc/i686-pc-linux-gnu/4.1.1/specs etc, but it's fine in anything set in GCC_SPECS.
Comment 34 Alexander Gabert (RETIRED) gentoo-dev 2006-10-08 07:05:11 UTC
okay, i think the best thing would be to take the best from both solutions:

1) gcc patch: adding minispecs for callouts to PIE and SSP logic

the only "top level" callout added in gcc/gcc.c should be the callout for SSP in CC1_SPECS

the patch for adding the top level callout to CC1_SPECS can be called gcc-ssp-callout.patch and kept minimal intrusive.

the rest of the PIE callouts should ONLY be added arch dependent in the respective subdirectories of config/<arch>/<arch>.h

2) gcc patch: branding the version to show a hardened flavour

the patch can be named gcc-hardened-branding.patch

3) toolchain eclass: manipulating the GCC_SPECS env var
4) toolchain eclass: adding additional specs to GCC_SPECS list for callouts

The toolchain eclass will create corresponding gcc-config profiles and the correct GCC_SPECS environment variables containing a list of specs files in the order of priority.

The specs files for resolving the callouts will resemble the behaviour, either nondefault or default hardened PIE SSP.

This is managed by setting the appropriate GCC_SPECS env var with the gcc-config profile.


The respective patches for the SSP callout in the gcc/ top level dir and PIE callouts in the arch dependent sub directories of the gcc/config/ dir can be triggered depending on

- hardened use flag
- libc version (do we have a glibc use flag?)
- architecture use flag

if use hardened && use glibc; then
    epatch XXX/gcc-4.1.1-ssp-noarch-glibc-callout.patch

    epatch XXX/gcc-hardened-noarch-glibc-branding.patch

    if use x86; then
        epatch XXX/gcc-pie-x86-glibc-callout.patch
    else if use amd64; then
        epatch XXX/gcc-pie-x86-glibc-callout.patch
    fi
fi




I would like to have a meeting in #gentoo-hardened-dev at the beginning of next week to coordinate the efforts of our team because i am getting the feeling there is a bit of a lack of communication here.

I will work out some test patches resembling this scheme now.


Thanks,

Alex
Comment 35 Alexander Gabert (RETIRED) gentoo-dev 2006-10-08 07:17:31 UTC
as i can see from the eclass, branding is done without a special hardened patch applied.

but at least these parts of the original toolchain eclass have to be rewritten because the concept is undergoing changes with the minispecs approach:

        # corrects startfile/endfile selection and shared/static/pie flag usage
        EPATCH_MULTI_MSG="Applying upstream pie patches ..." \
        epatch "${WORKDIR}"/piepatch/upstream
        # adds non-default pie support (rs6000)
        EPATCH_MULTI_MSG="Applying non-default pie patches ..." \
        epatch "${WORKDIR}"/piepatch/nondef
        # adds default pie support (rs6000 too) if DEFAULT_PIE[_SSP] is defined
        EPATCH_MULTI_MSG="Applying default pie patches ..." \
        epatch "${WORKDIR}"/piepatch/def
Comment 36 Kevin F. Quinn (RETIRED) gentoo-dev 2006-10-08 08:37:00 UTC
One other issue, is -fno-stack-protector-all.  Since it's a '-f' switch, it gets rejected by gcc-4.

We could add it to gcc-4, but I would prefer to avoid that.

I suggest we do something like this:

cc1_ssp: %{!nostdlib:%{!nodefaultlibs:%{!fno-stack-protector:%{!fstack-protector:%{!fstack-protector-all:-fstack-protector %(cc1_ssp_all) }}}}}

cc1_ssp_all: %{!fno-stack-protector-all:-fstack-protector-all}

The '-fno-stack-protector-all' in cc1_ssp_all is ineffective but causes no problems on gcc-4.  The adjustment to cc1_ssp ensure that any requested setting of stack-protector does not get overridden by the specs.


With regards to glibc - I intend either to get it building ok with ssp-all, or to adjust the glibc build process to switch off ssp where it needs to.  Thus we should be able to drop the D_LIBC/D_LIBC_REENTRANT conditions.
Comment 37 Kevin F. Quinn (RETIRED) gentoo-dev 2006-10-08 10:22:53 UTC
(In reply to comment #34)
> I would like to have a meeting in #gentoo-hardened-dev at the beginning of next
> week to coordinate the efforts of our team because i am getting the feeling
> there is a bit of a lack of communication here.

I feel we're progressing just fine.  We've explored various possibilities, which has helped us understand things better - discarded some ideas, developed others.  We've decided we'll go with Peter's style of patches, hopefully with the GCC_SPECS approach to building if that works out ok.

I guess, Alex, you're perhaps at a bit of a disadvantage.  Peter and I came up with the minispecs idea some months ago; we've worked at it independently on and off for some time now so we understand a lot about the issues it raises.  Perhaps you could put together a list of what you think should be discussed at the meeting (a meeting agenda, if you like).  I'd prefer to deal with that here rather than IRC, though.

Lastly, I'm not going to rush this.  I've mentioned to people before that we hope to have stuff in a decent shape in the tree by the end of the year.  The switch from gcc-3/glib-2.3.6 to gcc-4/glibc-2.4 is a time to introduce significant changes, which is what we're considering, and I don't want to be changing things significantly again after that.
Comment 38 Alexander Gabert (RETIRED) gentoo-dev 2006-10-08 10:31:01 UTC
vapier says we should work on glibc-2.5

glibc-2.4 is fixed according his plans (see gentoo-dev ML)


Alex
Comment 39 Peter S. Mazinger 2006-10-08 11:41:08 UTC
Alex, I am not in a hurry either and don't want to rush ...
I am using these changes since about May, but I am not quite happy with it
uClibc has support for gcc4-style ssp only in svn, the glibc implementation
(haven't looked though lately because I do not have any glibc based box running)
probably needs also some changes.
Kevin: I will do some testing with your GCC_SPECS use in eclass, if that goes
fine, I can remove all the DEFAULT_* from my patches and leave the handling
exclusively to the eclass. Do you need earlier the changes needed for x86[_64]?
Comment 40 Peter S. Mazinger 2006-10-08 12:18:38 UTC
Kevin: I diffed your toolchain.eclass, you missed a '\' after -n ${PIE_VER}
I have added -DTARGET_LIBC_PROVIDES_SSP to *_CFLAGS to be sure that I have it
the configure check does a simplistic check for glibc-2.4 (checking for
GLIBC_MAJOR/MINOR that I didnt liked and I am not sure how good that will be
in a cross-compiled env. I have attempted to discuss this on gcc-ml to generalize
the check (really doing a check in ld.so for __stack_chk_guard and looking for
__stack_chk_fail in libc, haven't gotten any answer from there.
I recall having some problem with the sed expression editing LIBGCC2_CFLAGS
because the line breaks and -fno-stack-protector was appended after the trailing
'\'
Maybe someone else could also try to provide the piepatches/upstream stuff to
the gcc list, less one of them they are real corrections for the case HAVE_LD_PIE
is not defined - true only for really old binutils - , but were discarded by JJ,
saying that 1-2 cycles more in processing do not matter.
Does anyone know if it is expected to have boundschecking patches for gcc-4.1?

by JJ,
Comment 41 Alexander Gabert (RETIRED) gentoo-dev 2006-10-08 13:18:29 UTC
I wanted the meeting because i'm lost in your ideas and don't know how we can solve the problems together.  My goal is to get 2007.0 hardened stages and a livecd perhaps.  But for this we need to get the stuff in late october or at least november to have enough time for the ~arch -> arch move.

As far as it goes for my research, changes to the following sections can be outsourced to callouts to specially named sections in external specs files:

cc1_specs (needs stuff for SSP and PIE added)
asm_specs (needs stuff for PIE added on some arches like sparc and ppc)
link_pie_specs (needs stuff for PIE added but leaving the original intact when we carefully craft the trigger list with respect to the existing %{pie:-pie})

because of the complexity of startfile and endfile we need to completely replace startfile and endfile for the given situations:
- different arches
- different libc types
- different model (default PIE, nondefault PIE)

Now,my idea is to tackle it this way:

adding a header framework to tm_file:

gcc/config.gcc:tm_file="gentoo-hardened/hardened.h ${cpu_type}/${cpu_type}.h"

then adding the "empty" definition fallbacks to gcc.c and the sections prepared for overwriting

gcc/gcc.c:

/* Gentoo Hardened appends extra args to the ASM_SPECS */
#ifndef GENTOO_HARDENED_ASM_SPEC
#define GENTOO_HARDENED_ASM_SPEC ""
#endif

/* Gentoo Hardened appends extra args to the CC1_SPECS */
#ifndef GENTOO_HARDENED_CC1_SPEC
#define GENTOO_HARDENED_CC1_SPEC ""                                                                           #endif

/* Gentoo Hardened definitions may overwrite the default specs */
#ifdef GENTOO_HARDENED_STARTFILE_SPEC
#undef STARTFILE_SPEC
#define STARTFILE_SPEC GENTOO_HARDENED_STARTFILE_SPEC
#endif

/* Gentoo Hardened definitions may overwrite the default specs */                                             #ifdef GENTOO_HARDENED_ENDFILE_SPEC
#undef ENDFILE_SPEC
#define ENDFILE_SPEC GENTOO_HARDENED_ENDFILE_SPEC                                                             #endif

/* Gentoo Hardened definitions appended to LINK_PIE if LD_PIE is available */
#ifndef HAVE_LD_PIE
#undef GENTOO_HARDENED_LINK_PIE_SPEC
#define GENTOO_HARDENED_LINK_PIE_SPEC ""
#else
#ifndef GENTOO_HARDENED_LINK_PIE_SPEC
#define GENTOO_HARDENED_LINK_PIE_SPEC ""
#endif
#endif

#ifndef LINK_COMMAND_SPEC
#define LINK_COMMAND_SPEC "\
%{!fsyntax-only:%{!c:%{!M:%{!MM:%{!E:%{!S:\
    %(linker) %l " LINK_PIE_SPEC GENTOO_HARDENED_LINK_PIE_SPEC " %X %{o*} %{A} %{d} %{e*} %{m} %{N} %{n} %{r}\
    %{s} %{t} %{u*} %{x} %{z} %{Z} %{!A:%{!nostdlib:%{!nostartfiles:%S}}}\
    %{static:} %{L*} %(mfwrap) %(link_libgcc) %o %(mflib)\
    %{fprofile-arcs|fprofile-generate|coverage:-lgcov}\
    %{!nostdlib:%{!nodefaultlibs:%(link_ssp) %(link_gcc_c_sequence)}}\
    %{!A:%{!nostdlib:%{!nostartfiles:%E}}} %{T*} }}}}}}"
#endif

static const char *cc1_spec = CC1_SPEC " " GENTOO_HARDENED_CC1_SPEC;
static const char *cc1plus_spec = CC1PLUS_SPEC;
static const char *link_gcc_c_sequence_spec = LINK_GCC_C_SEQUENCE_SPEC;
static const char *link_ssp_spec = LINK_SSP_SPEC;
static const char *asm_spec = ASM_SPEC " " GENTOO_HARDENED_ASM_SPEC;

The rest of the work is done by either defining the DEFINES or letting it undefined.

This is keeping the intrusive parts out of gcc.c and move the logic to a separate directory of header files responsible for setting up the exclude triggers, deciding about the arches and the libc needed.

Hope you like the idea and it will help make things clearer for all of us.


Alex
Comment 42 Peter S. Mazinger 2006-10-08 13:35:42 UTC
Alex, it seems you were not listening ;-)
Kevin and me agreed upon having only one set of minispecs for all archs. That
means, that gcc.c will get added new sections (minimal possible list: asm_pie_spec,
cc1_hardened_spec - covering pie and ssp -, crtfile_pie_spec, endfile_pie_spec,
startfile_pie_spec and link_hardened_spec - covering pie, relro and now). If you
do not go that route, you have to provide for each arch separate minispecs and
probably with each gcc release you will have to regenerate at least some of them
Comment 43 Kevin F. Quinn (RETIRED) gentoo-dev 2006-10-08 15:07:40 UTC
Alex - re glibc-2.5, that's too far in the future for the moment.  I'm assuming that vapier is talking about non-hardened use; I don't think he'll object if we put stuff into glibc-2.4 for hardened use, provided we don't break things for anyone else.  Worst case is that we won't be able to have ssp-built glibc-2.4 in the tree, since it may involve quite a bit of patching to the build system.

Re your ideas about specs; best thing for you to do is to implement it and see how it works in practice, then compare it to what we're doing to see if it's better or not.  Probably my efforts were a bit confusing as I didn't get as far as sorting out startfile/endfile properly.  As Peter says one of the core concepts is that the minispecs are generic - it's just where and how they get called that can be target-specific.


Peter - re x86[_64] changes, I'm happy to wait while you try out the GCC_SPECS thing.  I'll focus on glibc in the meantime.

I saw the problem with sed'ing LIBGCC2_CFLAGS - later (4.x) gcc's have the definition in gcc/Makefile.in split over more than one line, and as you note the existing sed puts '-fno-stack-protector' on the end of the line - after the '\', which messes things up :/  I fixed it by putting the -fno-stack-protector as the first thing after the ' = ' - I think you've fixed it pretty much the same way.

On the missing '\' - FWIW it's not necessary after '&&' (or '||') as a command conjunction; only time you need it after '&&' is if '&&' is part of a [[ ... ]] phrase; so:

  [[ -n ${PIE_VER} ]] &&
     PIE_CORE=...

and

  if [[ ${GCCMAJOR} == 3 ]] &&
     [[ ${GCCMINOR} -ge 3 ]]
  then

are ok (putting the trailing '\' in doesn't hurt, though), but

  if [[ -n ${PIE_VER} &&
      -z ${PIE_CORE} ]]; then

is broken.
Comment 44 Alexander Gabert (RETIRED) gentoo-dev 2006-10-08 15:09:54 UTC
Peter, i'm reading this bug since it was created and all i can read is two sides of one story from which i don't understand either side of it.

The only things i am understanding is:

- we should minimize the amount of gcc patching
- there should be a way to provide for automatic/nonautomatic PIE SSP building
- the support for nonglibc versions and arches has to be improved/cleaned
- the existing macros cannot be redefined without losing the old information

So the question for me remains: where to put the information about the arches.

At the moment it's like this:

A directory structure in the pie-patches tarball controls information about general patches for all architectures and architecture-specific patches which dive into subdirs of gcc/config and apply to local architecture-specific files.

Previously this information has been stored in a complicated "generator" header-file which directly applied the information to the specs definitions.


Now you are telling me you want to make a "mini specs" for all arches the same, which makes no sense to me because it just can't work.

The reason is that x86 for example doesn't use a modified -K PIC switch in the asm section while sparc and ppc do.  While hppa does not use SSP, it makes use of PIE in the usual ways.

Simply put: i don't get the idea.


Alex
Comment 45 Peter S. Mazinger 2006-10-08 15:48:41 UTC
Alex, read my 9.0.3 patches nondef directory, it will show you what I do
for ex: #define ASM_SPEC <original blah> %(asm_pie)
I create additional asm_pie section (but only the affected archs ASM_SPEC uses
asm_pie), so if I make *asm_pie: -K PIC generally, it will only affect those archs
where, %(asm_pie) was added. This way I am sure that I get all the original info
from ASM_SPEC and add my own one. An affected arch will show you in its *asm: 
<original blah> %(asm_pie), you only have to set asm_pie in minispecs
Why I dont want separate minispecs:
- we have 2 sorts of crt parts in startfile, with and without profile
- we have 2 sorts of crtbegin* users, those who have crtbeginT.o for static, others
using crtbegin.o for static
- we have at least 2 sorts of endfile* sections, with and without fastmath
- we have archs that do not use startfile/endfile, they have linux specific
naming
- we have archs that need for PIC/PIE asm section modifications
Comment 46 Peter S. Mazinger 2006-10-08 15:55:36 UTC
Note: I have even checked with the gcc devs the possibility of (cant recall exact
syntax): if-exist crtbeginT.o else crtbegin.o, they said that this will have heavy
performance issues
Comment 47 Alexander Gabert (RETIRED) gentoo-dev 2006-10-08 16:02:41 UTC
okay, so if i understand you right:

you keep modifying the same locations in gcc sourcecode with patches, but this time you don't enter the direct information, you only put a %(XXX) callout to a minispec file into it and let the real implementation be handled by the minispecs file taking care of the problems.

And the minispec is generated with respect to the arch and libc it is running on.

Am i misunderstanding something or something missing?


Alex
Comment 48 Peter S. Mazinger 2006-10-08 16:17:24 UTC
I provide for each newly created sections defaults as vanilla gcc would behave
(lets assume for a moment that hardened built of gcc is not used), so unless
you do not provide any file to GCC_SPECS, you have a vanilla gcc, but you get
the possibility to provide 1-n files in GCC_SPECS, each having some sections
redefined:
default *link_relro: %{norelro:}, but you can provide a minispec having
*link_relro: %{!norelro: -z relro} and that gets added to link_command, because
*link_command is defined as: ... %(link_relro) ...
Comment 49 Alexander Gabert (RETIRED) gentoo-dev 2006-10-08 16:29:44 UTC
okay, sounds good.


Alex
Comment 50 Peter S. Mazinger 2006-10-08 16:40:41 UTC
Kevin: does the default specs file support having 2 sections with the same name
where the last one wins? wouldnt it be easier to leave the default specs alone
and only default GCC_SPECS to the proper one, depending on use hardened or not?
I think the procedure setting GCC_SPECS in toolchain.eclass to build.specs will
cause some trouble, if we have an old compiler without the new sections, the
compiler will either bail out due to unknown section, or it wont do anything
Comment 51 Alexander Gabert (RETIRED) gentoo-dev 2006-10-08 17:18:00 UTC
Created attachment 99170 [details, diff]
gcc-hardened-callouts.patch

This patch does multiple things.

If gcc/config/gentoo-hardened/hardened.h is an empty file, it makes gcc behave like a normal gcc without any callouts and changes.

If hardened.h contains the mentioned definitions of the callouts, these are added to the "safe" sections of specs where we can append or insert our callout handler without doing harm to the section itself.

The handling of startfile and endfile is fully left to the additional minispecs in GCC_SPECS env variable.

If no such env variable is present, the gcc behaves like a normal one, even though compiled with the "hooks" to trigger hardened behaviour on demand.

If it finds the env var, the additional specs file is supposed to:
a) implement the sections of the callouts embedded in the original sections
b) overwrite startfile specs section
c) overwrite endfile specs section

if the additional specs file only implements SSP, it can skip b) and c) and only implement the cc1_hardened_ssp section for example.

if the additonal specs file implements only PIE, it can skip parts of a) but must do b) and c) according to Peters logic he explained previously.


Thanks for helping me understand your patches,


Alex
Comment 52 Alexander Gabert (RETIRED) gentoo-dev 2006-10-08 17:24:06 UTC
oh, and what i forgot.

This patch also makes it easy to go both routes:

a PIE SSP patch can either

- directly patch up the GENTOO_HARDENED_STARTFILE_SPEC inside the hardened.h header in the config/gentoo-hardened directory

- or use the new startfile section in the specs file available via GCC_SPECS env var

It is also safe to have a patch add a customized GENTOO_HARDENED_STARTFILE_SPEC and a GENTOO_HARDENED_ENDFILE_SPEC via patching hardened.h in the ebuild or toolchain eclass (for example an improved nondefault PIE version for uclibc or glibc) and then add a specs file to the ENV var which does DEFAULT PIE in the startfile and endfile and fills out the handlers for the callouts with the respective action (like triggering -pie automatic).

I hope you understand what i mean :)


Alex
Comment 53 Alexander Gabert (RETIRED) gentoo-dev 2006-10-08 17:42:18 UTC
first testing shows it works:

 # cat /tmp/ssp.specs
*cc1_hardened_ssp:
%{!fno-stack-protector: %{!fno-stack-protector-all: %{!fstack-protector: %{!fstack-protector-all: -fstack-protector-all} } } }

 # GCC_SPECS="/tmp/ssp.specs" ./gcc/xgcc -o vuln-stack vuln-stack.c ; ./vuln-stack AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
*** stack smashing detected ***: vuln-stack terminated
Killed
 02:41:04 [/space/chroots/chroot001:6690.pts-2.miranda]miranda /var/tmp/portage/gcc-4.1.1-r1.TESTBED/work/build
 # GCC_SPECS="" ./gcc/xgcc -o vuln-stack vuln-stack.c ; ./vuln-stack AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Segmentation fault
 02:41:10 [/space/chroots/chroot001:6690.pts-2.miranda]miranda /var/tmp/portage/gcc-4.1.1-r1.TESTBED/work/build
 # GCC_SPECS="/tmp/ssp.specs" ./gcc/xgcc -o vuln-stack vuln-stack.c ; ./vuln-stack AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
*** stack smashing detected ***: vuln-stack terminated
Killed

 # GCC_SPECS="/tmp/ssp.specs" ./gcc/xgcc -v -o vuln-stack vuln-stack.c ; ./vuln-stack AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Using built-in specs.
Reading specs from /tmp/ssp.specs
Target: x86_64-pc-linux-gnu
Configured with: /var/tmp/portage/gcc-4.1.1-r1.TESTBED/work/gcc-4.1.1/configure --prefix=/usr --bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/4.1.1 --includedir=/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/include --datadir=/usr/share/gcc-data/x86_64-pc-linux-gnu/4.1.1 --mandir=/usr/share/gcc-data/x86_64-pc-linux-gnu/4.1.1/man --infodir=/usr/share/gcc-data/x86_64-pc-linux-gnu/4.1.1/info --with-gxx-include-dir=/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/include/g++-v4 --host=x86_64-pc-linux-gnu --build=x86_64-pc-linux-gnu --disable-altivec --enable-nls --without-included-gettext --with-system-zlib --disable-checking --disable-werror --disable-libunwind-exceptions --disable-multilib --disable-libmudflap --disable-libssp --disable-libgcj --enable-languages=c,c++ --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu
Thread model: posix
gcc version 4.1.1 (Gentoo 4.1.1-r1)
 /usr/libexec/gcc/x86_64-pc-linux-gnu/4.1.1/cc1 -quiet -v -iprefix /var/tmp/portage/gcc-4.1.1-r1.TESTBED/work/build/gcc/../../../lib/gcc/x86_64-pc-linux-gnu/4.1.1/ vuln-stack.c -fstack-protector-all -quiet -dumpbase vuln-stack.c -mtune=k8 -auxbase vuln-stack -version -o /tmp/ccT1OWUN.s
ignoring nonexistent directory "/var/tmp/portage/gcc-4.1.1-r1.TESTBED/work/build/gcc/../../../lib/gcc/x86_64-pc-linux-gnu/4.1.1/include"
ignoring nonexistent directory "/var/tmp/portage/gcc-4.1.1-r1.TESTBED/work/build/gcc/../../../lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../../../x86_64-pc-linux-gnu/include"
ignoring nonexistent directory "/usr/local/include"
ignoring nonexistent directory "/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../../../x86_64-pc-linux-gnu/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/include
 /usr/include
End of search list.
GNU C version 4.1.1 (Gentoo 4.1.1) (x86_64-pc-linux-gnu)
        compiled by GNU C version 4.1.1 (Gentoo 4.1.1).
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
Compiler executable checksum: 54533a42eb6b472ac36fa421b5271c16
 /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../../../x86_64-pc-linux-gnu/bin/as -V -Qy -o /tmp/cc2OGymt.o /tmp/ccT1OWUN.s
GNU assembler version 2.16.1 (x86_64-pc-linux-gnu) using BFD version 2.16.1
 /usr/libexec/gcc/x86_64-pc-linux-gnu/4.1.1/collect2 --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o vuln-stack /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../../../lib64/crt1.o /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../../../lib64/crti.o /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/crtbegin.o -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../../../x86_64-pc-linux-gnu/lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../.. /tmp/cc2OGymt.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/crtend.o /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../../../lib64/crtn.o
*** stack smashing detected ***: vuln-stack terminated
Killed
Comment 54 Peter S. Mazinger 2006-10-08 17:48:41 UTC
Alex, I have no idea what your intention with this patch is, but it seems you
should reread the patch piepatch/nondef/00* and explain to me how your proposed
changes will improve that one.
Comment 55 Kevin F. Quinn (RETIRED) gentoo-dev 2006-10-09 00:22:49 UTC
(In reply to comment #50)
> Kevin: does the default specs file support having 2 sections with the same
> name where the last one wins?

Yes.  Later specs definitions override previous ones.

> wouldnt it be easier to leave the default specs alone
> and only default GCC_SPECS to the proper one, depending on use hardened or not?

> I think the procedure setting GCC_SPECS in toolchain.eclass to build.specs will
> cause some trouble, if we have an old compiler without the new sections, the
> compiler will either bail out due to unknown section, or it wont do anything

Yes; in that case it won't do anything.

In my mods to toolchain.eclass, I have this conditional on the new specs arrangement, so that older compiler versions are unaffected.  I had done that by setting another variable in the ebuild, but the easiest way would be to do it conditional on the PIE_VER, so that if PIE_VER is >=9.0 we do the new stuff, if it's <9 we do things the old way.

Note that if you're building from hardened 3.4.6-r1 (or earlier), the existing default specs file (/usr/lib/gcc/i686-pc-linux-gnu/3.4.6/specs) is always read, so will be hardened even if GCC_SPECS is set to build.specs (in that case the contents of build.specs make no difference).

It is also useful to create ${LIBPATH}/specs for the final installation - I do that in my toolchain.eclass at the end of gcc-compiler_src_install().  This means that the compiler is hardened even in an empty environment (i.e. GCC_SPECS unset).
Comment 56 Alexander Gabert (RETIRED) gentoo-dev 2006-10-09 02:49:08 UTC
(In reply to comment #54)
> Alex, I have no idea what your intention with this patch is, but it seems you
> should reread the patch piepatch/nondef/00* and explain to me how your proposed
> changes will improve that one.
> 

Hey Peter,

the only thing i am intending is that i was working out an idea because i didn't understand the setup of your patch.

As i said, to me it makes more sense to change startfile/endfile in a central location instead of in the subdirectories of the arches.

And the rest of the patch resembles your and Kevins idea about modularizing the specs into an external file.

The only thing i changed was to modifications to gcc.c and adding an own config dir for the hardened work which looks more transparent to me.

This whole thing is not meant to replace your work or make you look bad, it's just a way of opening up my mind into another perspective.

Thanks,

Alex
Comment 57 Peter S. Mazinger 2006-10-09 03:37:14 UTC
To me it does not make sense to have a central location to change startfile/endfile
due to the variety of them mentioned in another post. You could search the gcc-ml
for a proposed patch that would have done the generalization, it was unacceptable
Comment 58 Kevin F. Quinn (RETIRED) gentoo-dev 2006-10-09 04:21:05 UTC
(In reply to comment #39)
> Kevin: I will do some testing with your GCC_SPECS use in eclass, if that goes
> fine, I can remove all the DEFAULT_* from my patches and leave the handling
> exclusively to the eclass. Do you need earlier the changes needed for x86[_64]?

Actually, I've changed my mind :) Could you provide the differences for 4.1.1?

I've just committed a version of toolchain.eclass to my SVN area that I think should work with the GCC_SPECS method of building gcc hardened and your patchset once you've removed the EFAULT_XXX bits (also includes the uclibc changes).  Probably clearer than some of my explanations :)  Obviously it's untested as it needs your updated patchset.
Comment 59 Peter S. Mazinger 2006-10-09 05:06:11 UTC
I think want_pie should be rethought, we need to apply even with minispecs the
piepatches, want_ssp does not make sense for gcc_has_native_ssp, I would maybe use
_create_specs_file() even for minispecs to generate for now from pie,ssp,sspall,
znow,zrelro.specs the ones we are already used (hardened[nopiessp])
Comment 60 Peter S. Mazinger 2006-10-09 06:15:15 UTC
Created attachment 99192 [details, diff]
correction for i386 endfile handling, goes to upstream dir
Comment 61 Peter S. Mazinger 2006-10-09 06:17:42 UTC
Created attachment 99193 [details, diff]
gcc-4.1.1 correction for i386 endfile handling, goes to nondef dir
Comment 62 Kevin F. Quinn (RETIRED) gentoo-dev 2006-10-09 06:26:48 UTC
I think my current toolchain.eclass does apply piepatches for minispecs:

want_pie() { _want_stuff PIE_VER !nopie ; }
(unchanged from original - this seems ok as with minispecs we'll still be supplying a piepatches tarball)

want_ssp() { gcc_has_native_ssp || _want_stuff PP_VER !nossp ; }
(return TRUE if gcc has native ssp, from your toolchain.eclass)

want_minispecs() {
    version_is_at_least "9.0" ${PIE_VER}
}
(new, to differentiate between old-style piepatches and minispec piepatches)

want_split_specs() {
    [[ ${SPLIT_SPECS} == "true" ]] && want_pie
}
(unchanged from original)

I've skipped make_gcc_hard & split_out_specs_files when want_minispecs() is true, and simply installed the specs directly in do_gcc_PIE_patches (to $WORKDIR) and gcc-compiler_src_install (from ${WORKDIR} to ${D}).  This also means that the older ebuilds continue to work.

do_gcc_PIE_patches() {
...
    if want_minispecs ; then
        # Install minispecs to working directory, where they can be used
        # in the compile/install phases.
        mkdir "${WORKDIR}"/specs ||
            die "failed to make working specs directory"
        cp -f ${FILESDIR}/specs/*specs "${WORKDIR}"/specs/ ||
            die "failed to install minispecs to working specs directory"
    fi
...
(which then continues to install the piepatches as before)

gcc-compiler_src_install() {
...
        insinto ${LIBPATH}
        if want_minispecs ; then
            doins "${WORKDIR}"/specs/*.specs || die "failed to install minispecs"
        else
            doins "${WORKDIR}"/build/*.specs || die "failed to install specs"
        fi
...


There's the generation of the system specs for GCC_SPECS at the start or gcc_src_compile:

    if use hardened && want_minispecs ; then
        if hardened_gcc_works ; then
            concat_minispecs hardened "${WORKDIR}"/build.specs
        elif hardened_gcc_works pie ; then
            concat_minispecs hardenednossp "${WORKDIR}"/build.specs
        elif hardened_gcc_works ssp ; then
            concat_minispecs hardenednopie "${WORKDIR}"/build.specs
        else
            concat_minispecs hardenednopiessp "${WORKDIR}"/build.specs
        fi
        export GCC_SPECS="${WORKDIR}"/build.specs
    fi

and the generation of the installed system spec file at the end of gcc-compiler_src_install:

...
    # Build system specs file which, if it exists, must be a complete set of
    # specs as it completely and unconditionally overrides the builtin specs.
    if use hardened && want_minispecs ; then
        "${WORKDIR}"/build/gcc/xgcc -dumpspecs > "${WORKDIR}"/build/specs
        if hardened_gcc_works ; then
            for s in pie ssp sspall znow zrelro; do
                cat "${WORKDIR}"/specs/${s}.specs >> "${WORKDIR}"/build/specs
            done
        elif hardened_gcc_works pie ; then
            for s in pie znow zrelro; do
                cat "${WORKDIR}"/specs/${s}.specs >> "${WORKDIR}"/build/specs
            done
        elif hardened_gcc_works ssp ; then
            for s in ssp sspall znow zrelro; do
                cat "${WORKDIR}"/specs/${s}.specs >> "${WORKDIR}"/build/specs
            done
        else
            for s in znow zrelro; do
                cat "${WORKDIR}"/specs/${s}.specs >> "${WORKDIR}"/build/specs
            done
        fi
        insinto ${LIBPATH}
        doins "${WORKDIR}"/build/specs || die "Failed to install system specs"
    fi
}


I think that's enough, and I think it doesn't break the older or existing versions.  One item on my TODO is perhaps to hard-code the names of the minispecs, rather than relying on ${FILESDIR}/specs/*.specs to find them.

I'll try it out with the updated 4.1.1 files.
Comment 63 Peter S. Mazinger 2006-10-09 08:12:41 UTC
Created attachment 99204 [details, diff]
toolchain diff

Kevin, sorry that I made you copy-paste your toolchain.eclass, I have read it
myself, attached is with what I have rebuilt gcc-4.1.1 (using gcc-4.1.1 already
patched for minispecs
Comment 64 Peter S. Mazinger 2006-10-09 10:13:44 UTC
Can we do anything to get -dumpspecs provide the proper results with the
finally generated specs?
Comment 65 Kevin F. Quinn (RETIRED) gentoo-dev 2006-10-09 10:17:10 UTC
No problem - doesn't hurt to have an explanation of some of the new pieces anyway.  I've aligned with your diff; the only changes I've made is to use concat_minispec when creating specs files (both in _create_specs_file and  gcc-compiler_src_install) to generate minispecs from "microspecs".  Saves some duplication of text in the mini/microspecs files, yet at the same time creates single specs files in the installed system (eliminating the multiple file opens that Alex mentioned before).  In svn revision 30 (including my '%include' minispecs and the microspecs you supplied before, plus some no* microspecs for the vanilla compiler on hardened systems.).
Comment 66 Kevin F. Quinn (RETIRED) gentoo-dev 2006-10-09 10:30:16 UTC
(In reply to comment #64)
> Can we do anything to get -dumpspecs provide the proper results with the
> finally generated specs?

Well, we could go patching what -dumpspecs does, but I don't think it's a good idea.  It moves us further from upstream when we don't need to, and means you can't get the builtin specs without unsetting GCC_SPECS and more disruptively deleting ${LIBPATH}/specs.  -dumpspecs has _always_ provided the builtin specs, regardless of whether there's a ${LIBPATH}/specs file or any -specs directives on the compiler invocation.  It's designed to tell you about the compiler driver, not what specs the compilation will actually use.

I've already dealt with it in toolchain-funcs.eclass some time ago. See the 'gcc-specs-directive' function; it's simple enough to do.  The function just 'cat's the built-in specs and all the listed specs in order as reported by 'gcc -v', retaining just the last full definition with any subsequent appends for the requested directive.
Comment 67 Peter S. Mazinger 2006-10-09 10:44:09 UTC
thanks doing the concat_minispecs part. I was thinking of adding some more
exclude conditions to *file_pie sections (the ones that apply from cc1_pie and
link_pie) to be really "correct". You shouldnt unset for vanilla the sections
because you wont get the non-default behaviour. You can find the defaults in my
specs tarball in vanilla.specs.
If you prefer to have them
empty for vanilla, I will try to redo the piepatch to have it that way
Comment 68 Peter S. Mazinger 2006-10-09 10:54:19 UTC
well, then I wouldnt care to create at all a specs file (on gcc4) having it
always as vanilla and setting GCC_SPECS=hardened.specs if use hardened
Comment 69 Peter S. Mazinger 2006-10-09 11:03:12 UTC
Kevin: are you intending to do gcc-3.4.6 minispecs aware too? If yes, I will
recheck the %{some_condition:some_flag:;%(some_section)} usage to minimize
piepatch-v9
Comment 70 Alexander Gabert (RETIRED) gentoo-dev 2006-10-09 15:37:29 UTC
(In reply to comment #54)
> Alex, I have no idea what your intention with this patch is, but it seems you
> should reread the patch piepatch/nondef/00* and explain to me how your proposed
> changes will improve that one.
> 

Hello Peter, sorry for disturbing you again.

My idea is to also have a modularized aspect of applying the hardened patches for the modularized specs.

I know that this doesn't sound good for you because it means heavy work for you redoing all your patch directory structures and so on if you accept my changes which i doubt you will.

And i also can imagine that this is maybe the solution to a problem that does not exist.

What i was thinking about is splitting the work into clearly defined header files and defining three modes of operation for patching gcc:

- vanilla mode:
leaving everything upstream behaviour, no callouts, no specs modifications, no improvements and changes to gcc specs

- internal (builtin) mode:
manipulating the internal specs with one of the three build policies:
- automatic hardenedpiessp
- automatic hardenedpie, optional hardenedssp can be turned on with switches
- automatic hardenedssp, optional hardenedpie can be turned on with switches

in either case of chosen build policy, the internal builtin specs are containing direct modifications in the asm, cc1, startfile, endfile and link_spec section

- callout (minispecs) mode:
manipulating the internal specs to add callout handlers to certain sections:
- the external minispecs take care of the build policy and replace startfile/endfile themselves while implementing the minispecs sections called by the callout handlers in the builtin specs.



The basic idea behind it is:
Having these patches in gcc allows for developers to go one of the three possible ways for each arch/libc combination:

Developers for a hardened untested arch can leave the definitions alone and the default fallback "vanilla mode" is retained.

Developers preferring callouts to external minispecs in GCC_SPECS environment variable can define the callout behaviour for the patch and this will let them define the names of the callouts and it will automatically be included in gcc.c for the sections where the callout can be appended or interleaved.  In my opinion the callout behaviour should NOT be applied to the internal gcc startfile/endfile behaviour because these sections are left for getting implemented fully by the external minispecs in an "all or nothing" approach- for patch clarity reasons which your current setup does not have.

This means: when the external minispecs provide automatic -pie building for a given arch and libc, they are responsible for setting up appropriate and matching startfile and endfile sections in the minispecs overwriting the gcc internal ones.

Last but not least, the developers preferring internal gcc changes can do so by applying their architecture and libc dependent patches to a set of skeleton include files (called mode-defs.h, libc-defs.h, arch-defs.h) for providing:

- mode-defs.h: the mode of operation (internal in this case)
- libc-defs.h: the names of the libc dependent startup object files (uclibc or glibc)
- arch-defs.h: the order and appearance of these object files in the architecture dependent startfile/endfile sections (by defining these sections with definitions used from libc-defs.h)

We can provide a "skeleton" patch to developers creating a hardened patch for a new arch/libc target configuration and they just have to patch their respective changes into this layout.

I know it sounds revolutionary and overbloated, but this is what i think after having read about all your both work with the toolchain.eclass, the uclibc, glibc work and the implementation of upstream, default, nondefault behaviour for the different arches.

I don't give much about upstream directories.
Because i myself are confident we can reach more consensus and have better working systems when concentrating on the work we can do for Gentoo and not for upstream.  The reaction to your patch requests is one more sign of the negative attitude towards such manipulations, Peter.

Which, speaking in a positive direction, absolutely leaves us free to do it our way.

The previous attempt to clean up the patches into nondefault and default directories is a good step forward.

But i think you should give my idea a try and think about it yourself.
IF you like it, i can try setting up the header files and working with you to put some life into it.  If you don't like it and want to go on with the current "one way" direction of implementing your changes, i can't do anything about it either.

It's your decision.

Thank you,

Alex
Comment 71 Alexander Gabert (RETIRED) gentoo-dev 2006-10-09 16:06:49 UTC
The positive advantages of my approach is:

-- the vanilla-default patch (which only consists of empty header files, the nonintrusive gcc.c and config.gcc modification) is applied to the gcc source in the ebuild regardless of arch, libc used, hardened profile or use flag

the Gentoo toolchain can look at this patch and easily realize it does nothing "bad" to the gcc but only adding some empty defines and empty string constants to the section string constants.

-- smaller and better structured patches for achieving support for different arch/libc/modus/policy combinations

The word "arch" in this case refers to the target the gcc should be producing executables for:x86, hppa, ppc, x86_64

The word "libc" refers to either uclibc or glibc

The word "modus" refers to the way the gcc is achieving the hardened modifications: either no modifications, or callouts or internal specs modifications

The word "policy" refers to the desired automatic security features (resembled in gcc-config profiles):
hardenedpiessp
hardenedpie
hardenedssp

-- less toolchain.eclass logic needed
The toolchain eclass currently in portage is responsible for determining about the support for hardened (as defined in VARS in the ebuilds), adding the correct -DEFAULT_* switches to the Makefile, regenerating the specs files for the different security features wanted to be automatically enabled during gcc compiles.  It also creates the profiles for gcc-config accordingly.

Now, instead of adding and checking the -DEFAULT_* switches in the Makefile, it would rather replace the header files describing the mode of operation, the used libc and the given arch.  Creating templates of these header files will release pressure from the complicated toolchain logic because the toolchain just has to look up the corresponding files for the features in the patch tarball.

If the header files or patches are found, they are applied or copied to the headers in the gcc/config/gentoo-hardened/ directory.

If not, the default logic takes place which means: nothing evil happens.

The motivation behind these changes is that i have lost overview about the gcc ebuilds, the toolchain eclass functions and the patch tarballs responsible for successfully setting up PIE and SSP functionality in gcc.

Thanks again for your attention,


-Alex
Comment 72 Kevin F. Quinn (RETIRED) gentoo-dev 2006-10-09 16:56:28 UTC
(In reply to comment #67)
> If you prefer to have them
> empty for vanilla, I will try to redo the piepatch to have it that way

No, they're fine as they are - my mistake.  I've fixed the no* microspecs for building vanilla.specs.  They're only needed if we build the main specs file with the hardened definitions.  It also means that both hardened.specs and vanilla.specs will work correctly regardless whether we have a hardened main 'specs' or not.

> well, then I wouldnt care to create at all a specs file (on gcc4) having it
> always as vanilla and setting GCC_SPECS=hardened.specs if use hardened

The reason for creating a specs file for hardened users is to ensure the hardened default even if GCC_SPECS becomes undefined (e.g. if some unusual build process does 'env -i gcc <blah>').  It was one of solar's concerns.  It's a minor point, however.  Not so much a problem with eselect-compiler, as it sets GCC_SPECS itself in the compiler wrapper.

> are you intending to do gcc-3.4.6 minispecs aware too? If yes, I will
> recheck the %{some_condition:some_flag:;%(some_section)} usage to minimize
> piepatch-v9

I wasn't going to bother with 3.4.6, just leave it as it is.  I don't understand what you mean about rechecking the specs though - if we leave 3.4.6 as it is, it'll use the v8 piepatch so v9 will only have to worry about 4.x.
Comment 73 Kevin F. Quinn (RETIRED) gentoo-dev 2006-10-09 17:10:56 UTC
Alex, if I understand you correctly, you're advocating providing three different ways to build gcc; one properly vanilla, the second similar to what happens now with gcc-3, and the third being what we're doing now.

You can get vanilla currently by setting USE="nopie -hardened".  USE="nopie" prevents the piepatches being applied.  You can also set SPLIT_SPECS="false" to stop it generating the variations of the specs (leaving you with just the builtin default), and you can set USE="nossp" to prevent the SSP patch being applied in gcc-3.x

With regards providing both (2) and (3), I don't see any point - one or the other, but not both.

With regards your comment about startfile/endfile callouts vs full definitions - with the modifications we're making you can still replace them completely in your own specs file - just define *startfile: rather than *startfile_pie: etc.  You can always redefine a spec in a later specfile - last definition wins.

On the complexity in toolchain.eclass; for our approach we've now dropped the EFAULT_PIE etc definitions and the multiple rebuilds of the driver in favour of effectively building gcc's builtin specs vanilla (the callouts exist, but they're defined the vanilla way).  Note that we can't go deleting code in toolchain.eclass as it's needed to support the older gcc versions in the tree.

Perhaps it would be best to wait a little while; I think we'll have something testable soon, and you'll be able to get a better feel for what can and can't be done with our current solution.
Comment 74 Peter S. Mazinger 2006-10-09 23:10:56 UTC
Kevin, I propose following changes:
noznow.specs: *link_now:\n%{nonow:} only for the case some ebuild used that
to provide dummy as we use %{!nonow:-z now} in the hardened one
lazy was a bad thought of me, we havent used that at all, so we not have to
provide dummy for it
znow.specs: *link_now:\n%{!nonow:-z now} leaving out dummy %{now}, we havent
ever used it, I have no idea how to handle the case if -z lazy is used, we
should check, if it will win over default -z now if provided in append-ldflags
zrelro.psecs: *link_relro:\n%{!norelro:} without %{relro:}, again, relro was
never used, so no need to provide dummy
nossp and nosspall.specs should have 2 empty lines (the first for empty entry,
the second as space delimiter to the next section
Comment 75 Peter S. Mazinger 2006-10-09 23:27:27 UTC
Created attachment 99255 [details, diff]
some more bits for toolchain.eclass

some cosmetics, remove some unneeded parts
I have "downgraded" to 9.0.3, to allow using the gcc-3.4.6 patch I have provided
tested with gcc-3.4.6 as it is, and the result is ok
what is left: should we at all add dummy entries for htb (there is no gcc-4.1.x
htb patch), the uclibc parts that are changed in my toolchain.eclass should not
be applied to portage, the most of it is unneeded/unused, some others are only
usable with the svn version
Comment 76 Peter S. Mazinger 2006-10-09 23:55:01 UTC
Created attachment 99257 [details]
updated set of specs

updated specs, implements all the changes I proposed earlier, adds some more
conditions to the *file_pie sections, I have no other ideas, havent checked yet
-z lazy handling on a hardened profile
Comment 77 Kevin F. Quinn (RETIRED) gentoo-dev 2006-10-10 00:14:14 UTC
Thanks Peter - the specs & toolchain.eclass are now up-to-date in svn.

Re -z lazy - if specified with append-ldflags (-Wl,-z,lazy), it will override the -z now in specs.  It's "last man wins" as usual (I tested to make sure).
Comment 78 Peter S. Mazinger 2006-10-10 02:25:01 UTC
thanks, I bug introduced by me, please rename *.spec to *.specs
Comment 79 Alexander Gabert (RETIRED) gentoo-dev 2006-10-10 03:33:47 UTC
(In reply to comment #73)
> Alex, if I understand you correctly, you're advocating providing three
> different ways to build gcc; one properly vanilla, the second similar to what
> happens now with gcc-3, and the third being what we're doing now.
> [...]
> Note that we can't go deleting code in
> toolchain.eclass as it's needed to support the older gcc versions in the tree.

That's exactly what i mean.

The toolchain.eclass is overly populated with backward compatibility changes for gcc3 versions.  I am not planning to invest any more time mocking up the toolchain.eclass for doing the same to gcc4.

Because for gcc4 it looks different: SSP is in the compiler, there is no need for patching it in or testing for availability (note for vapier: hppa needs SSP testing in gcc4, probably they made it work).

I don't vote for replacing toolchain.eclass because i know we need it for gcc3 versions.  My only concern is to stop adding more code to toolchain.eclass thus adding more code to a problem that existed since the patches have been changed in 2003 and 2004.

There is no way around storing the information about the sections.

The first approach was a bash script using sed.
Then came a jumbo patch doing all arches together.
Then came header logic dividing the work for the arches.
Then came small patches dividing up the work for the arches and getting used by toolchain.eclass, which also was responsible for setting up specs and sorting out issues like version management and availability checking.

The information about the specs still needs to be stored somewhere.  This is what all solutions have in common.

The only thing i did was taking an evolutionary step forward and aggregate the different models you both proposed with adding my own idea about a more "generalized" set of header files.

Your stuff is not "wrong" or technically unsuitable.

It's just that my idea is better in my eyes because it means less work for the toolchain.eclass and less work for developers when they are about to create patches for adding automaticPIE SSP for arches and standard C libraries ;)

Code following.


Alex
Comment 80 Kevin F. Quinn (RETIRED) gentoo-dev 2006-10-10 04:26:29 UTC
Peter,

already caught the spec->specs name thing :)

re. htb stubs - I was thinking it's likely that an htb patch will be created in the future for gcc-4.1 or later, and we might as well leave the stubs in.  That can be dealt with separately, anyway.

I'll remove the uclibc changes from my toolchain.eclass, so that I can just transfer my svn into portage when the time comes.  Best to deal with uclibc separately, as well.

The piepatches themselves, are you going to rework them to remove the EFAULT_* pieces, or just leave those in (since they don't hurt as we simply never set those macros anyway)?
Comment 81 Peter S. Mazinger 2006-10-10 04:42:55 UTC
(In reply to comment #70)
> My idea is to also have a modularized aspect of applying the hardened patches
> for the modularized specs.
> 
> I know that this doesn't sound good for you because it means heavy work for you

the work can be done by you ;)
I am interested in anything that makes the sum of patches + minispecs smaller

I will take the time to explain what the current piepatch does:

I consider the stuff in piepatch/upstream general correction (both PIC/PIE related), you can see what was in the 3.4.6 patch there, parts were accepted upstream, others not. Those patches could be moved to the generic set of gcc patches or tried to get them applied upstream (less the 00_all_gcc-4.0-cvs-incompat.patch, that guards against bad combinations like using shared and static or pie and shared at the same time, the guard was added to be sure that the startfile/endfile are handled correctly even on vanilla-gcc. Some time ago a patch was accepted in binutils before 2.16.1, that takes care of the combos shared|static and pie|static in the linker. Combo shared|pie is no problem as of startfile/endfile, because it will fail immediately because no *crt1.o will be linked in. The remaining combos guarded
against are really weak, like fPIE|shared or fno-PIC|shared (this will create a lib with text relocations), but most of the time noone compiles and links the shared lib in one step.
The start_endfile patch corrects the case if HAVE_LD_PIE is not defined (true only for old binutils), the m68k corrects this arch to be like all the others and not use -fPIC... in CPP_SPEC (eliminating the need to do anything specific for this arch), the arm patch aligns this arch with all the others and makes sure that static|shared do not occur at the same time and dynamic-linker/rdynamic is used only if !shared & !static

the only patch that we definitely need from there is the m68k one

nondef subdir:
the 00_* patch was earlier (v8) adding -pie option to ASM_SPEC/CPP_SPEC and reordering the startfile/endfile sections to all behave the same, if shared, elseif static, else... The logic remained the same for v9, but instead of using HARDENED_*SPEC for each and every arch I have replaced it by the use of newly created sections, so that the parts that we modify are on all archs the same. This patch does not modify the behaviour of gcc at all, it only
makes it cleaner for a vanilla build to use -pie everywhere (compiler/assembler/linker), instead of forcing the user to provide -K PIC or -fPIE|-fpie for assembler on some archs (note: the need to provide -KPIC /*without space*/ for mips is removed by an upstream-accepted patch, it is in the upstream dir for 3.4.6), -fPIE|fpie for compiler and -pie for the linker. This patch also adds the empty sections for ssp handling.
If you apply upstream and nondef, you have only a cleaner gcc, but it is "vanilla".

The def subdir is the one in charge of allowing to change vanilla-defaults to hardened defaults, but unless you do not define EFAULT_PIE|SSP it does not do anything. The other patch there adds nopie flag handling to the unallowed combos mentioned earlier.
With the current changes in toolchain.eclass the def directory can go, because we do not need to define EFAULT_PIE|SSP, we only create a build.specs based on use flags and export that as GCC_SPECS.
If you need more fine-grained possibilities (to build gcc for ex. as hardenednopiessp), you can add the logic to toolchain.eclass.

> in either case of chosen build policy, the internal builtin specs are
> containing direct modifications in the asm, cc1, startfile, endfile and
> link_spec section

you mean the earlier used HARDENED_*SPEC? This way you will duplicate code

> - callout (minispecs) mode:
> manipulating the internal specs to add callout handlers to certain sections:
> - the external minispecs take care of the build policy and replace
> startfile/endfile themselves while implementing the minispecs sections called
> by the callout handlers in the builtin specs.

i definitely do not understand what callouts and callout handlers mean ;-(

> 
> The basic idea behind it is:
> Having these patches in gcc allows for developers to go one of the three
> possible ways for each arch/libc combination:

who has touched gcc hardened related stuff (also toolchain.eclass) in the last 2-3 years? you, Travis, Kevin, Mike, I don
Comment 82 Peter S. Mazinger 2006-10-10 04:42:55 UTC
(In reply to comment #70)
> My idea is to also have a modularized aspect of applying the hardened patches
> for the modularized specs.
> 
> I know that this doesn't sound good for you because it means heavy work for you

the work can be done by you ;)
I am interested in anything that makes the sum of patches + minispecs smaller

I will take the time to explain what the current piepatch does:

I consider the stuff in piepatch/upstream general correction (both PIC/PIE related), you can see what was in the 3.4.6 patch there, parts were accepted upstream, others not. Those patches could be moved to the generic set of gcc patches or tried to get them applied upstream (less the 00_all_gcc-4.0-cvs-incompat.patch, that guards against bad combinations like using shared and static or pie and shared at the same time, the guard was added to be sure that the startfile/endfile are handled correctly even on vanilla-gcc. Some time ago a patch was accepted in binutils before 2.16.1, that takes care of the combos shared|static and pie|static in the linker. Combo shared|pie is no problem as of startfile/endfile, because it will fail immediately because no *crt1.o will be linked in. The remaining combos guarded
against are really weak, like fPIE|shared or fno-PIC|shared (this will create a lib with text relocations), but most of the time noone compiles and links the shared lib in one step.
The start_endfile patch corrects the case if HAVE_LD_PIE is not defined (true only for old binutils), the m68k corrects this arch to be like all the others and not use -fPIC... in CPP_SPEC (eliminating the need to do anything specific for this arch), the arm patch aligns this arch with all the others and makes sure that static|shared do not occur at the same time and dynamic-linker/rdynamic is used only if !shared & !static

the only patch that we definitely need from there is the m68k one

nondef subdir:
the 00_* patch was earlier (v8) adding -pie option to ASM_SPEC/CPP_SPEC and reordering the startfile/endfile sections to all behave the same, if shared, elseif static, else... The logic remained the same for v9, but instead of using HARDENED_*SPEC for each and every arch I have replaced it by the use of newly created sections, so that the parts that we modify are on all archs the same. This patch does not modify the behaviour of gcc at all, it only
makes it cleaner for a vanilla build to use -pie everywhere (compiler/assembler/linker), instead of forcing the user to provide -K PIC or -fPIE|-fpie for assembler on some archs (note: the need to provide -KPIC /*without space*/ for mips is removed by an upstream-accepted patch, it is in the upstream dir for 3.4.6), -fPIE|fpie for compiler and -pie for the linker. This patch also adds the empty sections for ssp handling.
If you apply upstream and nondef, you have only a cleaner gcc, but it is "vanilla".

The def subdir is the one in charge of allowing to change vanilla-defaults to hardened defaults, but unless you do not define EFAULT_PIE|SSP it does not do anything. The other patch there adds nopie flag handling to the unallowed combos mentioned earlier.
With the current changes in toolchain.eclass the def directory can go, because we do not need to define EFAULT_PIE|SSP, we only create a build.specs based on use flags and export that as GCC_SPECS.
If you need more fine-grained possibilities (to build gcc for ex. as hardenednopiessp), you can add the logic to toolchain.eclass.

> in either case of chosen build policy, the internal builtin specs are
> containing direct modifications in the asm, cc1, startfile, endfile and
> link_spec section

you mean the earlier used HARDENED_*SPEC? This way you will duplicate code

> - callout (minispecs) mode:
> manipulating the internal specs to add callout handlers to certain sections:
> - the external minispecs take care of the build policy and replace
> startfile/endfile themselves while implementing the minispecs sections called
> by the callout handlers in the builtin specs.

i definitely do not understand what callouts and callout handlers mean ;-(

> 
> The basic idea behind it is:
> Having these patches in gcc allows for developers to go one of the three
> possible ways for each arch/libc combination:

who has touched gcc hardened related stuff (also toolchain.eclass) in the last 2-3 years? you, Travis, Kevin, Mike, I don´t think that many developers are interested in touching gcc ...
You are expecting too much from the devs, I don´t think that each arch will have a developer who will want to do these minispecs and maintain them

> Last but not least, the developers preferring internal gcc changes can do so by
> applying their architecture and libc dependent patches to a set of skeleton
> include files (called mode-defs.h, libc-defs.h, arch-defs.h) for providing:
> 
> - mode-defs.h: the mode of operation (internal in this case)
> - libc-defs.h: the names of the libc dependent startup object files (uclibc or
> glibc)
> - arch-defs.h: the order and appearance of these object files in the
> architecture dependent startfile/endfile sections (by defining these sections
> with definitions used from libc-defs.h)

Even doing this, you cant avoid touching each archs config/arch/linux*.h or elf.h or sysv4.h, because not all archs will include the generic linux.h and they are not expected to do it in future either (I have asked on gcc-ml)
If you do not touch them, you will have to duplicate code from there

> But i think you should give my idea a try and think about it yourself.

I can´t, because I do not really understand all these options you are speaking about, present something, then I will think about it, don´t try to convince me though that your old approach before I took over maintaining the piepatches is better ;)

> IF you like it, i can try setting up the header files and working with you to
> put some life into it.  If you don't like it and want to go on with the current
> "one way" direction of implementing your changes, i can't do anything about it
> either.

implement it and I´ll take a look

Tell me what you are missing and is not implementable based on what is currently done (but first try out Kevin´s overlay)

Peter
Comment 83 Peter S. Mazinger 2006-10-10 05:00:28 UTC
(In reply to comment #80)
> re. htb stubs - I was thinking it's likely that an htb patch will be created in
> the future for gcc-4.1 or later, and we might as well leave the stubs in.  That
> can be dealt with separately, anyway.
haven
Comment 84 Peter S. Mazinger 2006-10-10 05:00:28 UTC
(In reply to comment #80)
> re. htb stubs - I was thinking it's likely that an htb patch will be created in
> the future for gcc-4.1 or later, and we might as well leave the stubs in.  That
> can be dealt with separately, anyway.
haven´t checked if the added ifdefs go correctly if boundschecking used, the
piepatch handles it though, havent checked the stubs themselves

> I'll remove the uclibc changes from my toolchain.eclass, so that I can just
> transfer my svn into portage when the time comes.  Best to deal with uclibc
> separately, as well.
ok (I would like 2 changes go in, use --enable-sjlj-exceptions to be used for all gcc<4 and the removal of *uclibc* also for -z ${UCLIBC_VER}, but that can be done by Mike

> The piepatches themselves, are you going to rework them to remove the EFAULT_*
> pieces, or just leave those in (since they don't hurt as we simply never set
> those macros anyway)?

I have thought of syncing everything with the current "microspecs" (if at all needed) and remove them. As I stated earlier the mandatory patches are upstream/*m68k* and nondef/*. I could eliminate the corrections for !HAVE_LD_PIE (we have only binutils>=2.16.1) and redo the main nondef patch to apply directly without the need for upstream patches.

Peter

Comment 85 Alexander Gabert (RETIRED) gentoo-dev 2006-10-10 11:23:59 UTC
Created attachment 99310 [details, diff]
gcc-hardened-framework.patch

hardened headers with demo implementation for x86_64

what's still missing: the filter chains and the real implementation of the internal builtin hardened automatic PIE SSP sections


Alex
Comment 86 Alexander Gabert (RETIRED) gentoo-dev 2006-10-10 11:32:58 UTC
Created attachment 99312 [details, diff]
gcc -DHAVE_LD_PIE -DUSE_UCLIBC -o test test.c; ./test

test case for testing the hardened header logic without compiling gcc
Comment 87 Alexander Gabert (RETIRED) gentoo-dev 2006-10-10 11:35:38 UTC
the list of empty hardened-stubs.h is not needed any more, it can be discarded from the patch, it was before i put the "empty" logic into the hardened.h for the vanilla case also.

Alex
Comment 88 Peter S. Mazinger 2006-10-10 15:47:36 UTC
Alex, I looked at your framework:
a. the UCLIBC [S]crt1.o is not needed
b. gcc.c has to provide for the *CALLOUT case the new sections too
c. you will need to duplicate the startfile/endfile sections in your headers for
each arch (when a new gcc will be released, your patch will probably apply, but
the one updating the framework patch will have to review all archs' *file sections
(as a comparison, currently I have to grep for STARTFILE_SPEC/ENDFILE_SPEC in
config/arch/* and see if any new one occured
d. does not achieve my only goal I told you to have smaller patches
I had some time ago the idea to make the patches even smaller by adding some new
defines to each arch (and yes, these defines could be done in an external header)
like HAVE_CRTBEGIN_T/HAVE_PROFILE/NEED_ASM_PIE/HAVE_FASTMATH/
HAVE_SPECIAL_START/HAVE_SPECIAL_END and based on these defines provide all the
possible versions of the changed sections. I have dropped the idea exactly because
I loose track of changes, it becomes difficult to compare with the "originals"
Out of curiousity, why do you need BUILTIN_MODE?
Comment 89 Peter S. Mazinger 2006-10-11 06:04:40 UTC
Created attachment 99356 [details]
piepatch 9.0.5 for gcc-4.1.1

"release" 9.0.5 piepatch for gcc-4.1.1-r1:
- synced ./def with microspecs (although not used), provided only for completeness
- renamed arm patch to reflect that it is an uclibc patch dependent part,
needs update toolchain.eclass (use vanilla || [[ -z ${UCLIBC_VER} ]]) && \
rm -f ${WORKDIR}/piepatch/*/*uclibc*
added the x86[_64] parts into the main start_endfile patch, rediffing it
toolchain.eclass should require "9.0.5" for want_minispecs
Comment 90 Peter S. Mazinger 2006-10-11 06:07:23 UTC
Created attachment 99357 [details]
piepatch 9.0.5 for gcc-3.4.6

patch in sync with the gcc-4.1.1 version, allows minispecs to be used with gcc34
same comments apply as for gcc-4.1.1 (less the x86 fixes not needed here)
Comment 91 Alexander Gabert (RETIRED) gentoo-dev 2006-10-26 17:00:51 UTC
okay, i don't feel like meaning to push you or something, rather a soft and gentle "smooching" you into having this hardened stuff merged into CVS for ~arch testing on x86 and amd64... what's the status of your work, can we have it in one piece for gcc ebuild in portage now?

As said, i would like to have progress for being prepared to the release of 2007.0 with the new glibc and gcc...

Alex
Comment 92 Alexander Gabert (RETIRED) gentoo-dev 2006-10-26 17:03:43 UTC
adding dependency on glibc bug for introducing backwards compatible symbols and functions for SSP gcc-3.* and SSP gcc-4.* versions.

Comment 93 Alexander Gabert (RETIRED) gentoo-dev 2006-11-06 14:38:16 UTC
can we have the SSP filtering on LIBC back please until it's fixed in glibc building.

recent glibc versions (2.4) fail building with hardened SSP gcc 3.4.6 because of symbol fluff up in the main libc parts leading to compile errors.


i think this problem is known.


Alex
Comment 94 Alexander Gabert (RETIRED) gentoo-dev 2006-12-05 14:06:04 UTC
...
stage1/xgcc -Bstage1/ -B/usr/x86_64-pc-linux-gnu/bin/   -O2 -pipe -fprofile-generate -DIN_GCC   -W -Wall -Wwrite-strings -Wstrict-prototypes -Wmissing-prototypes -pedantic -Wno-long-long -Wno-variadic-macros -Wold-style-definition -Wmissing-format-attribute     -DHAVE_CONFIG_H -DGENERATOR_FILE  -o build/genmodes \
         build/genmodes.o build/errors.o ../build-x86_64-pc-linux-gnu/libiberty/libiberty.a
/usr/x86_64-pc-linux-gnu/bin/ld: ../build-x86_64-pc-linux-gnu/libiberty/libiberty.a(hashtab.o): relocation R_X86_64_32S against `a local symbol' can not be used when making a shared object; recompile with -fPIC
../build-x86_64-pc-linux-gnu/libiberty/libiberty.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
make[2]: *** [build/genmodes] Error 1
make[2]: Leaving directory `/var/tmp/portage/gcc-4.1.1-r3/work/build/gcc'
make[1]: *** [stageprofile_build] Error 2
make[1]: Leaving directory `/var/tmp/portage/gcc-4.1.1-r3/work/build/gcc'
make: *** [profiledbootstrap] Error 2

!!! ERROR: sys-devel/gcc-4.1.1-r3 failed.
Call stack:
  ebuild.sh, line 1546:   Called dyn_compile
  ebuild.sh, line 937:   Called src_compile
  ebuild.sh, line 1255:   Called toolchain_src_compile
  toolchain.eclass, line 24:   Called gcc_src_compile
  toolchain.eclass, line 1588:   Called gcc_do_make
  toolchain.eclass, line 1449:   Called die

!!! emake failed with profiledbootstrap
!!! If you need support, post the topmost build error, and the call stack if relevant.

!!! This ebuild is from an overlay: '/usr/local/chroot/tmp/kevquinn_overlays/toolchain'

chroot001 miranda ~ # emerge --info
Portage 2.1.1-r2 (default-linux/amd64/2006.0, gcc-3.4.4, glibc-2.4-r4, 2.6.18-hardened x86_64)
=================================================================
System uname: 2.6.18-hardened x86_64 Dual Core AMD Opteron(tm) Processor 280
Gentoo Base System version 1.12.6
Last Sync: Mon, 04 Dec 2006 02:30:01 +0000
app-admin/eselect-compiler: [Not Present]
dev-java/java-config: [Not Present]
dev-lang/python:     2.4.3-r4
dev-python/pycrypto: 2.0.1-r5
dev-util/ccache:     [Not Present]
dev-util/confcache:  [Not Present]
sys-apps/sandbox:    1.2.17
sys-devel/autoconf:  2.13, 2.60
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2
sys-devel/binutils:  2.16.1-r3
sys-devel/gcc-config: 1.3.13-r4
sys-devel/libtool:   1.5.22
virtual/os-headers:  2.6.11-r2
ACCEPT_KEYWORDS="amd64"
AUTOCLEAN="yes"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/X11/xkb"
CONFIG_PROTECT_MASK="/etc/env.d /etc/gconf /etc/terminfo"
CXXFLAGS="-O2 -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoconfig buildpkg cvs distclean distlocks metadata-transfer sandbox sfperms strict"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
MAKEOPTS="-j3"
PKGDIR="/usr/portage/packages"
PORTAGE_RSYNC_EXTRA_OPTS=""
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude='/distfiles' --exclude='/local' --exclude='/packages'"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/chroot/tmp/kevquinn_overlays/toolchain"
SYNC="rsync://owl.gentoo.org/gentoo-portage"
USE="amd64 X alsa berkdb bitmap-fonts cli cracklib crypt cups dlloader dri eds elibc_glibc emboss encode foomaticdb fortran gif gnome gpm gstreamer gtk gtk2 hardened iconv imlib input_devices_evdev input_devices_keyboard input_devices_mouse ipv6 isdnlog jpeg kde kernel_linux lzw lzw-tiff mp3 mpeg ncurses nls nptl nptlonly opengl pam pcre perl png pppd python qt3 qt4 quicktime readline reflection sdl session spell spl ssl tcpd tiff truetype-fonts type1-fonts usb userland_GNU video_cards_apm video_cards_ark video_cards_ati video_cards_chips video_cards_cirrus video_cards_cyrix video_cards_dummy video_cards_fbdev video_cards_glint video_cards_i128 video_cards_i810 video_cards_mga video_cards_neomagic video_cards_nv video_cards_rendition video_cards_s3 video_cards_s3virge video_cards_savage video_cards_siliconmotion video_cards_sis video_cards_sisusb video_cards_tdfx video_cards_tga video_cards_trident video_cards_tseng video_cards_v4l video_cards_vesa video_cards_vga video_cards_via video_cards_vmware video_cards_voodoo xorg xpm xv zlib"
Unset:  CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, LDFLAGS, LINGUAS

i guess it's the automatic specs doing something bad with the PIC or PIE building during object code generation for stage1 gcc library...

Alex
Comment 95 Alexander Gabert (RETIRED) gentoo-dev 2006-12-05 14:08:22 UTC
another failure on tinderbox using the same overlay:

stage1/xgcc -Bstage1/ -B/usr/i386-pc-linux-gnu/bin/ -c   -O2 -mtune=i686 -pipe -fprofile-generate -DIN_GCC   -W -Wall -Wwrite-strings -Wstrict-prototypes -Wmissing-prototypes -pedantic -Wno-long-long -Wno-variadic-macros -Wold-style-definition -Wmissing-format-attribute     -DHAVE_CONFIG_H -DGENERATOR_FILE -I. -Ibuild -I/var/tmp/portage/gcc-4.1.1-r3/work/gcc-4.1.1/gcc -I/var/tmp/portage/gcc-4.1.1-r3/work/gcc-4.1.1/gcc/build -I/var/tmp/portage/gcc-4.1.1-r3/work/gcc-4.1.1/gcc/../include -I/var/tmp/portage/gcc-4.1.1-r3/work/gcc-4.1.1/gcc/../libcpp/include     -o build/genmodes.o /var/tmp/portage/gcc-4.1.1-r3/work/gcc-4.1.1/gcc/genmodes.c
stage1/xgcc -Bstage1/ -B/usr/i386-pc-linux-gnu/bin/ -c   -O2 -mtune=i686 -pipe -fprofile-generate -DIN_GCC   -W -Wall -Wwrite-strings -Wstrict-prototypes -Wmissing-prototypes -pedantic -Wno-long-long -Wno-variadic-macros -Wold-style-definition -Wmissing-format-attribute     -DHAVE_CONFIG_H -DGENERATOR_FILE -I. -Ibuild -I/var/tmp/portage/gcc-4.1.1-r3/work/gcc-4.1.1/gcc -I/var/tmp/portage/gcc-4.1.1-r3/work/gcc-4.1.1/gcc/build -I/var/tmp/portage/gcc-4.1.1-r3/work/gcc-4.1.1/gcc/../include -I/var/tmp/portage/gcc-4.1.1-r3/work/gcc-4.1.1/gcc/../libcpp/include     -o build/errors.o /var/tmp/portage/gcc-4.1.1-r3/work/gcc-4.1.1/gcc/errors.c
stage1/xgcc -Bstage1/ -B/usr/i386-pc-linux-gnu/bin/   -O2 -mtune=i686 -pipe -fprofile-generate -DIN_GCC   -W -Wall -Wwrite-strings -Wstrict-prototypes -Wmissing-prototypes -pedantic -Wno-long-long -Wno-variadic-macros -Wold-style-definition -Wmissing-format-attribute     -DHAVE_CONFIG_H -DGENERATOR_FILE  -o build/genmodes \
         build/genmodes.o build/errors.o ../build-i386-pc-linux-gnu/libiberty/libiberty.a
/usr/i386-pc-linux-gnu/bin/ld: warning: creating a DT_TEXTREL in object.
build/genmodes.o: In function `cmp_modes':
genmodes.c:(.text+0x136): undefined reference to `__stack_chk_fail_local'
build/genmodes.o: In function `eq_mode':
genmodes.c:(.text+0x1fc): undefined reference to `__stack_chk_fail_local'
build/genmodes.o: In function `validate_mode':
genmodes.c:(.text+0x6c7): undefined reference to `__stack_chk_fail_local'
build/genmodes.o: In function `vector_class':
genmodes.c:(.text+0x76c): undefined reference to `__stack_chk_fail_local'
build/genmodes.o: In function `new_adjust':
genmodes.c:(.text+0x9b4): undefined reference to `__stack_chk_fail_local'
build/genmodes.o:genmodes.c:(.text+0xba4): more undefined references to `__stack_chk_fail_local' follow
collect2: ld returned 1 exit status
make[2]: *** [build/genmodes] Error 1
make[2]: Leaving directory `/var/tmp/portage/gcc-4.1.1-r3/work/build/gcc'
make[1]: *** [stageprofile_build] Error 2
make[1]: Leaving directory `/var/tmp/portage/gcc-4.1.1-r3/work/build/gcc'
make: *** [profiledbootstrap] Error 2

!!! ERROR: sys-devel/gcc-4.1.1-r3 failed.
Call stack:
  ebuild.sh, line 1546:   Called dyn_compile
  ebuild.sh, line 937:   Called src_compile
  ebuild.sh, line 1255:   Called toolchain_src_compile
  toolchain.eclass, line 24:   Called gcc_src_compile
  toolchain.eclass, line 1588:   Called gcc_do_make
  toolchain.eclass, line 1449:   Called die

!!! emake failed with profiledbootstrap
!!! If you need support, post the topmost build error, and the call stack if relevant.

!!! This ebuild is from an overlay: '/usr/local/chroot/tmp/kevquinn_overlays/toolchain'

chroot001 tinderbox ~ # emerge --info
Portage 2.1.1-r2 (default-linux/x86/no-nptl, gcc-3.4.4, glibc-2.3.6-r5, 2.6.11-hardened-r15 i686)
=================================================================
System uname: 2.6.11-hardened-r15 i686 Intel(R) Pentium(R) 4 CPU 2.60GHz
Gentoo Base System version 1.12.6
Last Sync: Mon, 04 Dec 2006 06:00:01 +0000
app-admin/eselect-compiler: [Not Present]
dev-java/java-config: [Not Present]
dev-lang/python:     2.4.3-r4
dev-python/pycrypto: 2.0.1-r5
dev-util/ccache:     [Not Present]
dev-util/confcache:  [Not Present]
sys-apps/sandbox:    1.2.17
sys-devel/autoconf:  2.13, 2.60
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2
sys-devel/binutils:  2.16.1-r3
sys-devel/gcc-config: 1.3.13-r4
sys-devel/libtool:   1.5.22
virtual/os-headers:  2.6.17-r2
ACCEPT_KEYWORDS="x86"
AUTOCLEAN="yes"
CBUILD="i386-pc-linux-gnu"
CFLAGS="-O2 -mtune=i686 -pipe"
CHOST="i386-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/texmf/dvipdfm/config/ /usr/share/texmf/dvips/config/ /usr/share/texmf/tex/generic/config/ /usr/share/texmf/tex/platex/config/ /usr/share/texmf/xdvi/ /var/bind"
CONFIG_PROTECT_MASK="/etc/env.d /etc/gconf /etc/terminfo"
CXXFLAGS="-O2 -mtune=i686 -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoconfig buildpkg cvs distclean distlocks metadata-transfer sandbox sfperms strict"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
MAKEOPTS="-j3"
PKGDIR="/usr/portage/packages"
PORTAGE_RSYNC_EXTRA_OPTS=""
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude='/distfiles' --exclude='/local' --exclude='/packages'"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/chroot/tmp/kevquinn_overlays/toolchain"
SYNC="rsync://owl.gentoo.org/gentoo-portage"
USE="x86 elibc_glibc hardened input_devices_evdev input_devices_keyboard input_devices_mouse kernel_linux multiuser ncurses nls offensive pam server userland_GNU video_cards_apm video_cards_ark video_cards_ati video_cards_chips video_cards_cirrus video_cards_cyrix video_cards_dummy video_cards_fbdev video_cards_glint video_cards_i128 video_cards_i740 video_cards_i810 video_cards_imstt video_cards_mga video_cards_neomagic video_cards_nsc video_cards_nv video_cards_rendition video_cards_s3 video_cards_s3virge video_cards_savage video_cards_siliconmotion video_cards_sis video_cards_sisusb video_cards_tdfx video_cards_tga video_cards_trident video_cards_tseng video_cards_v4l video_cards_vesa video_cards_vga video_cards_via video_cards_vmware video_cards_voodoo"
Unset:  CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, LDFLAGS, LINGUAS


bye Alex
Comment 96 Peter S. Mazinger 2006-12-05 14:37:32 UTC
the other failure about __stack_chk_fail_local is probably not easy to solve
I would disable the use of ssp, if gcc-4 is compiled with a gcc-3 and if glibc
was not compiled with gcc-4 (the latter is your case)
Comment 97 Kevin F. Quinn (RETIRED) gentoo-dev 2006-12-05 17:02:32 UTC
(from ps.m - https://bugs.gentoo.org/show_bug.cgi?id=94325#c142)

> the patch used earlier to get the PIC version of libiberty.a is not enough
> for >=4.0.2, you need sed -i 's:^PICFLAG.*:PICFLAG = -fPIC:'
> ${S}/libiberty/Makefile.in

Why do we want libiberty.a built -fPIC?  I thought libiberty.a was for building static binaries (and hence gets -fPIE from specs?)
Comment 98 Peter S. Mazinger 2006-12-05 22:35:39 UTC
libiberty.a is used for all intermediate files (these are not static), there
is no libiberty.so. If you default to -fPIE, all those files will have text
relocations and if you use a PaX enabled kernel with TEXTREL support off, or
uClibc with TEXTREL support switched off, these binaries won't run, so on a
hardened box you have to compile every library, even *.a with -fPIC.
Comment 99 Alexander Gabert (RETIRED) gentoo-dev 2006-12-05 23:40:09 UTC
Created attachment 103422 [details]
sparc failure building glibc-2.3.6-r5 with hardened kevquinn
Comment 100 Kevin F. Quinn (RETIRED) gentoo-dev 2006-12-06 03:21:48 UTC
(In reply to comment #92)
> chroot001 miranda ~ # emerge --info
> Portage 2.1.1-r2 (default-linux/amd64/2006.0, gcc-3.4.4, glibc-2.4-r4,
> 2.6.18-hardened x86_64)

I've just done the following on both x86 and x86_64 boxes:

1) Built glibc-2.4-r4 with gcc-3.4.6-r2 (hardened)
2) Built gcc-4.1.1-r3 with gcc-3.4.6-r2 (hardened)

with no problem.  Can you try updating to gcc-3.4.6 first?  -r2 is still ~amd64 - I don't know if what went into -r2 over -r1 (which is stable) makes a difference here.
Comment 101 Alexander Gabert (RETIRED) gentoo-dev 2006-12-06 08:44:20 UTC
Created attachment 103458 [details]
hake build failure
Comment 102 Alexander Gabert (RETIRED) gentoo-dev 2006-12-06 08:45:00 UTC
Created attachment 103459 [details]
luna build failure
Comment 103 Peter S. Mazinger 2006-12-06 08:56:56 UTC
luna needs -fPIC for libiberty.a, check readelf -d genmodes for TEXTREL
Comment 104 Kevin F. Quinn (RETIRED) gentoo-dev 2006-12-07 13:36:57 UTC
For those of you seeing the libiberty.a/TEXTREL issue, it occurs when you start with a non-hardened compiler (e.g. starting from a non-hardened stage 3) and try to go straight to a fully hardened compiler.  I reproduced the issue doing that, and resolved it with this sequence (starting with a stage 3, adding my overlay, and unmasking/keywording sys-libs/glibc-2.4-r4 and sys-devel/gcc-4.1.1-r3):


1) USE="-hardened" emerge --oneshot sys-devel/gcc
   This gets a non-hardened compiler, including the specs switching to enable hardened compilations (the stage 3 will not have the specs switching)

2) Switch to the -4.1.1-hardened compiler (gcc-config, source /etc/profile)

3) emerge --oneshot sys-devel/gcc
   This builds the compiler itself hardened (i.e. the compiler itself is a PIE with SSP), gives you PIE static compiler libraries, and causes the default compiler to be -hardened rather than -vanilla.


Note the difference between a fully-hardened compiler, and a hardened-spec compiler obtained by switching to the -hardened variant that non-hardened installations provide.  A fully-hardened compiler is built from hardened objects - yet when trying to build the hardened compiler from the vanilla compiler, that only comes in when building stage2 from stage1.  Clearly there must be some linking going on between stuff built for stage1 (non-PIE) and stuff built for stage2 (PIE).  Getting the hardened specs first (step 2) then using the hardened-spec compiler to build a fully-hardened compiler works because the stuff built for stage1 is then PIE, so no conflict occurs when going from stage1 to stage2.


I think what's happening, is that trying step (3) without doing steps 1&2 causes the build to create conflicting object types from one stage to the next.  The reason I hadn't see it is that I never did it that way; it never occurred to me that it would work (and indeed in general it doesn't).  It's possible that we have gotten away with it in some situations, since although there could be TEXTRELs in stage2, they're gone by the time you get to stage3.  On some arches the TEXTREL'ed stage2 objects may not actually cause a fatal error - and when stage2 builds stage3, stage3 ends up with no TEXTRELs.


Of course, if anyone has a sequence that leads to the failure in a different way, I'd like to hear about it.
Comment 105 Peter S. Mazinger 2006-12-07 14:20:12 UTC
it always leads to failure, if you use a kernel that blocks textrels and do not
patch libiberty.a's objects to be built with -fPIC|-fpic, you havent seen this
in earlier gcc's, because there is a patch that handled this in another way, but
is not enough for gcc-4
Comment 106 Peter S. Mazinger 2006-12-07 17:16:14 UTC
kevquinn, the build of specs is now ok as of svn, but there could be another
problem with them, if lets say you create a hardened defaults specs file, then
hardenednopie.specs have to have the "vanilla" pie parts (has to overwrite
the one in the created default specs, else pie remains enabled, so any hardenedno*
have to include no*.specs too
Comment 107 Kevin F. Quinn (RETIRED) gentoo-dev 2006-12-08 01:39:49 UTC
thanks - you're right; fixed in svn.  introduced that error when it started installing a full specs file (which is unconditionally parsed as a replacement for the builtin-specs, which are always vanilla).
Comment 108 Alexander Gabert (RETIRED) gentoo-dev 2006-12-12 10:06:36 UTC
Created attachment 103879 [details]
failure on bender (latest kevquinn overlay)
Comment 109 Alexander Gabert (RETIRED) gentoo-dev 2006-12-12 17:11:48 UTC
Created attachment 103907 [details]
current build failures
Comment 110 Alexander Gabert (RETIRED) gentoo-dev 2006-12-12 18:59:25 UTC
Created attachment 103909 [details]
another build failure on bender
Comment 111 Alexander Gabert (RETIRED) gentoo-dev 2006-12-12 23:06:06 UTC
x86 WORKSFORME


good job, Kevin

http://tinyurl.com/preview.php?num=y6un4o

glibc-2.5 buildpkg USE=hardened

http://tinyurl.com/preview.php?num=u6ofd

gcc-4.1.1 buildpkg USE=hardened
Comment 112 Kevin F. Quinn (RETIRED) gentoo-dev 2006-12-13 06:22:38 UTC
(In reply to comment #108)
> Created an attachment (id=103909) [edit]
> another build failure on bender

> genmodes.c:(.text+0x194): undefined reference to `__stack_chk_fail'

Looks like you don't have glibc-2.4 built, which you need before doing any compilation with the gcc-4.1.1-hardened compiler.
Comment 113 Alexander Gabert (RETIRED) gentoo-dev 2006-12-17 02:59:59 UTC
Kevin and Peter,

can i please have mips and alpha hardened PIE and SSP patches from your overlays?

I need them now.

If you need default specs from alpha or mips: shout.
Remember that mips is a true pic arch and crt1S.o is the same as crt1.o (this means that you don't need to tweak the startfile setup for hardened PIE vs. unhardened nonPIE)

i have a mips at home waiting for the fun and access to an alpha in Virginia thanks to kloeri.

Would be nice if i get the patches before xmas, because i will be gone until 7th of january then.


Alex
Comment 114 Peter S. Mazinger 2006-12-17 04:26:29 UTC
both mips and alpha are supported by the piepatches
Comment 115 Peter S. Mazinger 2006-12-20 00:48:15 UTC
Created attachment 104424 [details]
piepatch 9.0.6 for gcc-4.1.1

correct typo for sparc/ia64 pointed out by kevquinn, 3.4.6 is not affected
Comment 116 Alexander Gabert (RETIRED) gentoo-dev 2006-12-26 15:27:58 UTC
solar wants us to include this piece:

http://developer.momonga-linux.org/viewvc/trunk/pkgs/gcc/gcc-ppc32-msecure-plt.patch?revision=7866&pathrev=8134

2005-06-02  Alan Modra <amodra@bigpond.net.au>

	* configure.ac: Add --enable-secureplt.
Comment 117 Alexander Gabert (RETIRED) gentoo-dev 2006-12-26 15:31:14 UTC
Created attachment 104775 [details, diff]
msecure-plt support for ppc32

this patch needs love: should be default with PIE building (see specs logic)
/possibly/ needs love for other arches than ppc

- Alex
Comment 118 solar (RETIRED) gentoo-dev 2006-12-26 15:38:09 UTC
(In reply to comment #114)
> solar wants us to include this piece:

Note. I'm not sure that is the most recent copy. It may or may not be in 
mainline. It for sure should be in the redhat branches. testing by a Gentoo user shows sucess with this patch on ppc. 
With the patch the PT_LOAD segments are properly split between R-E and RW-
(Should work for ppc,ppc64,alpha,sparc)
Comment 119 solar (RETIRED) gentoo-dev 2006-12-26 15:45:26 UTC
testing on SpaNKY's ppc64 shows that the flag is supported.
Comment 120 PaX Team 2006-12-26 15:48:42 UTC
i think msecure-plt is part of binutils 2.17, if that's not part of hardened yet, it should be.
Comment 121 SpanKY gentoo-dev 2006-12-26 18:16:38 UTC
Comment on attachment 104775 [details, diff]
msecure-plt support for ppc32

gcc-4.1 already has this
Comment 122 solar (RETIRED) gentoo-dev 2006-12-26 20:03:13 UTC
Please test and see if we can enable per default.. with target being $ARCH default !hardened
Comment 123 Kevin F. Quinn (RETIRED) gentoo-dev 2006-12-29 03:25:28 UTC
Works fine for ppc32.  Just needs --enable-secureplt on configure for gcc to use the secure-plt ABI by default (which my latest branches/gcc-glibc-nopie now does).  For platforms where secureplt is irrelevant or doesn't work, it's effectively ignored.  I see no reason not to enable it always (including non-hardened), but if that's not acceptable we could do it conditional on USE=hardened.

Other than ppc, I've only tried it on sparc but as I expected it does nothing there.   The ABI has only been defined for ppc; at least for 4.1.1.
Comment 124 PaX Team 2006-12-29 04:38:33 UTC
(In reply to comment #121)
> Other than ppc, I've only tried it on sparc but as I expected it does nothing
> there.   The ABI has only been defined for ppc; at least for 4.1.1.

isn't it also present for alpha? also, there was an attempt for sparc (for a different reason, but it effectively does secure-plt iirc): http://sourceware.org/ml/binutils/2006-03/msg00087.html . i haven't followed the fate of this code, but it'd be a good opportunity for any toolchain hacker to get it to work for us ;-).
Comment 125 Kevin F. Quinn (RETIRED) gentoo-dev 2006-12-29 07:44:16 UTC
(In reply to comment #121)
> I see no reason not to enable it always (including
> non-hardened), but if that's not acceptable we could do it conditional on
> USE=hardened.

Seems it may cause a small loss in performance - so conditional on USE=hardened makes sense.
Comment 126 PaX Team 2006-12-29 08:40:57 UTC
(In reply to comment #123)
> Seems it may cause a small loss in performance - so conditional on USE=hardened
> makes sense.

do you have any numbers? i saw only the 0.5% slowdown on kernel compilation so far but i think that was more due to PaX/PAGEEXEC on ppc, not the secure-plt itself.
Comment 127 SpanKY gentoo-dev 2006-12-29 13:26:41 UTC
unless the #'s are significant, i'm going to leave the eclass as is (i committed the configure flag to always on days ago)
Comment 128 Kevin F. Quinn (RETIRED) gentoo-dev 2006-12-30 02:18:12 UTC
No numbers - just a comment Alan Modra made (I think - can't find it now).  The implication was the impact was small; I'll have to build a chroot without it now to compare.

re. alpha - support for a read-only plt went into the dynamic linker (glibc) May last year: http://sourceware.org/ml/libc-alpha/2005-05/msg00090.html  Don't know about gcc support, if that's necessary, though.  I don't have access to an alpha box to try (and alpha doesn't support ssp in gcc-4.1 anyway, so I'm not inclined to worry about it much for now).
Comment 129 Kevin F. Quinn (RETIRED) gentoo-dev 2006-12-30 02:55:05 UTC
re alpha again - supported in glibc and binutils since May last year, so that's already present (http://sourceware.org/ml/binutils/2005-05/msg00736.html http://sourceware.org/ml/libc-alpha/2005-05/msg00090.html)
Just needs --enable-secureplt on the binutils 2.17 configure to switch it on.  There's no impact on gcc for alpha (only ppc).

re ppc - the glibc configure detects whether gcc is built to default to secure-plt - which means glibc needs to be rebuilt after gcc is built that way.
Comment 130 SpanKY gentoo-dev 2006-12-30 03:25:08 UTC
i'm not worried about the 'need to rebuild gcc and then rebuild glibc and ...' aspect as it'll filter down into users' systems without any issues

ive added the --enable-secureplt to toolchain-binutils.eclass as it'll simply be ignored if it isnt supported
Comment 131 Peter S. Mazinger 2007-01-06 20:00:39 UTC
I know that redhat did not enable msecure-plt in any "production" gcc's up to
gcc-4.1.0, although they had that patch around since long, havent checked the
state of gcc-4.1.1
Comment 132 solar (RETIRED) gentoo-dev 2007-01-06 20:29:57 UTC
secureplt ported to uclibc-ppc
http://uclibc.org/cgi-bin/viewcvs.cgi?view=rev&rev=17162
Comment 133 Alexander Gabert (RETIRED) gentoo-dev 2007-01-10 23:06:51 UTC
Hello,

after a talk with psm i would suggest we postpone the experiments with glibc and gcc built ET_DYN.

It is easy to put the old filters back to it's place in the new mini specs.

So the process of building should be:

- filter automatic PIE building on glibc itself
- filter automatic PIE building on gcc itself

The gcc compiler will be ET_EXEC but have no text relocations and the gcc libraries and glibc will work fine with any kind of executables (static, pie, normal).

If SSP building gcc works with the cc1 patch Peter explained to me, i'm not worried about that part of the hardened toolchain.

For several reasons (libiberty woes, PIE vs. PIC object code generation problem, text relocations in PIE executables with the wrong object code), the building of gcc and glibc is problematic when done with PIE enabled on itself.  Until we have a working solution for that, i suggest we will be better off not trying to play with that stuff at this moment.

I hope we can get the transition done smooth and in a good time, it's already 2007 and the next release is knocking on the door...

Please bear in mind that hardened gcc never was expected to be a "full scale" solution to solve all problems building PIE executables.

There is always the problem that "gcc -c" is creating object code for which it is not known where this object code will end up finally when gcc or ld is called for linking the object code into a statically linked executable, a dynamically shared library, or a PIE executable.

As Peter noted, the only point where this is clear is when there is only one single "run" of compiling and linking, for example "gcc -o executable source.c".

However, this is almost never the case.

Futher construction yards:

* -K PIC behaviour
* postponing -fPIE on certain arches and falling back to -fPIC for everything object code we build
* suppressing the building of -static executables and forcing -pie executables even though gcc or ld is called with -static

However, this is something for the future, at the moment we should stick with what we have on the arches where -fPIE is working.

cheers,


-Alex
Comment 134 Kevin F. Quinn (RETIRED) gentoo-dev 2007-01-11 00:19:23 UTC
For the record I think I'm pretty close to a reliable solution, one way or another, so don't worry so much.  I've some stuff to work through, and I'll report back when I have things clear.
Comment 135 Jakub Moc (RETIRED) gentoo-dev 2007-04-01 14:13:47 UTC
*** Bug 172942 has been marked as a duplicate of this bug. ***
Comment 136 Jakub Moc (RETIRED) gentoo-dev 2007-04-04 09:22:38 UTC
*** Bug 172997 has been marked as a duplicate of this bug. ***
Comment 137 Xake 2007-07-17 13:34:54 UTC
So what is the status of this currently? Any progress towards the main tree?
Comment 138 Magnus Granberg gentoo-dev 2007-10-03 00:16:08 UTC
Any progress?
Comment 139 Alexander Gabert (RETIRED) gentoo-dev 2007-10-20 19:43:56 UTC
sys-libs/hardened-glibc and sys-devel/hardened-gcc in the works

with SSPX and AT_ENTROPY

enjoy and keep it up,


Alex
Comment 140 Peter S. Mazinger 2007-10-20 23:36:01 UTC
I would want to see an AT_ENTROPY backport for 2.4 if possible, thanks
Comment 141 Alexander Gabert (RETIRED) gentoo-dev 2007-10-28 13:35:49 UTC
reassigning
Comment 142 Magnus Granberg gentoo-dev 2008-04-13 15:26:15 UTC
Any updates on the progress?
New patches or files?
Comment 143 Łukasz Damentko (RETIRED) gentoo-dev 2008-05-27 13:20:58 UTC
pappy was retired (#74378), reassigning remaining bugs.
Comment 144 Jan Kundrát (RETIRED) gentoo-dev 2008-05-27 13:46:47 UTC
This is pappy's child, which means that it won't be completed without him.
Comment 145 Magnus Granberg gentoo-dev 2008-05-28 01:24:20 UTC
(In reply to comment #144)
> This is pappy's child, which means that it won't be completed without him.

Okey open a new bug for hardened gcc 4.2.x?
Comment 146 Jan Kundrát (RETIRED) gentoo-dev 2008-05-28 09:56:05 UTC
(In reply to comment #145)
> Okey open a new bug for hardened gcc 4.2.x?

If you have it ready, be my guest. If you don't, you might want to talk to our hardened team before that.
Comment 147 Magnus Granberg gentoo-dev 2008-05-28 11:15:14 UTC
(In reply to comment #146)
> (In reply to comment #145)
> > Okey open a new bug for hardened gcc 4.2.x?
> If you have it ready, be my guest. If you don't, you might want to talk to our
> hardened team before that.
We are working on a hardened gcc 4.2.x and some of the haredened team now it.
Will make new bug when it works.
http://forums.gentoo.org/viewtopic-t-668885.html



Comment 148 Peter S. Mazinger 2008-05-28 20:27:30 UTC
you might not know of it, the piepatches were "maintained" by me, even those in kevquinns overlay
i have currently some newer addons like -D_FORTIFY_SOURCE=2 support, flag-o-matic
support is though not done (I consider most of the filter-flags for ssp not needed
anymore with gcc-4, many of them were added due to problems with c++ apps or due
to lazyness of some developers who did not care about why something failed, only
"solved" the issue). Issues not solved yet or maybe not solvable at all:
glibc compiling with -D_FORTIFY_SOURCE=2 and/or -fstack-protector[-all], it has to
be evaluated which one would fit better if at all we would want it. The only cases
where I consider -fssp not needed to be enabled are emulators and boot-managers,
all the others should work without filtering
Comment 149 Magnus Granberg gentoo-dev 2008-05-29 00:13:45 UTC
(In reply to comment #148)
> you might not know of it, the piepatches were "maintained" by me, even those in
> kevquinns overlay
> i have currently some newer addons like -D_FORTIFY_SOURCE=2 support,
> flag-o-matic
> support is though not done (I consider most of the filter-flags for ssp not
> needed
> anymore with gcc-4, many of them were added due to problems with c++ apps or
> due
> to lazyness of some developers who did not care about why something failed,
> only
> "solved" the issue). Issues not solved yet or maybe not solvable at all:
> glibc compiling with -D_FORTIFY_SOURCE=2 and/or -fstack-protector[-all], it has
> to
> be evaluated which one would fit better if at all we would want it. The only
> cases
> where I consider -fssp not needed to be enabled are emulators and
> boot-managers,
> all the others should work without filtering
I did't now thet. Good woork on pie patch.
We have a working toolchain now for x86 and x86_86 (flag-o-matic,toolchain and toolchain-func)
Trying to fix the bugs we have on grub and e2fsprog on x86_86
Next for us i to get glibc build with ssp and foritfy.
Are u willing to help us?
Comment 150 Peter S. Mazinger 2008-05-29 15:33:02 UTC
I would not care about grub. I can build it pie+ssp, but it is questionable, how useful this might be. I do not have problems with e2fsprogs on i386, do you mean x86_64 or  i386? what happens?
I can help out, sometimes I am around on #gentoo-embedded
Comment 151 Xake 2008-05-29 17:23:57 UTC
We do our cooking over at https://hardened.gentooexperimental.org/secure but it is nothing official yet. 
There is a ticket for failures while building e2fsprogs PIE with x86_64.
Comment 152 Peter S. Mazinger 2008-05-29 19:07:45 UTC
some comments on your work there:
do not try to get rtld (ld-linux.so.2) compiled with -fssp[-all], since you will
bite in your own tail, ld-linux.so.2 is the one providing the __[stack_chk_]guard (at least
if TLS is disabled (like for a target i386-*-linux-*) or if TLS is enabled, does
the calculation of the guard (for consistency, this applies to libssp as well,
although not used in this context in hardened gentoo).
the fix proposed for e2fsprogs is not correct, since it will add overhead to all
binaries, -fPIC should be applied only to the shared library link line (although
there is something wrong with the linking, the binaries should link against libuuid.so
and not *.a). to be sure make all objects going into libuuid.* libraries, both shared
and archive be compiled with -fPIC (applies really to the other libraries as well)
but omit using -fPIC for the executables (then -fPIE will win and make them smaller)
Comment 153 Magnus Granberg gentoo-dev 2008-05-30 11:34:08 UTC
(In reply to comment #152)
> some comments on your work there:
> do not try to get rtld (ld-linux.so.2) compiled with -fssp[-all], since you
> will
> bite in your own tail, ld-linux.so.2 is the one providing the
> __[stack_chk_]guard (at least
> if TLS is disabled (like for a target i386-*-linux-*) or if TLS is enabled,
> does
> the calculation of the guard (for consistency, this applies to libssp as well,
> although not used in this context in hardened gentoo).
> the fix proposed for e2fsprogs is not correct, since it will add overhead to
> all
> binaries, -fPIC should be applied only to the shared library link line
> (although
> there is something wrong with the linking, the binaries should link against
> libuuid.so
> and not *.a). to be sure make all objects going into libuuid.* libraries, both
> shared
> and archive be compiled with -fPIC (applies really to the other libraries as
> well)
> but omit using -fPIC for the executables (then -fPIE will win and make them
> smaller)

Thanks for the input.
We a on #friendly-coders


Comment 154 Attila Tóth 2008-06-01 08:10:18 UTC
(In reply to comment #149)
> (In reply to comment #148)
> > you might not know of it, the piepatches were "maintained" by me, even those in
> > kevquinns overlay
> > i have currently some newer addons like -D_FORTIFY_SOURCE=2 support,
> > flag-o-matic
> > support is though not done (I consider most of the filter-flags for ssp not
> > needed
> > anymore with gcc-4, many of them were added due to problems with c++ apps or
> > due
> > to lazyness of some developers who did not care about why something failed,
> > only
> > "solved" the issue). Issues not solved yet or maybe not solvable at all:
> > glibc compiling with -D_FORTIFY_SOURCE=2 and/or -fstack-protector[-all], it has
> > to
> > be evaluated which one would fit better if at all we would want it. The only
> > cases
> > where I consider -fssp not needed to be enabled are emulators and
> > boot-managers,
> > all the others should work without filtering
> I did't now thet. Good woork on pie patch.
> We have a working toolchain now for x86 and x86_86 (flag-o-matic,toolchain and
> toolchain-func)
> Trying to fix the bugs we have on grub and e2fsprog on x86_86
> Next for us i to get glibc build with ssp and foritfy.
> Are u willing to help us?
> 

Nice to hear about it. We will end up letting Ubuntu to outperform Hardened Gentoo's security measures. Hardened Gentoo should show the path above other regular distros instead.
I'm using gcc-4.2 with some experimental specs for months now. Can be found at: http://atoth.sote.hu/~atoth/specs/. The only issue I had was that I had to switch back to a vanilla profile to compile a working grub. SSP-PIE features of the gcc-4.2 toolchain will be complete by getting rid of the widely abundant flag filtering habit in ebuilds.
I'm not a developer for myself, but willing to help as I can (testing) if my schedule permits it (summer vacation eg.).

The upcoming gcc-4.3 must be also considered with even more PIE-SSP features incorporated.

Regards:
Dw.
Comment 155 Magnus Granberg gentoo-dev 2008-06-06 12:41:22 UTC
(In reply to comment #154)
> (In reply to comment #149)
> > (In reply to comment #148)
> > > you might not know of it, the piepatches were "maintained" by me, even those in
> > > kevquinns overlay
> > > i have currently some newer addons like -D_FORTIFY_SOURCE=2 support,
> > > flag-o-matic
> > > support is though not done (I consider most of the filter-flags for ssp not
> > > needed
> > > anymore with gcc-4, many of them were added due to problems with c++ apps or
> > > due
> > > to lazyness of some developers who did not care about why something failed,
> > > only
> > > "solved" the issue). Issues not solved yet or maybe not solvable at all:
> > > glibc compiling with -D_FORTIFY_SOURCE=2 and/or -fstack-protector[-all], it has
> > > to
> > > be evaluated which one would fit better if at all we would want it. The only
> > > cases
> > > where I consider -fssp not needed to be enabled are emulators and
> > > boot-managers,
> > > all the others should work without filtering
> > I did't now thet. Good woork on pie patch.
> > We have a working toolchain now for x86 and x86_86 (flag-o-matic,toolchain and
> > toolchain-func)
> > Trying to fix the bugs we have on grub and e2fsprog on x86_86
> > Next for us i to get glibc build with ssp and foritfy.
> > Are u willing to help us?
> > 
> Nice to hear about it. We will end up letting Ubuntu to outperform Hardened
> Gentoo's security measures. Hardened Gentoo should show the path above other
> regular distros instead.
> I'm using gcc-4.2 with some experimental specs for months now. Can be found at:
> http://atoth.sote.hu/~atoth/specs/. The only issue I had was that I had to
> switch back to a vanilla profile to compile a working grub. SSP-PIE features of
> the gcc-4.2 toolchain will be complete by getting rid of the widely abundant
> flag filtering habit in ebuilds.
> I'm not a developer for myself, but willing to help as I can (testing) if my
> schedule permits it (summer vacation eg.).
> The upcoming gcc-4.3 must be also considered with even more PIE-SSP features
> incorporated.
> Regards:
> Dw.

Have a new grub ebuild and patch on my overlay for testing.
We are going to add enhancement when the toolchain is working.
What security enhancement du u whan to see?
Thanks for testing when u have time.


Comment 156 Magnus Granberg gentoo-dev 2008-06-06 12:45:29 UTC
(In reply to comment #150)
> I would not care about grub. I can build it pie+ssp, but it is questionable,
> how useful this might be. I do not have problems with e2fsprogs on i386, do you
> mean x86_64 or  i386? what happens?
> I can help out, sometimes I am around on #gentoo-embedded
Can u help whit uclibc?
Zorry on irc


Comment 157 Peter S. Mazinger 2008-06-07 09:12:46 UTC
sure, I know uClibc well enough, I am really more interested in uClibc than glibc
Comment 158 Magnus Granberg gentoo-dev 2008-06-10 03:58:20 UTC
(In reply to comment #157)
> sure, I know uClibc well enough, I am really more interested in uClibc than
> glibc

gcc 4.3.1 need a new piepatch to support arm/uClibc

sandra gcc-4.3.1 # patch -p1 < 00_all_gcc-4.1.1-v9.0.5-start_endfile.patch
patching file gcc/config/arm/uclinux-elf.h
Hunk #1 FAILED at 45.
Hunk #2 FAILED at 62.
2 out of 2 hunks FAILED -- saving rejects to file gcc/config/arm/uclinux-elf.h.rej
sandra gcc-4.3.1 # patch -p1 < 00_all_gcc.4.1.1-v9.0.7-start_endfile-nondef.patch
patching file gcc/config/arm/uclinux-elf.h
Hunk #1 FAILED at 47.
Hunk #2 FAILED at 64.
2 out of 2 hunks FAILED -- saving rejects to file gcc/config/arm/uclinux-elf.h.rej