Summary: | [PATCH] sys-fs/e2fsprogs-1.40 fails to compile on x86_64 and textrel on x86 in tst_uuid Hardened gcc4 | ||
---|---|---|---|
Product: | Gentoo Linux | Reporter: | Magnus Granberg <zorry> |
Component: | Hardened | Assignee: | The Gentoo Linux Hardened Team <hardened> |
Status: | VERIFIED CANTFIX | ||
Severity: | major | CC: | base-system, kfm, ranjit, tytso |
Priority: | High | ||
Version: | 2008.0 | ||
Hardware: | All | ||
OS: | Linux | ||
URL: | https://hardened.gentooexperimental.org/secure/changeset/be379de124f52816831a3508fc03f3d864fc3368 | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Attachments: |
makes static libs links whit -fPIC to fix TEXTREL
gcc-spec-env.patch this file i use to filter or append flags New ebuild to fix this bug fix some errors in the ebuild ebuild patch ebuild patch ebuild patch |
Description
Magnus Granberg
2008-07-21 21:43:02 UTC
Created attachment 161045 [details, diff]
makes static libs links whit -fPIC to fix TEXTREL
sys-libs/com_err and sys-libs/ss need the patch to Fix works here; has it been sent upstream yet? [url, please] Upstrem link https://sourceforge.net/tracker/index.php?func=detail&aid=2026032&group_id=2406&atid=102406 This is a problem that is caused by a non-standard gcc, I take it. "This hardened GCC" is a non-standard patch which is not in the upstream gcc, right? In that case, why should all e2fsprogs users suffer with a slowdown caused by a reduction of the x86's pathetically small register file by another 25%? If Gentoo wants to use this non-standard GCC patch, then let it carry this patch.... The only way I can see taking this patch in e2fsprogs upstream is if it is an configure option, ideally one with a test that can detect the use of this non-standard gcc patch. -Ted Created attachment 161228 [details, diff] gcc-spec-env.patch (In reply to comment #5) > This is a problem that is caused by a non-standard gcc, I take it. "This > hardened GCC" is a non-standard patch which is not in the upstream gcc, right? Hmm the only patches we're adding to the standard Gentoo build are: gcc-spec-env.patch (attached herewith) gcc-4.0.2-softfloat.patch (if the target needs soft FPU) It'd be up to Gentoo-toolchain to provide more detail as to other patches to gcc although Zorry might know more. Gentoo does stay as close to upstream as possible. > In that case, why should all e2fsprogs users suffer with a slowdown caused by a > reduction of the x86's pathetically small register file by another 25%? They absolutely shouldn't, this is only for hardened builds, which I agree should be tested for sanely. > If > Gentoo wants to use this non-standard GCC patch, then let it carry this > patch.... > Er yeah, and we will, although you haven't actually verified that these are non-standard patches, so bit early for the rhetoric ;) For instance, the gcc-4.2.0-cc1-no-stack-protector.patch simply adds -fno-stack-protector to gcc builds, which doesn't strike me as a "non-standard patch". My impression is that it's more a case of getting the whole toolchain to compile with existing options, and occasionally patches that have been around since gcc3 at least, but need work to be integrated into 4, most likely due to the change in backend. Every distro that supports compiler hardening has their own patches. Given that security is important, and e2fsprogs are a vital part of practically every Linux system.. This is about -fPIC being used to compile static libs, not just .so's, on ELF, which again does not strike me as incredibly wierd. > The only way I can see taking this patch in e2fsprogs upstream is if it is an > configure option, ideally one with a test that can detect the use of this > non-standard gcc patch. > Well how you decide whether to include it is up to you. The most basic test would be for -fPIE in default CFLAGS, although on Gentoo the spec file can be changed out at runtime (so a definitive test would be either -fPIE in default CFLAGS or in the gcc spec?) Zorry might know how we test the latter. Personally, given a definitive test, I'd simply include it based on the result of that test. If you're saying you won't include it until upstream gcc takes all the Gentoo hardened patches (or at least those which would necessitate static libs requiring -fPIC) that is of course your decision. Created attachment 161261 [details]
this file i use to filter or append flags
loock for the filer-flags stuff
(In reply to comment #6) > Created an attachment (id=161228) [edit] > gcc-spec-env.patch > > (In reply to comment #5) > > This is a problem that is caused by a non-standard gcc, I take it. "This > > hardened GCC" is a non-standard patch which is not in the upstream gcc, right? > > Hmm the only patches we're adding to the standard Gentoo build are: > gcc-spec-env.patch (attached herewith) > gcc-4.0.2-softfloat.patch (if the target needs soft FPU) > > It'd be up to Gentoo-toolchain to provide more detail as to other patches to > gcc although Zorry might know more. Gentoo does stay as close to upstream as > possible. > We add gentoo's gcc patch and a uclibc patch. For the hardened compile we add a pie patch to. Name one dist that don't add patches to the standard gcc sources. > > In that case, why should all e2fsprogs users suffer with a slowdown caused by a > > reduction of the x86's pathetically small register file by another 25%? > They absolutely shouldn't, this is only for hardened builds, which I agree > should be tested for sanely. > U have -fPIC to the sheared libs allready on elf > > If > > Gentoo wants to use this non-standard GCC patch, then let it carry this > > patch.... > > > Er yeah, and we will, although you haven't actually verified that these are > non-standard patches, so bit early for the rhetoric ;) For instance, the > gcc-4.2.0-cc1-no-stack-protector.patch simply adds -fno-stack-protector to gcc > builds, which doesn't strike me as a "non-standard patch". My impression is > that it's more a case of getting the whole toolchain to compile with existing > options, and occasionally patches that have been around since gcc3 at least, > but need work to be integrated into 4, most likely due to the change in > backend. > > Every distro that supports compiler hardening has their own patches. Given that > security is important, and e2fsprogs are a vital part of practically every > Linux system.. > Some reading on PIE and PIC http://www.gentoo.org/proj/en/hardened/hardened-toolchain.xml > This is about -fPIC being used to compile static libs, not just .so's, on ELF, > which again does not strike me as incredibly wierd. > tst_uid is compile -fPIE and then it try to link no -fPIC lib 1.39 was working fine before the update on gen_uuid > > The only way I can see taking this patch in e2fsprogs upstream is if it is an > > configure option, ideally one with a test that can detect the use of this > > non-standard gcc patch. > > > Well how you decide whether to include it is up to you. The most basic test > would be for -fPIE in default CFLAGS, although on Gentoo the spec file can be > changed out at runtime (so a definitive test would be either -fPIE in default > CFLAGS or in the gcc spec?) Zorry might know how we test the latter. > u can use some code from the filter-flag in flag-o-matic to check for -fPIE and do a sed to add fix to the Makefile.library only if --enable-elf-shlibs and no static > Personally, given a definitive test, I'd simply include it based on the result > of that test. > > If you're saying you won't include it until upstream gcc takes all the Gentoo > hardened patches (or at least those which would necessitate static libs > requiring -fPIC) that is of course your decision. > sooner or later will some dist have ELF text relocations on and the if e2fsprogs have TEXTREL it will be killed or not compile. Is okey if we only add it on hardened gentoo with gcc 4 and later upstream if needed? ugh, i dont know why this got pushed upstream like that. sorry about that Theodore ... please just punt that bug. "hardened gcc" is just gcc with custom spec files. basically everything is built as PIC/PIE. there may be a build system limitation in e2fsprogs, but if it comes to that, i'd push a generic patch upstream for people to leverage without screwing anyone else. (In reply to comment #10) > ugh, i dont know why this got pushed upstream like that. sorry about that > Theodore ... please just punt that bug. > > "hardened gcc" is just gcc with custom spec files. basically everything is > built as PIC/PIE. > > there may be a build system limitation in e2fsprogs, but if it comes to that, > i'd push a generic patch upstream for people to leverage without screwing > anyone else. > Sorry for that SpanKY and Theodore. We are using Kevquinn and Psm's work. Psm did hit textrel on x86. It is okey to have gentoo only and make something upstream if it is a build system limitation in e2fsprogs? Is needed in e2fsprogs-lib-1.41 to. Have you tried just simply passing in the the configuration option "--with-cflags=-fPIC"? Or just passing in CFLAGS set to -fPIC as an environment variable before calling configure? I'm not sure I understand how this non-standard "hardened gcc" spec file works, so I'm not sure whether *all* objects need to be built -fPIC, or just libraries (which seems wierd, but whatever). Setting CFLAGS to -fPIC will compile all object files as position indendent code, and if you benchmark CPU intensive programs you'll see why this isn't the default for gcc, especially on x86. So if what is really needed is a configuration option which only adds -fPIC when compiling objects destined for a static "ar" library, I'm willing to accept a patch which provides that level of configuration control. But just blindly compiling all static libraries with -fPIC is just crazy. Note --- given that my impression of Gentoo users is that some are so monomaniacally focused on performance that their stated reason for using Gentoo is so they can compile their programs with exactly the right gcc flags so it can take advantage of specific CPU functions (instead of the generic x86 compiler options used by most distributions that ship binaries), blindly shredding 25% of a 32-bit x86's register file is no laughing matter, and more than completely overwhelms the micro-optimizations of compiling with specific gcc CPU optimization flags. If it is also true that most Gentoo users are not compiling with the hardened gcc option, I would gently suggest that you probably do NOT want to add -fPIC for all Gentoo users, but only for those that have elected to use the hardened gcc. Best regards, - Ted [[ gcc-fullversion > 4.2.0 ]] && gcc-specs-pie && epatch "${FILESDIR}"/${PN}-make-libs.a-fpic.patch Will fix that only users with > gcc 4.2 and have PIE specs support will get the patch. If it put in the ebuild of the package. (In reply to comment #13) > Note --- given that my impression of Gentoo users is that some are so > monomaniacally focused on performance that their stated reason for using Gentoo > is so they can compile their programs with exactly the right gcc flags so it > can take advantage of specific CPU functions (instead of the generic x86 > compiler options used by most distributions that ship binaries), blindly > shredding 25% of a 32-bit x86's register file is no laughing matter, and more > than completely overwhelms the micro-optimizations of compiling with specific > gcc CPU optimization flags. If it is also true that most Gentoo users are not > compiling with the hardened gcc option, I would gently suggest that you > probably do NOT want to add -fPIC for all Gentoo users, but only for those that > have elected to use the hardened gcc. > > Best regards, > > - Ted > [[ gcc-fullversion > 4.2.0 ]] && gcc-specs-pie && epatch "${FILESDIR}"/${PN}-make-libs.a-fpic.patch Will fix that only users with > gcc 4.2 and have PIE specs support will get the patch. If it put in the ebuild of the package. (In reply to comment #12) > Have you tried just simply passing in the the configuration option > "--with-cflags=-fPIC"? Or just passing in CFLAGS set to -fPIC as an > environment variable before calling configure? > > I'm not sure I understand how this non-standard "hardened gcc" spec file works, > so I'm not sure whether *all* objects need to be built -fPIC, or just libraries > (which seems wierd, but whatever). Setting CFLAGS to -fPIC will compile all > object files as position indendent code, and if you benchmark CPU intensive > programs you'll see why this isn't the default for gcc, especially on x86. So > if what is really needed is a configuration option which only adds -fPIC when > compiling objects destined for a static "ar" library, I'm willing to accept a > patch which provides that level of configuration control. > > But just blindly compiling all static libraries with -fPIC is just crazy. > http://overlays.gentoo.org/dev/kevquinn (pieworld) branch we have updated to work on gcc 4.2 The pie part is the same. Take a look on bug 149649 will try the -with-cflags=-fPIC in the e2fsprogs-lib-1.41 ebuild If it works it okey and no need for patch. have to test it to. Did not work with --with-cflags=-fPIC but --with-ccopts=-fPIC did work on e2fsprog-lib-1.41.1 Need to test it now. laptop1 uuid # scanelf -a tst_uuid TYPE PAX PERM ENDIAN STK/REL/PTL TEXTREL RPATH BIND FILE ET_DYN ---xe- 0755 LE RW- R-- RW- - - NOW tst_uuid laptop1 uuid # looks okey Created attachment 165429 [details, diff]
New ebuild to fix this bug
Working ebuild for e2fsprogs
Created attachment 165538 [details]
fix some errors in the ebuild
if [[ gcc-version..... did work.
Created attachment 165539 [details, diff]
ebuild patch
Created attachment 165541 [details, diff]
ebuild patch
fix some more.
Created attachment 165543 [details, diff]
ebuild patch
sorry wrong patch last time.
If we have to make *.a files contain pic code then the toolchain sounds like it's doing something incorrectly and or a .so is not being created and linked to when it should be. SpanKY will most likely reject pic patches to *.a files unless that *.a file is used directly to make a .so vs just a say a pie. Solar, reading through http://www.gentoo.org/proj/en/hardened/hardened-toolchain.xml, it sounds like what *should* happen is that object files that are meant for static libraries and executables should be compiled with -fPIE, and shared libraries should be compiled with -fpic, as before. Furthermore, while compiling all objects with -fpic would work, the generated object code will not be as efficient since some symbol references would get vectored through the ELF global object table that isn't strictly necessary. (It also preserves the same semantics as static linking since with static linking LD_PRELOAD libraries can't intercept function/variable bindings that were static linked, symbols referenced via shared libraries.) Am I understanding things correctly? If so, then what we probably need is configure flags to pass compile and static link options when building static libraries as well as separate flags to pass specific compile and link flags that would only be used when building executables, for maximum flexibility and clarity of what these flags actually do. Is that right? If so, I have no problem accepting such a patch upstream for e2fsprogs, if this would make distributions' lives easier. I'm not really convinced that e2fsprogs needs to be built with the hardened toolchain, but what the heck, it's not normally run in CPU bound situations anyway. (Although that might change with solid state disks....) Theodore way is tst_uuid liked agenst libuuid.a instead of libuuid.so ? tst_uuid.o: $(srcdir)/tst_uuid.c @echo " CC $@" @$(CC) $(ALL_CFLAGS) -c $(srcdir)/tst_uuid.c -o tst_uuid.o tst_uuid: tst_uuid.o $(DEPSTATIC_LIBUUID) @echo " LD $@" @$(CC) $(ALL_LDFLAGS) -o tst_uuid tst_uuid.o $(STATIC_LIBUUID) tst_uuid is linked against libuuid.a since it's a test program and it makes it easier to run it out of the tree without needing to specify an LD_LIBRARY_PATH variable. I'll note that in order to build a static e2fsck, we link against the static libraries as well. If the hardened gcc patches make that difficult, then it's more stuff that people will need to get right before it doesn't break on various random applications' build systems.... Somehow, the marketing tagline, "hardened gcc --- it will break all of your static libraries" doesn't sound that good.... Solar seems to claim that if .a files need -pic then the toolchain is probably doing something wrong. Can someone please come up with comprehensive rules about exactly what constraints hardened gcc is inflicting on applications, and make sure everyone agrees with them? This is starting to get a little annoying.... (In reply to comment #24) > what *should* happen is that object files that are meant for static libraries > and executables should be compiled with -fPIE, and shared libraries should be > compiled with -fpic, as before. > Am I understanding things correctly? > That sounds right to me (although -fPIC is commonly needed on amd64) Sorry for my confusion with -fPIC which solar was rightly highlighted as an issue, since it's the wrong flag. > If so, then what we probably need is configure flags to pass compile and static > link options when building static libraries as well as separate flags to pass > specific compile and link flags that would only be used when building > executables, for maximum flexibility and clarity of what these flags actually > do. Is that right? > Yes, although it needs to be borne in mind that an object which is used in both contexts will need to be built twice, with -fPIC for .so and -fPIE for static libs or executables. > If so, I have no problem accepting such a patch upstream for e2fsprogs, if this > would make distributions' lives easier. I'm not really convinced that > e2fsprogs needs to be built with the hardened toolchain, but what the heck, > it's not normally run in CPU bound situations anyway. (Although that might > change with solid state disks....) > If we can agree on the variables, I'm sure we can start to work on patching, though I think it will most likely end up needing your knowledge of the build. As you say, providing separate options for all three, .so, .a and executable would be most flexible, although it's not needed in this case. AIUI libtool keeps separate objects according to where they'll be used (I'm not a user of it) via faked libtool objects, so perhaps the cleanest solution for hardened would be to target libtool to ensure it uses -fPIE for static libs. (In reply to comment #27) > (In reply to comment #24) > > what *should* happen is that object files that are meant for static libraries > > and executables should be compiled with -fPIE, and shared libraries should be > > compiled with -fpic, as before. > > Am I understanding things correctly? > > > That sounds right to me (although -fPIC is commonly needed on amd64) Sorry for > my confusion with -fPIC which solar was rightly highlighted as an issue, since > it's the wrong flag. > > > If so, then what we probably need is configure flags to pass compile and static > > link options when building static libraries as well as separate flags to pass > > specific compile and link flags that would only be used when building > > executables, for maximum flexibility and clarity of what these flags actually > > do. Is that right? > > > Yes, although it needs to be borne in mind that an object which is used in both > contexts will need to be built twice, with -fPIC for .so and -fPIE for static > libs or executables. > > > If so, I have no problem accepting such a patch upstream for e2fsprogs, if this > > would make distributions' lives easier. I'm not really convinced that > > e2fsprogs needs to be built with the hardened toolchain, but what the heck, > > it's not normally run in CPU bound situations anyway. (Although that might > > change with solid state disks....) > > > If we can agree on the variables, I'm sure we can start to work on patching, > though I think it will most likely end up needing your knowledge of the build. > As you say, providing separate options for all three, .so, .a and executable > would be most flexible, although it's not needed in this case. > > AIUI libtool keeps separate objects according to where they'll be used (I'm not > a user of it) via faked libtool objects, so perhaps the cleanest solution for > hardened would be to target libtool to ensure it uses -fPIE for static libs. As of right now I don't see any changes that need to be made to e2fsprogs. I would assert there is an error at this time in the gcc-4.x experimental branch they are using. *.a files should not be compiled with -fPIE as that will likely needlessly lead to TEXTREL's in applications. If this same bug presents itself using Gentoo's hardened gcc-3.x compiler then reopen. Otherwise. We don't handle bug reports for things not even in the tree. (In reply to comment #28) > As of right now I don't see any changes that need to be made to e2fsprogs. I > would assert there is an error at this time in the gcc-4.x experimental branch > they are using. *.a files should not be compiled with -fPIE as that will likely > needlessly lead to TEXTREL's in applications. "if an ebuild creates both executables and libraries then more detailed modifications need to be made, to add the -fno-PIE to the compilation of objects destined for the libraries. Where an object is used for both a shared library and an executable, it is necessary to modify the build process significantly in order to obtain two objects, one built -fPIC and one built -fPIE for linking to the library and the executable respectively."[1] "Static libraries contain non-PIC objects (normal/non-PIE systems only)" [2] "Most packages that provide both a shared library and a static archive do so by using libtool which does the right thing automatically" So you may be right about it not being anything to do with upstream, in that we can fix it via some libtool/automake foo. One would think it cogent however to the software, both for Gentoo users and on other distros or installs using gcc. Certainly if it were my project and such a fix was required, I'd want to know about it. > If this same bug presents itself > using Gentoo's hardened gcc-3.x compiler then reopen. Otherwise. We don't > handle bug reports for things not even in the tree. > Well hardened Gentoo is being left radically behind; one would think you'd welcome some help getting it into the 21st Century, rather than being dismissive. CANTFIX seems inappropriate, when Zorry has prima facie got it working, at least to the stage where more QA would be needed. [1] http://www.gentoo.org/proj/en/hardened/hardened-toolchain.xml#doc_chap3 [2] http://www.gentoo.org/proj/en/hardened/pic-fix-guide.xml#doc_chap5 This error is gcc thing. http://gcc.gnu.org/ml/gcc-patches/2005-03/msg00923.html trigger this. Sorry for all trubel. |