thinkpad ~ # emerge -1 -vD sbcl Calculating dependencies... done! >>> Emerging (1 of 1) dev-lisp/sbcl-0.9.18 to / * sbcl-0.9.14-x86-linux-binary.tar.bz2 MD5 ;-) ... [ ok ] ... * checking sbcl-0.9.18-source.tar.bz2 ;-) ... [ ok ] * checking sbcl-0.9.14-x86-linux-binary.tar.bz2 ;-) ... [ ok ] * * So-called "hardened" compiler features are incompatible with SBCL. You * must use gcc-config to select a profile with non-hardened features * (the "vanilla" profile) and "source /etc/profile" before continuing. * !!! ERROR: dev-lisp/sbcl-0.9.18 failed. thinkpad ~ # gcc-config -l [1] i686-pc-linux-gnu-3.4.6 * [2] i686-pc-linux-gnu-3.4.6-hardened [3] i686-pc-linux-gnu-3.4.6-hardenednopie [4] i686-pc-linux-gnu-3.4.6-hardenednopiessp [5] i686-pc-linux-gnu-3.4.6-hardenednossp [6] i686-pc-linux-gnu-4.1.1 thinkpad ~ # gcc --version gcc (GCC) 3.4.6 (Gentoo 3.4.6-r2, ssp-3.4.6-1.0, pie-8.7.9) Copyright (C) 2006 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. gcc is not hardened, the same happens with i686-pc-linux-gnu-4.1.1. Juergen
lisp folks, not really sure what you are after but the check plain won't work, it will either produce false positives as in this bug, or it will miss a real hardened compiler and presumably bomb out in src_compile() then (hint - see the package.use.mask file in default-linux profile).
Can someone from the "hardened" team give a detailed solution to this problem. I don't use a hardened whatever.
I suspect sbcl is doing unusual things with the compiler, similar to other langauges that use gcc to do some of the work for them, and in so doing is making assumptions about the compiler that fail in a Gentoo hardened environment. OK; first off, sarcastic comments ("So-called hardened") are not appreciated; such childish behaviour should be avoided in ebuilds. You may not care what Hardened Gentoo does, but that's no reason for pejorative language. Secondly, telling people to switch compilers with gcc-config is irresponsible; people using the hardened compiler are using it to implement various modifications to standard system behaviour as part of a range of protections for their systems, and telling them to switch to vanilla is effectively telling them to weaken the protections they would otherwise have in place. Lastly, and probably most importantly, if you have a report of problems involving hardened, _please_ CC: the bug to hardened@g.o (or even re-assign to hardened if you wish) - don't implement nasty hacks like this. FI there are four behaviours that are modified in a hardened toolchain: 1) It builds position-independent executables by default (a bit like having PIC in executables as well as shared libraries) 2) It includes the stack protector, on for all functions by default 3) It builds everything with immediate binding (BIND_NOW) and RELRO There are functions in toolchain-funcs.eclass to detect which hardened features are active in the toolchain. However we need to know what fails and how, before suggesting the correct ebuild behaviour. I'll have a go at 0.9.18, see what fails.
It is not intended as sarcasm. We are reffering to the class of solutions used to limit the damage errant programs (usually ones in C) can cause. Thats GCC features as well as kernel features. Now this hack in pkg_setup has been around for quite a while and followed earlier bugs (88399 from April 2005) about the problem hitherto missed by hardened@gentoo.org.
Created attachment 102029 [details, diff] Additions to deal with hardened toolchain, and PaX kernels Here goes; attempt #1. Here's a run-down of the changes: Global ------ Addition of QA_EXECSTACK_X86 definition This tells the QA checks to expect the files listed to need executable stack, and therefore to omit the warning. Those who absolutely can't accept executable stack can set QA_STRICT_EXECSTACK to get the warnings back (errors if FEATURES=stricter). I can only test on x86 at the moment, so I've only added the definition for X86. Similar definitions may prove necessary for other arches - just look at emerge logs on those arches. Common files can be listed in QA_EXECSTACK - arch-specific ones in QA_EXECSTACK_<arch>. src_unpack() ------------ Added pax-marking of the bootstrap compiler, if the host system is running a PaX-enabled kernel. This uses the (new ;)) eclass pax-utils.eclass. Also for PaX, the final sbcl binary needs to be similarly marked. Normally this would be done by adding a couple of things to the link command, but that won't work here as we don't quite have enough options in the toolchain to get what we need, as sbcl needs all of the PaX restrictions to be relaxed. I think this is because sbcl doesn't call mprotect() or similar to mark appropriate pages executable before jumping to code created on the stack and/or heap. Since the created sbcl is then used to build stuff again, it has to be marked half-way through the build. Unfortunately, at the moment we don't have a simple one-stop-shop for PaX marking on the host; depends too much on the host configuration. The pax-mark() function in pax-utils.eclass deals with this, but of course that can't be called directly from the sbcl build scripts. So what I've done is setup a background process from the ebuild to call the pax-mark() function, with some file-based synchronisation with the main build process. A little unusual, and not ideal, but it works and should be reliable. For the hardened compiler, it turns out that just PIE needs to be missed out. The sbcl build scripts don't pass through CFLAGS to the compiler, so there are some sed commands to make that work. This is a change to the existing behaviour; in particular it may be useful to consider adding a call to strip-flags() in src_compile(), to drop silly flags (see flag-o-matic.eclass for details). src_compile() ------------- This now includes a call: filter-flags -fPIE which automatically deals with the hardened toolchain (once CFLAGS is passed though to compilation and link commands, thanks to the seds in src_unpack()). If a pie-default compiler is being used, filter-flags works that out and adds stuff to CFLAGS to inhibit the pie-default behaviour. A call to pax-mark is added to mark the final executable. This may seem a little redundant, but the concept is that the previous markings were to manage the build process, and are only done if the host is running a PaX-enabled kernel. Marking the final executable is unconditional, so that delivering the built package to other machines will work. Lastly, one little tidyup; I changed '! use nosource' to 'use source' to agree with the USE flag name listed in IUSE (solves a QA notice about 'nosource' being missing from IUSE).
Oh; one more comment on the unusual mid-build PaX thing - if sbcl did use mprotect() et. al. properly to mark pages on which it generates code as executable, we could drop the hoopy stuff and simply add some flags to the link command (i.e. -Wl,z,execstack). Might be worth taking that up with upstream; it may start failing anyway if the mainline kernel starts implemnting executable protection (something that can be done easily with the NX bit now found on x64_64 and some x86_32 processors).
(In reply to comment #5) > This tells the QA checks to expect the files listed to need executable stack, > and therefore to omit the warning. Those who absolutely can't accept > executable stack can set QA_STRICT_EXECSTACK to get the warnings back (errors > if FEATURES=stricter). Here I still get the warning (without strict as far as I can see): # cat /var/tmp/portage/dev-lisp/sbcl-1.0.6/temp/scanelf-execstack.log !WX --- --- work/sbcl-1.0.6/src/runtime/x86-assem.o !WX --- --- work/sbcl-1.0.6/src/runtime/ldso-stubs.o RWX --- --- work/sbcl-1.0.6/src/runtime/sbcl RWX --- --- image/usr/bin/sbcl Emerge works though.
I'm not sure what to do about this. Does the issue still exist? If so, what would be a good solution? Please use >=sbcl-1.0.26 for testing.
(In reply to comment #8) > I'm not sure what to do about this. > Does the issue still exist? > If so, what would be a good solution? > > Please use >=sbcl-1.0.26 for testing. my portage has .19 as latest...
This issue still happens with sbcl-1.0.26-r10 --- gcc (Gentoo 4.3.2-r2 p1.5, pie-10.1.5) 4.3.2 x86_64-pc-linux-gnu Portage 2.1.6.11 (hardened/linux/amd64/2008.0, gcc-4.3.2, glibc-2.9_p20081201-r2, 2.6.28-11-generic x86_64)
if sbcl doesn't generate code at runtime then i suggest an execstack -c on it or something equivalent if portage has the functionality already. otherwise it'll need paxctl -m and the like.
@Facundo: but does sbcl work if you eliminate the check from the ebuild and mark it with paxctl -m ?
@Stelian : After removing the check the ebuild reaches the compile stage but fails. I receive the following error: /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.2/../../../../x86_64-pc-linux-gnu/bin/ld: x86-64-assem.o: relocation R_X86_64_32S against `all_threads' can not be used when making a shared object; recompile with -fPIC x86-64-assem.o: could not read symbols: Bad value collect2: ld returned 1 exit status make: *** [sbcl] Error 1 make: Leaving directory `/var/ tmp/portage/dev-lisp/sbcl-1.0.26-r10/work/sbcl-1.0.26/src/runtime' In any case, I see exactly the same error when trying to build clisp so I suspect that is something in my environment.
(In reply to comment #13) > /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.2/../../../../x86_64-pc-linux-gnu/bin/ld: > x86-64-assem.o: relocation R_X86_64_32S against `all_threads' can not be used > when making a shared object; recompile with -fPIC > x86-64-assem.o: could not read symbols: Bad value just what the error says, that file (based on its name, it's probably written in asm, not C) wasn't compiled as position independent while it was going to be linked into a shared lib or PIE. since the former would have been noticed by upstream, and you're using a hardened profile, i guess it's the latter case. recompile it with the vanilla specs and it should work better (modulo the possible need for paxctl as i mentioned above). the alternative is that someone will have to look at that .o and its source and modify it according to the gentoo PIC guide.
I tried to install sbcl-1.0.28 with i686-pc-linux-gnu-4.3.3-vanilla. It fails with: * //testing for consistency of first and second GENESIS passes //header files match between first and second GENESIS -- good real 20m54.585s user 20m26.569s sys 0m23.125s //entering make-target-2.sh //doing warm init - compilation phase This is SBCL 1.0.28.gentoo-r0, an implementation of ANSI Common Lisp. More information about SBCL is available at <http://www.sbcl.org/>. SBCL is free software, provided as is, with absolutely no warranty. It is mostly in the public domain; some portions are provided under BSD-style licenses. See the CREDITS and COPYING files in the distribution for more information. CORRUPTION WARNING in SBCL pid 22395(tid 3084760768): Memory fault at 1dbc0 (pc=0x1dbc0, sp=0xb7a93f1c) The integrity of this image is possibly compromised. Exiting. The system is too badly corrupted or confused to continue at the Lisp level. If the system had been compiled with the SB-LDB feature, we'd drop into the LDB low-level debugger now. But there's no LDB in this build, so we can't really do anything but just exit, sorry. * * ERROR: dev-lisp/sbcl-1.0.28 failed.
snake ~ # emerge --info Portage 2.1.6.13 (hardened/linux/x86, gcc-4.3.3-vanilla, glibc-2.9_p20081201-r2, 2.6.28-gentoo-r4 i686) ================================================================= System uname: Linux-2.6.28-gentoo-r4-i686-Pentium_III_-Katmai-with-gentoo-2.0.1 Timestamp of tree: Sun, 24 May 2009 23:45:01 +0000 distcc 3.1 i686-pc-linux-gnu [disabled] app-shells/bash: 4.0_p24 dev-java/java-config: 2.1.8 dev-lang/python: 2.5.4-r2, 2.6.2 dev-python/pycrypto: 2.0.1-r8 dev-util/cmake: 2.6.4 sys-apps/baselayout: 2.0.1 sys-apps/openrc: 0.4.3-r2 sys-apps/sandbox: 1.9 sys-devel/autoconf: 2.13, 2.63-r1 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10.2 sys-devel/binutils: 2.19.1-r1 sys-devel/gcc-config: 1.4.1 sys-devel/libtool: 2.2.6a virtual/os-headers: 2.6.29 ACCEPT_KEYWORDS="x86 ~x86" CBUILD="i686-pc-linux-gnu" CFLAGS="-pipe -O3 -march=pentium3 -fomit-frame-pointer" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/config /var/lib/hsqldb" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/eselect/postgresql /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c /etc/udev/rules.d" CXXFLAGS="-pipe -O3 -march=pentium3 -fomit-frame-pointer" DISTDIR="/usr/portage/distfiles" FEATURES="distlocks fixpackages parallel-fetch protect-owned sandbox sfperms strict unmerge-orphans userfetch" FFLAGS="-pipe -O3 -march=pentium3 -fomit-frame-pointer" GENTOO_MIRRORS="ftp://sunsite.informatik.rwth-aachen.de/pub/Linux/gentoo http://linux.rz.ruhr-uni-bochum.de/download/gentoo-mirror http://ftp-stud.fht-esslingen.de/pub/Mirrors/gentoo/ rsync://ftp.fi.muni.cz/pub/linux/gentoo ftp://ftp.wh2.tu-dresden.de/pub/mirrors/gentoo rsync://gd.tuwien.ac.at/opsys/linux/gentoo ftp://sunsite.informatik.rwth-aachen.de/pub/Linux/gentoo http://gentoo.oregonstate.edu http://212.219.247.17/sites/www.ibiblio.org/gentoo/" LDFLAGS="-Wl,-O1" LINGUAS="de fr" MAKEOPTS="-j1" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage/layman/science /usr/local/portage/layman/java-overlay /usr/local/portage/layman/zugaina /usr/local/portage/layman/dotnet /usr/local/portage" SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage" USE="X Xaw3d a52 aac acl administrator afs alsa amrr ao aotuv apache2 asf atlas auctex audacious audiofile automount bash-completion beagle berkdb bjam blas bonobo boo boost bzip2 c++ cairo cdda cddb cdf cdio cdparanoia cdr cgi chm cli consolekit corba cracklib cran crypt curl cxx daap dbus device-mapper dga dia divx djvu dlloader dmi doc dri ds3490 dv dvb dvd dvdr dvi dxr3 dynagraph eds effects elf emacs emboss encode epiphany esd evo examples exif expat extra fam fame ffmpeg fftw firefox fits flac fltk foomaticdb fpx galago gd gdal geos gif gimp gimpprint ginac glade glitz gml gmp gnokii gnome gnome-keyring gnuplot gnutls gphoto2 graphics graphviz gs gsl gsm gstreamer gtk guile hal hardened harness hddtemp hdf hdf5 hlapi http iconv icq icu id3 ide imagemagick imap innodb irda irmc isdnlog ithreads jabber java java5 java6 jbig john jpeg jpeg2k kerberos kexi keyring ladspa lame lapack latex lcms ldap libffi libgda libsamplerate lm_sensors lua lzo lzw mad maildir math matroska matrox midi mmx mmxext mng mod mono motif mozdevelop mozilla mozsvg mozxmlterm mp3 mp4 mp4live mpeg mpeg2 mplayer mtp mudflap musicbrainz mysql mysqli nautilus ncurses neXt netcdf netpbm network networkmanager nfs nntp nptl nptlonly nsplugin ntfs numarray numeric obex ocaml octave odbc ogdi ogg ole opengl pae pam pcre pda pdf perl pic pipechan plotutils plugins png podcast policykit posix postgres postscript pppd preview-latex proj projectx pstricks python qhull quicktime readline reflection reiserfs rhythmbox rle rpc rrdcgi rrdtool samba sasl science sdl session slang slp smp sndfile snmp soup sox speex spell spl sqlite sse ssl stlport subtitles subversion suexec svg sysfs t1lib tcl tcpd tex theora threads thunderbird tidy tiff tk truetype unicode urandom userlocales utempter v4l2 vorbis wav webkit win32codecs wmf wxwindows x86 xattr xcb xemacs xext xine xml xmlreader xmlrpc xorg xpm xulrunner xv xvid xvmc zlib zvbi" ALSA_CARDS="ens1371" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="keyboard mouse " KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="de fr" NETBEANS_MODULES="ide websvccommon nb harness apisupport groovy gsf j2ee java xml" USERLAND="GNU" VIDEO_CARDS="mga vesa fbdev" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
I had that same problem after doing a 'prelink -amR'. You might want to try undo the prelinking on any libraries that the sbcl might use or on all of them using 'prelink -au' (please try this option as a last resource and beware that it might mess some off you binaries up - never happened to me but in any case I would be careful). In any case I think you might be dealing with a new bug and not the one this report refers to.
(In reply to comment #14) > (In reply to comment #13) > > /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.2/../../../../x86_64-pc-linux-gnu/bin/ld: > > x86-64-assem.o: relocation R_X86_64_32S against `all_threads' can not be used > > when making a shared object; recompile with -fPIC > > x86-64-assem.o: could not read symbols: Bad value > > just what the error says, that file (based on its name, it's probably written > in asm, not C) wasn't compiled as position independent while it was going to be > linked into a shared lib or PIE. since the former would have been noticed by > upstream, and you're using a hardened profile, i guess it's the latter case. > recompile it with the vanilla specs and it should work better (modulo the > possible need for paxctl as i mentioned above). the alternative is that someone > will have to look at that .o and its source and modify it according to the > gentoo PIC guide. > I'm seeing this same error, trying to build sbcl with the -vanilla profile. I hacked the sbcl build process to add -fPIC into Config.x86_64-linux, and the build output shows: cc -g -Wall -Wsign-compare -O3 -fno-omit-frame-pointer -fPIC -I. -c -o x86-64-assem.o x86-64-assem.S cc -g -Wall -Wsign-compare -O3 -fno-omit-frame-pointer -fPIC -I. -c -o ldso-stubs.o ldso-stubs.S cc -g -Wl,--export-dynamic -o sbcl alloc.o backtrace.o breakpoint.o coreparse.o dynbind.o funcall.o gc-common.o globals.o interr.o interrupt.o largefile.o monitor.o os-common.o parse.o print.o purify.o pthread-futex.o pthread-lutex.o regnames.o run-program.o runtime.o save.o search.o thread.o time.o util.o validate.o vars.o wrap.o x86-64-arch.o linux-os.o x86-64-linux-os.o gencgc.o x86-64-assem.o ldso-stubs.o -ldl -lpthread -lm /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.1/../../../../x86_64-pc-linux-gnu/bin/ld: x86-64-assem.o: relocation R_X86_64_32S against `all_threads' can not be used when making a shared object; recompile with -fPIC x86-64-assem.o: could not read symbols: Bad value This is sbcl-1.0.30 from the lisp overlay, but the same behavior happens with sbcl-1.0.28 from the base tree. I should also note that I can build sbcl fine with a non-hardened compiler, and unlike a previous comment, I can build clisp fine even with the full hardened profile.
Ok, people, I believe that it's time to mask sbcl in hardened profiles, as it just doesn't work. I have wasted many hours in the past trying to understand its build system, but I don't want to do the same mistake again. Yesterday, I did some tests and sbcl fails even with the *vanilla gcc enabled, so obviously the hardened toolchain check is wrong, for the time being. I will leave the bug open for some days, if you want to say something, say it now, please. For reference, sbcl failed on me, on amd64 when the linker tried to link the x86-64-assem.o(created from x86-64-assem.S(asm file obviously). On x86 it failed when the target sbcl tried to run for first time with a corruption error (on the core image I think).
Created attachment 238727 [details, diff] Fix the build error with the asm files with full hardened compiler This fix the commpile error with the asm files and full hardened compiler. dev-lisp/sbcl-1.0.36-r1 USE="threads -cobalt -doc -ldb -source -unicode" jasmin / # emerge --info Portage 2.2_rc67 (hardened/linux/amd64/10.0, gcc-4.4.4, glibc-2.11.2-r0, 2.6.32-hardened-r4 x86_64) ================================================================= System uname: Linux-2.6.32-hardened-r4-x86_64-Intel-R-_Xeon-R-_CPU_E5420_@_2.50GHz-with-gentoo-2.0.1 Timestamp of tree: Mon, 12 Jul 2010 11:45:02 +0000 app-shells/bash: 4.1_p7 dev-lang/python: 2.6.5-r2, 3.1.2-r3 sys-apps/baselayout: 2.0.1 sys-apps/openrc: 0.6.1-r1 sys-apps/sandbox: 2.2 sys-devel/autoconf: 2.65-r1 sys-devel/automake: 1.11.1 sys-devel/binutils: 2.20.1-r1 sys-devel/gcc: 4.4.4-r1 sys-devel/gcc-config: 1.4.1 sys-devel/libtool: 2.2.10 virtual/os-headers: 2.6.34 ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="*" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -pipe -march=core2" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -pipe -march=core2"
Created attachment 238731 [details, diff] We use flag-o-matic to filter pie stuff Use flag-o-matic to filter hardened flags
Hello, is this package maintained?
Yes
I'd prefer: "Yes, and I've pushed a commit with the workaround of disabling PIE." What's necessary is filter-flags -pie and adding -nopie to LINKFLAGS (not sure if append-ldflags is the right thing here). SSP works well and can be left enabled just fine. PaX issues can be worked on later. And of course mention the issue to upstream, so they can provide PIC assembly (probably optional) if nobody from hardened herd feels like fixing it before that, as well as the two files built without respecting LDFLAGS.
I just pushed sbcl-1.0.45 in the tree. Can someone try to update and tell me if it works on hardened?
(In reply to comment #25) > I just pushed sbcl-1.0.45 in the tree. Can someone try to update and tell me > if it works on hardened? Yeah, from bug 417037 one could determine that 1.0.55-r1 is at at least OK. If you can still reproduce with current stable, 1.0.55-r1, bug 326217, please reopen the bug.
