Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 339914 - app-emulation/virtualbox: fails to compile with hardened gcc
Summary: app-emulation/virtualbox: fails to compile with hardened gcc
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: High normal
Assignee: Patrick Lauer
URL:
Whiteboard:
Keywords:
: 389393 (view as bug list)
Depends on:
Blocks: 350475
  Show dependency tree
 
Reported: 2010-10-06 07:45 UTC by Jaak Ristioja
Modified: 2012-02-16 17:43 UTC (History)
9 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
build.log (build.log,69.07 KB, text/plain)
2010-10-06 07:46 UTC, Jaak Ristioja
Details
Patch to add -fno-pic to C[XX]FLAGS for vbox (vbox-fnopic.patch,1.58 KB, text/plain)
2011-01-03 23:37 UTC, Anthony Basile
Details
Add -nopie (virtualbox_nopie.patch,12.06 KB, patch)
2011-10-26 21:04 UTC, Magnus Granberg
Details | Diff
Updated nopie patch (virtualbox_nopie.patch,11.16 KB, patch)
2012-02-13 18:42 UTC, Magnus Granberg
Details | Diff
diff on the ebuild (virtualbox_ebuild.patch,1.11 KB, patch)
2012-02-13 18:46 UTC, Magnus Granberg
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Jaak Ristioja 2010-10-06 07:45:55 UTC
USE="additions alsa hal opengl qt4 -headless -pulseaudio -python -sdk -vboxwebsrv"

/var/tmp/portage/app-emulation/virtualbox-ose-3.2.8/work/VirtualBox-3.2.8_OSE/src/VBox/Runtime/common/log/logellipsis.cpp:1: error: code model 
kernel does not support PIC mode
/var/tmp/portage/app-emulation/virtualbox-ose-3.2.8/work/VirtualBox-3.2.8_OSE/src/VBox/Runtime/common/log/logellipsis.cpp:1: error: code model 
'kernel' not supported in the 64 bit mode

Portage 2.1.8.3 (hardened/linux/amd64/10.0, gcc-4.4.4, glibc-2.11.2-r0, 2.6.34-hardened-r6-arm x86_64)
=================================================================
System uname: Linux-2.6.34-hardened-r6-arm-x86_64-Intel-R-_Core-TM-2_Duo_CPU_P8700_@_2.53GHz-with-gentoo-1.12.13
Timestamp of tree: Tue, 05 Oct 2010 19:00:01 +0000
distcc 3.1 x86_64-pc-linux-gnu [disabled]
ccache version 2.4 [disabled]
app-shells/bash:     4.0_p37
dev-lang/python:     2.6.5-r3, 3.1.2-r4
dev-util/ccache:     2.4-r7
dev-util/cmake:      2.8.1-r2
sys-apps/baselayout: 1.12.13
sys-apps/sandbox:    2.3-r1
sys-devel/autoconf:  2.13, 2.65-r1
sys-devel/automake:  1.8.5-r4, 1.9.6-r3, 1.10.3, 1.11.1
sys-devel/binutils:  2.20.1-r1
sys-devel/gcc:       4.4.4-r2
sys-devel/gcc-config: 1.4.1
sys-devel/libtool:   2.2.10
sys-devel/make:      3.81-r2
virtual/os-headers:  2.6.34
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="@FREE"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe -march=native -ggdb -floop-interchange -floop-strip-mine -floop-block"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/X11/xkb /usr/share/config"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/eselect/postgresql /etc/fonts/fonts.conf /etc/gconf /etc/revdep-rebuild /etc/sandbox.d /etc/splash /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-O2 -pipe -march=native -ggdb -floop-interchange -floop-strip-mine -floop-block"
DISTDIR="/usr/portage/distfiles"
FEATURES="assume-digests collision-detect distlocks fixpackages installsources news parallel-fetch protect-owned sandbox sfperms splitdebug strict unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en et et_EE"
MAKEOPTS="-j3"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_EXTRA_OPTS="--exclude-from=/etc/portage/rsync_excludes"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
USE="64bit X a52 aac acl acpi alsa amd64 amr aspell bindist boost bzip2 cairo caps cdda cdr cli consolekit cracklib crypt cups custom-optimization cxx dbus directfb doc dri dvd dvdr dvdread eds emboss enca encode evo exif faac faad fam fbcon ffmpeg firefox flac fortran gif git glitz gmp gnuplot gnutls gphoto2 gpm gstreamer gtk hal hardened hdri htmlhandbook iconv icu id3tag idn imagemagick ipv6 jpeg justify kate kde kontact kpathsea lapack laptop latex lcms lua lzma lzo mad mailwrapper matroska md5sum midi mikmod mmap mmx mng modules mp3 mp4 mpeg mudflap multilib ncurses networkmanager nptl nptlonly ogg openal openexr opengl openmp pam pcre pdf phonon pic plasma png policykit ppds pppd qt4 readline reflection rrdtool rtmp schroedinger sdl semantic-desktop session smp snmp source speex spell spl sqlite sse sse2 ssl ssse3 startup-notification subversion svg sysfs system-sqlite tcpd theora threads tiff truetype ucs2 unicode urandom usb vaapi vorbis webkit x264 xattr xcb xcomposite xetex xft xinerama xorg xosd xprint xscreensaver xulrunner xv xvid zlib" ELIBC="glibc" KERNEL="linux" LINGUAS="en et et_EE" USERLAND="GNU"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LANG, LC_ALL, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS
Comment 1 Jaak Ristioja 2010-10-06 07:46:20 UTC
Created attachment 249724 [details]
build.log
Comment 2 Jaak Ristioja 2010-10-08 07:30:47 UTC
Same error with unstable 3.2.6 and stable 3.1.8.
Comment 3 Jaak Ristioja 2010-10-08 09:09:37 UTC
# gcc-config -l
 [1] x86_64-pc-linux-gnu-4.4.4
 [2] x86_64-pc-linux-gnu-4.4.4-hardenednopie *
 [3] x86_64-pc-linux-gnu-4.4.4-hardenednopiessp
 [4] x86_64-pc-linux-gnu-4.4.4-hardenednossp
 [5] x86_64-pc-linux-gnu-4.4.4-vanilla

Compilation with x86_64-pc-linux-gnu-4.4.4 (hardened?) does not succeed, but using vanilla or hardenednopie works.
Comment 4 Jaak Ristioja 2010-10-12 21:57:41 UTC
Same error with unstable 3.2.10.

Comment 5 Giorgos Tsiapaliwkas 2010-10-13 21:42:25 UTC
i also receive the same error for 3.2.10.

any updated about the specific bug?
Comment 6 Magnus Granberg gentoo-dev 2010-10-14 15:56:13 UTC
The -mcmodel=kernel compiler command options fail when the compiler add -fPIE/-fPIC
We did have the same error in the kernel ssp check for amd64 see bug #312335
You should disable -fPIE with 
if gcc-specs-pie ; then 
 filter-flags -fPIE
 append-ldflags -nopie
fi
Comment 7 Lars Wendler (Polynomial-C) (RETIRED) gentoo-dev 2010-11-01 17:19:47 UTC
@hardened: any idea/suggestions what to do here?
Comment 8 Xake 2010-11-01 19:20:27 UTC
(In reply to comment #7)
> @hardened: any idea/suggestions what to do here?
> 

Magnus (zorry in comment 6) is the current hardened lead, so I think you already got that comment/suggestion?
Comment 9 Jaak Ristioja 2010-11-01 20:40:18 UTC
(In reply to comment #6)
> The -mcmodel=kernel compiler command options fail when the compiler add
> -fPIE/-fPIC
> We did have the same error in the kernel ssp check for amd64 see bug #312335
> You should disable -fPIE with 
> if gcc-specs-pie ; then 
>  filter-flags -fPIE
>  append-ldflags -nopie
> fi

Oh ye Good Sire! Thy words of wisdom are too high for us, mere lowly mortals, to grasp. Canst Thou please merely mend it, we beg thou?
Comment 10 Xake 2010-11-01 21:37:40 UTC
(In reply to comment #9)
> Oh ye Good Sire! Thy words of wisdom are too high for us, mere lowly mortals,
> to grasp. Canst Thou please merely mend it, we beg thou?
> 

Oh, go away. If you cannot understand that it as not for you.
Comment 11 Xake 2010-11-01 21:49:30 UTC
(In reply to comment #7)
> @hardened: any idea/suggestions what to do here?
> 

Parts of the buildsystem does not seems to listen to C[XX]FLAGS at all, which makes current strategies to filter PIC/PIE hard.
This means that we either needs to patch the build-system to be able to add flags to those parts, or find out which variable we may for hardened add -nopie to get it to those parts.
Comment 12 Anthony Basile gentoo-dev 2011-01-03 20:23:56 UTC
This affects 3.2.12 as well.  I tried to hack away at the kbuild system yesterday without success --- I'm just not familiar enough with kbuild.  It does not respect C[XX]FLAGS the way GNU autobuild does, but requires something like objfile_CFLAGS += "-nopie".  We need to get that flag in when compiling at least the following:

common/log/logellipsis.cpp
common/log/logrelellipsis.cpp
common/log/logcom.cpp
common/log/logformat.cpp
common/misc/RTAssertMsg1Weak.cpp
common/misc/RTAssertMsg2.cpp
common/misc/RTAssertMsg2Add.cpp

As a side note: I agree with Patrick and can only hope that upstrem will consider moving to a more sane build system like cmake.
Comment 13 Anthony Basile gentoo-dev 2011-01-03 23:37:31 UTC
Created attachment 258772 [details]
Patch to add -fno-pic to C[XX]FLAGS for vbox

This patch is a slight rework of the patch in bug #264980.  It does solve the compile time error, but leaves two textrels behind:

    /usr/lib64/virtualbox-ose/VBoxVMM.so
    /usr/lib64/virtualbox-ose/components/VBoxVMM.so
Comment 14 Anthony Basile gentoo-dev 2011-01-07 00:28:13 UTC
(In reply to comment #13)
> Created an attachment (id=258772) [details]
> Patch to add -fno-pic to C[XX]FLAGS for vbox

This patch does not work.  The correct solution was to use:

  gcc-spec-pie && append-flags "-nopie"

just before running the configure script in the ebuild.
Comment 15 Anthony Basile gentoo-dev 2011-01-07 00:34:19 UTC
This is fixed as of virtualbox-ose-3.2.12-r2.ebuild

I'm closing this one for now.  Please reopen if there are any further issues.
Comment 16 Vincent de Phily 2011-01-17 19:51:17 UTC
This still happens to me as of app-emulation/virtualbox-3.2.12-r4 and app-emulation/virtualbox-4.0.0-r1. The build error is very much the same :

  CXX     RuntimeR0 - {C}/src/VBox/Runtime/common/log/logellipsis.cpp
/home/portagetmp/portage/app-emulation/virtualbox-3.2.12-r4/work/VirtualBox-3.2.12_OSE/src/VBox/Runtime/common/log/logellipsis.cpp:1: error: code model kernel does not support PIC mode
/home/portagetmp/portage/app-emulation/virtualbox-3.2.12-r4/work/VirtualBox-3.2.12_OSE/src/VBox/Runtime/common/log/logellipsis.cpp:1: error: code model 'kernel' not supported in the 64 bit mode
kmk[2]: *** [/home/portagetmp/portage/app-emulation/virtualbox-3.2.12-r4/work/VirtualBox-3.2.12_OSE/out/linux.amd64/release/obj/RuntimeR0/common/log/logellipsis.o] Error 1
The failing command:
@x86_64-pc-linux-gnu-g++ -c -O2 -nostdinc -g -pipe  -pedantic -Wshadow -Wall -Wextra -Wno-missing-field-initializers -Wno-unused -Wno-trigraphs -fdiagnostics-show-option -Wno-long-long -Wno-variadic-macros -O2 -mtune=generic -fno-omit-frame-pointer -fno-strict-aliasing -fno-exceptions -fno-stack-protector -fvisibility-inlines-hidden -fvisibility=hidden -DVBOX_HAVE_VISIBILITY_HIDDEN -DRT_USE_VISIBILITY_DEFAULT -fno-rtti -m64 -mno-red-zone -mcmodel=kernel -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -fno-asynchronous-unwind-tables -I/home/portagetmp/portage/app-emulation/virtualbox-3.2.12-r4/work/VirtualBox-3.2.12_OSE/src/VBox/Runtime/include -I/home/portagetmp/portage/app-emulation/virtualbox-3.2.12-r4/work/VirtualBox-3.2.12_OSE/include/iprt/nocrt -I/home/portagetmp/portage/app-emulation/virtualbox-3.2.12-r4/work/VirtualBox-3.2.12_OSE/include -I/home/portagetmp/portage/app-emulation/virtualbox-3.2.12-r4/work/VirtualBox-3.2.12_OSE/out/linux.amd64/release -DVBOX -DVBOX_OSE -DVBOX_WITH_64_BITS_GUESTS -DVBOX_WITH_HARDENING -DRTPATH_APP_PRIVATE=\"/usr/share/virtualbox\" -DRTPATH_APP_PRIVATE_ARCH=\"/usr/lib64/virtualbox\" -DRTPATH_SHARED_LIBS=\"/usr/lib64/virtualbox\" -DRTPATH_APP_DOCS=\"\" -DRT_OS_LINUX -D_FILE_OFFSET_BITS=64 -DRT_ARCH_AMD64 -D__AMD64__ -DIN_RING0 -DIN_RING0_AGNOSTIC -DIPRT_NO_CRT -DRT_WITH_NOCRT_ALIASES -DHC_ARCH_BITS=64 -DGC_ARCH_BITS=64 -DIN_RT_R0 -DRT_WITH_VBOX -Wp,-MD,/home/portagetmp/portage/app-emulation/virtualbox-3.2.12-r4/work/VirtualBox-3.2.12_OSE/out/linux.amd64/release/obj/RuntimeR0/common/log/logellipsis.o.dep -Wp,-MT,/home/portagetmp/portage/app-emulation/virtualbox-3.2.12-r4/work/VirtualBox-3.2.12_OSE/out/linux.amd64/release/obj/RuntimeR0/common/log/logellipsis.o -Wp,-MP -o /home/portagetmp/portage/app-emulation/virtualbox-3.2.12-r4/work/VirtualBox-3.2.12_OSE/out/linux.amd64/release/obj/RuntimeR0/common/log/logellipsis.o /home/portagetmp/portage/app-emulation/virtualbox-3.2.12-r4/work/VirtualBox-3.2.12_OSE/src/VBox/Runtime/common/log/logellipsis.cpp


Not sure what to add from here (kernel config ?), as this bug looked like it was nicely fixed.

I do not have enough permissions to reopen the bug.
Comment 17 Anthony Basile gentoo-dev 2011-01-17 20:51:15 UTC
I know, I still haven't got it. *sigh*
Comment 18 Anthony Basile gentoo-dev 2011-03-29 18:41:10 UTC
Okay, rather than waste time on a brain dead build system, I've decided to add a workaround for people using a hardened gcc.  If you're using a hardened gcc, then pick on of the following profiles using gcc-config and it will compile fine:

    hardenednopie
    hardenednopiessp
    vanilla

(remember after switching to run source /etc/profile.)  The preferred profile is, however hardenednopie since it gives the most hardening while still working.

I've also added this message to the 4.0.4-r1.  It is triggered if the user is trying to compile with a profile that is known to fail.

When/if this is fixed, we'll remove the message.
Comment 19 Henrik Ala-Uotila 2011-05-09 10:45:58 UTC
(In reply to comment #18)
> Okay, rather than waste time on a brain dead build system, I've decided to add
> a workaround for people using a hardened gcc.  If you're using a hardened gcc,
> then pick on of the following profiles using gcc-config and it will compile
> fine:
> 
>     hardenednopie
>     hardenednopiessp
>     vanilla
> 
> (remember after switching to run source /etc/profile.)

It seems the current detection method relies on the system wide gcc-config option instead of the actually used GCC spec.

I've replaced the bit:
> gcc-config -c | grep -qv -E "hardenednopie|vanilla"
with:
> echo "${GCC_SPECS}" | grep -qv -E "hardenednopie|vanilla"
in my local overlay, after which the message is triggered by the compiler which is presently sourced rather than the one that is listed by gcc-config.

This also allows the workaround for putting:
> GCC_SPECS="/usr/lib/gcc/x86_64-pc-linux-gnu/4.4.5/hardenednopie.specs"
in /etc/portage/env/app-emulation/virtualbox to avoid going through the gcc-config & source cycle on every install.
Comment 20 Anthony Basile gentoo-dev 2011-05-09 15:08:15 UTC
(In reply to comment #19)
> (In reply to comment #18)
> > Okay, rather than waste time on a brain dead build system, I've decided to add
> > a workaround for people using a hardened gcc.  If you're using a hardened gcc,
> > then pick on of the following profiles using gcc-config and it will compile
> > fine:
> > 
> >     hardenednopie
> >     hardenednopiessp
> >     vanilla
> > 
> > (remember after switching to run source /etc/profile.)
> 
> It seems the current detection method relies on the system wide gcc-config
> option instead of the actually used GCC spec.
> 
> I've replaced the bit:
> > gcc-config -c | grep -qv -E "hardenednopie|vanilla"
> with:
> > echo "${GCC_SPECS}" | grep -qv -E "hardenednopie|vanilla"
> in my local overlay, after which the message is triggered by the compiler which
> is presently sourced rather than the one that is listed by gcc-config.
> 
> This also allows the workaround for putting:
> > GCC_SPECS="/usr/lib/gcc/x86_64-pc-linux-gnu/4.4.5/hardenednopie.specs"
> in /etc/portage/env/app-emulation/virtualbox to avoid going through the
> gcc-config & source cycle on every install.

Nice thanks.  I was worried about people forgetting to switch back and forth.  I know I do :)
Comment 21 Henrik Ala-Uotila 2011-05-19 09:47:09 UTC
> I've replaced the bit:
> > gcc-config -c | grep -qv -E "hardenednopie|vanilla"
> with:
> > echo "${GCC_SPECS}" | grep -qv -E "hardenednopie|vanilla"

Any plans on implementing something similar for the Gentoo virtualbox ebuilds?
Comment 22 Anthony Basile gentoo-dev 2011-05-19 10:12:18 UTC
(In reply to comment #21)
> > I've replaced the bit:
> > > gcc-config -c | grep -qv -E "hardenednopie|vanilla"
> > with:
> > > echo "${GCC_SPECS}" | grep -qv -E "hardenednopie|vanilla"
> 
> Any plans on implementing something similar for the Gentoo virtualbox ebuilds?

Normally I would say no.  The build system should be patched and the patch should go upstream.  But build system is unworkably messy and upstream doesn't care.  So, its up to Patrick.
Comment 23 Henrik Ala-Uotila 2011-05-19 10:40:53 UTC
(In reply to comment #22)
> The build system should be patched and the patch should go upstream.

Agreed. However, the suggested change in question is merely fine tuning the already implemented workaround introduced in comment #18.
Comment 24 Henrik Ala-Uotila 2011-05-19 11:00:54 UTC
(In reply to comment #23)
> (In reply to comment #22)
> > The build system should be patched and the patch should go upstream.
> 
> Agreed. However, the suggested change in question is merely fine tuning the
> already implemented workaround introduced in comment #18.

More accurately, it fixes a deficiency introduced by the workaround described in comment #18. When the GCC spec is set to hardenednopie in /etc/portage/env/app-emulation/virtualbox, the current detection method triggers an error (as gcc-config may show something entirely different than the currently used GCC spec) while the ebuild would otherwise work fine.
Comment 25 Magnus Granberg gentoo-dev 2011-10-26 21:04:07 UTC
Created attachment 290929 [details, diff]
Add -nopie

This patch add -nopie if the compiler works with it.
but the package still fail with hardened kernel.
Comment 26 Jeroen Roovers (RETIRED) gentoo-dev 2011-11-03 15:31:06 UTC
*** Bug 389393 has been marked as a duplicate of this bug. ***
Comment 27 Magnus Granberg gentoo-dev 2012-02-13 18:42:06 UTC
Created attachment 301797 [details, diff]
Updated nopie patch

This patch make it compile and install
So test it.
Comment 28 Magnus Granberg gentoo-dev 2012-02-13 18:46:13 UTC
Created attachment 301799 [details, diff]
diff on the ebuild

Patch for the ebuild
Comment 29 Magnus Granberg gentoo-dev 2012-02-16 17:43:53 UTC
Virtualbox-4.1.8-r1 have the fix