I am running a hardened system, and am using 3 nonstandard kernel modules, namely sys-fs/fuse, net-wireless/rt2500 and sys-fs/unionfs. The first two break when they are compiled using distcc. the last one seems to behave fine. Also, when I watch 'top -u distcc' on the other distcc host, there doesn't seem to happen anything. Which leads to my conclusion that the problem is with distcc itself, and not some weird problem caused by the fact that it's actually compiled on another machine. Reproducible: Always Steps to Reproduce: Scenario: - machine running hardened profile and hardened-sources (pax features enabled, not sure if that is part of the problem). - use genkernel for a new kernel - emerge mentioned modules (distcc enabled), this will succeed without any problems - reboot and modprobe modules Result: tomhendr@elaine ~ $ sudo modprobe fuse FATAL: Error inserting fuse (/lib/modules/2.6.18-hardened/fs/fuse.ko): Unknown symbol in module, or unknown parameter (see dmesg) and from dmesg: fuse: Unknown symbol _GLOBAL_OFFSET_TABLE_ after this, run: # FEATURES="-distcc" emerge sys-fs/fuse # modprobe -r fuse # modprobe fuse Now it loads fine. dmesg tells: fuse init (API version 7.8) fuse distribution version: 2.6.1 my emerge --info: tomhendr@elaine ~ $ emerge --info Portage 2.1.2-r9 (hardened/x86/2.6, gcc-3.4.6, glibc-2.3.6-r5, 2.6.18-hardened i686) ================================================================= System uname: 2.6.18-hardened i686 AMD Athlon(TM) XP 2000+ Gentoo Base System release 1.12.9 Timestamp of tree: Wed, 21 Feb 2007 00:20:01 +0000 distcc 2.18.3 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [enabled] dev-lang/python: 2.4.3-r4 dev-python/pycrypto: 2.0.1-r5 sys-apps/sandbox: 1.2.17 sys-devel/autoconf: 2.13, 2.60 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10 sys-devel/binutils: 2.16.1-r3 sys-devel/gcc-config: 1.3.14 sys-devel/libtool: 1.5.22 virtual/os-headers: 2.6.17-r2 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -march=athlon-xp -pipe -fomit-frame-pointer" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/X11/xkb" CONFIG_PROTECT_MASK="/etc/env.d /etc/gconf /etc/revdep-rebuild /etc/terminfo" CXXFLAGS="-O2 -march=athlon-xp -pipe -fomit-frame-pointer" DISTDIR="/usr/portage/distfiles" FEATURES="autoconfig collision-protect distcc distlocks fixpackages metadata-transfer parallel-fetch sandbox sfperms strict userfetch userpriv usersandbox" GENTOO_MIRRORS="ftp://ftp.belnet.be/mirror/rsync.gentoo.org/gentoo/ ftp://ftp.uni-erlangen.de/pub/mirrors/gentoo ftp://ftp.snt.utwente.nl/pub/os/linux/gentoo" LINGUAS="nl en_GB" MAKEOPTS="-j3" PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage" USE="X acpi alsa berkdb bzip2 cairo cdr crypt cups dbus dri esd gnome gstreamer gtk hal hardened jpeg midi nls nptl nptlonly opengl pam pic png python readline ssl svg tcpd tiff truetype unicode usb x86 xml xorg zlib" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null plug rate route share shm softvol" ELIBC="glibc" INPUT_DEVICES="evdev keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="nl en_GB" USERLAND="GNU" VIDEO_CARDS="mga" Unset: CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, LDFLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PORTDIR_OVERLAY I found only one bug in bugzilla resembling to this same problem: bug #44643
Not gcc-porting.
When using distcc, you must ensure that the gcc versions are identical. Please provide the exact gcc configuration on each machine; i.e. the output of 'gcc-config -l' showing the installed and selected compilers. > fuse: Unknown symbol _GLOBAL_OFFSET_TABLE_ This indicates the module was built -fPIC (or more likely -fPIE), which it shouldn't have been. This could happen if a distcc machine was using a hardened compiler, and the host wasn't. Please also attach the build log for the failure case. On a related note; my bug #128810 (now somewhat stale, probably) presents a way of allowing distcc (and ccache) to know reliably what hardened is up to; however objections mean it hasn't gone forward.
What gcc versions (hardened?) are being used on your volunteer machines?
Both machines are running hardened. emerge --info for the local machine is in comment #1. gcc-config -l for this machine: tomhendr@elaine ~ $ gcc-config -l [1] i686-pc-linux-gnu-3.4.6 * [2] i686-pc-linux-gnu-3.4.6-hardenednopie [3] i686-pc-linux-gnu-3.4.6-hardenednopiessp [4] i686-pc-linux-gnu-3.4.6-hardenednossp [5] i686-pc-linux-gnu-3.4.6-vanilla The (only) remote distcc host: tomhendr@gloria ~ $ emerge --info Portage 2.1.2-r9 (hardened/x86/2.6, gcc-3.4.6, glibc-2.3.6-r5, 2.6.18-hardened i686) ================================================================= System uname: 2.6.18-hardened i686 Pentium III (Coppermine) Gentoo Base System release 1.12.9 Timestamp of tree: Thu, 22 Feb 2007 23:50:01 +0000 distcc 2.18.3 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled] ccache version 2.4 [enabled] dev-lang/python: 2.4.3-r4 dev-python/pycrypto: 2.0.1-r5 dev-util/ccache: 2.4-r6 sys-apps/sandbox: 1.2.17 sys-devel/autoconf: 2.13, 2.60 sys-devel/automake: 1.6.3, 1.8.5-r3, 1.9.6-r2, 1.10 sys-devel/binutils: 2.16.1-r3 sys-devel/gcc-config: 1.3.14 sys-devel/libtool: 1.5.22 virtual/os-headers: 2.6.17-r2 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -march=pentium3 -pipe -fomit-frame-pointer" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc" CONFIG_PROTECT_MASK="/etc/env.d /etc/gconf /etc/revdep-rebuild /etc/terminfo" CXXFLAGS="-O2 -march=pentium3 -pipe -fomit-frame-pointer" DISTDIR="/usr/portage/distfiles" FEATURES="autoconfig ccache collision-protect distlocks fixpackages metadata-transfer parallel-fetch sandbox sfperms strict userfetch userpriv usersandbox" GENTOO_MIRRORS="ftp://ftp.snt.utwente.nl/pub/os/linux/gentoo ftp://ftp.belnet.be/mirror/rsync.gentoo.org/gentoo" LANG="en_GB.utf-8" LC_ALL="en_GB.utf-8" MAKEOPTS="-j2" PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage" USE="apache2 berkdb crypt cups hardened ipv6 mailwrapper midi nls nptl nptlonly pam pic readline ssl tcpd unicode vhosts x86 zlib" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null plug rate route share shm softvol" ELIBC="glibc" INPUT_DEVICES="mouse keyboard" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" USERLAND="GNU" Unset: CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LDFLAGS, LINGUAS, PORTAGE_RSYNC_EXTRA_OPTS, PORTDIR_OVERLAY tomhendr@gloria ~ $ gcc-config -l [1] i686-pc-linux-gnu-3.4.6 * [2] i686-pc-linux-gnu-3.4.6-hardenednopie [3] i686-pc-linux-gnu-3.4.6-hardenednopiessp [4] i686-pc-linux-gnu-3.4.6-hardenednossp [5] i686-pc-linux-gnu-3.4.6-vanilla Just for curiosity, I also ran distcc-config --set-hosts "localhost" before emerging and that module loads just fine :P
Created attachment 111070 [details] build.log of a non-working module
Okay. I've had a test with sys-fs/fuse: root@boron ~ # gcc-config -l [1] i386-pc-linux-gnu-3.3.4 [2] i686-pc-linux-gnu-3.4.3 * [3] i686-pc-linux-gnu-3.4.3-hardened [4] i686-pc-linux-gnu-3.4.3-hardenednopie [5] i686-pc-linux-gnu-3.4.3-hardenednossp root@helium ~ # gcc-config -l [1] i686-pc-linux-gnu-3.3.6 [2] i686-pc-linux-gnu-3.3.6-hardened [3] i686-pc-linux-gnu-3.3.6-hardenednopie [4] i686-pc-linux-gnu-3.3.6-hardenednopiessp [5] i686-pc-linux-gnu-3.3.6-hardenednossp [6] i686-pc-linux-gnu-3.4.6 * [7] i686-pc-linux-gnu-3.4.6-hardened [8] i686-pc-linux-gnu-3.4.6-hardenednopie [9] i686-pc-linux-gnu-3.4.6-hardenednopiessp [10] i686-pc-linux-gnu-3.4.6-hardenednossp (Minor versions are fine to mismatch usually.) root@boron ~ # modprobe fuse root@boron ~ # lsmod Module Size Used by fuse 45268 0 root@boron ~ # rmmod fuse root@boron ~ # lsmod Module Size Used by root@boron ~ # dmesg fuse init (API version 7.8) fuse distribution version: 2.6.3 fuse exit This is a similar setup to yours, both are using hardened setups. Host is athlon-xp and the volunteer is pentium3. There was distcc traffic during compilation as verified with `tcpdump -i eth0 port 3632`. I am unable to test net-wireless/rt2500. Please verify that there is traffic on the network between the two machines by using tcpdump or wireshark or some other monitoring tool. The default port is 3632. If you say there's nothing going on with the volunteer machine then that may indiciate some problem.
When I said earlier I saw no activity on the remote host, I 'checked' this by running 'top -u distcc' on the remote host. Now tested with tcpdump and distccmon-text and there is some traffic and compilation going on on the other host. I also tested (by accident, actually:P) what happens when there is outgoing traffic on port 3632, but on the other end there is no distccd listening. This produces a working module. So the problem seems to happen only when distcc actually gets some compilations done on the other host.
(In reply to comment #7) > I also tested (by accident, actually:P) what happens when there is outgoing > traffic on port 3632, but on the other end there is no distccd listening. This > produces a working module. So the problem seems to happen only when distcc > actually gets some compilations done on the other host. Now that's interesting. Can you enable verbose operations (and logging) on the remote host and then attach the log to the bug please?
Created attachment 112124 [details] distccd log distccd ran on the remote host with options: DISTCCD_OPTS="${DISTCCD_OPTS} --log-level debug --log-file /var/log/distccd"
re: comment #6: your setup is not exactly hardened, look at how your gcc profiles sport a -hardened version whereas comment #4 shows -vanilla instead (latter would be coming from an actual hardened gcc as its specs file defaults to what is -hardened on a non-hardened gcc). IOW, your profile was that of -vanilla, not -hardened and that would explain why you couldn't reproduce the problem. as for the actual problem, i think it's got something to do with how files belonging to a kernel module are passed down to distcc, of particular interest is the -D__KERNEL__ define that the hardened specs files rely on to know when not to forcibly apply -fpie et al.
A thought. Disable ccache on both machines, and try the compilation again. ccache and the toolchain as delivered by Gentoo don't recognise when the toolchain is switched from hardened to vanilla (see bug #128810), so maybe on one machine or other you're picking up a pre-compiled version that doesn't reflect the current compiler configuration.
(In reply to comment #11) Thanks for your thoughts on this. Although your scenario does not match the setup (both hosts were hardened from the start and I never needed to switch gcc any one of the hosts), I still tried your suggestion. The local machine had a reinstall for other reasons than this, but I had no problems reproducing the problem with the rt2500 module. Disabling ccache on both machines does not solve the problem, and yields the same result.
Had another thought, and I know what's happening - finally :) distcc pre-processes the source code on the master machine, and passes that pre-processed source to the slave for compilation. It is quite possible that distcc strips any macro definitions from the command line, as these should make no difference to source code that has already been pre-processed. However, a hack we do in the hardened compiler is to suppress the automatic setting of -fPIE (and -fstack-protector-all fwiw) when the pre-processor macro '__KERNEL__' is defined. All kernel compilations, including modules built outside of the kernel, set the macro __KERNEL__, so that any special code for kernel builds can be selected. If distcc is indeed stripping -D* from the compilation command - and it would be reasonable for it to do so - our hardened hacks will not be suppressed on the slave machine. Thus the object built on the slave machine will be built with -fPIE - which is incorrect for any kernel code (including modules). This object would define __GLOBAL_OFFSET_TABLE__, as you reported. ...time passes... checked distcc - it does strip -D* (see 'strip.c' in the distcc source). So, the up-shot is that our hardened stuff won't work with distcc, when it comes to compiling kernel code. Everything else should be ok; the only pre-processor trigger we depend on is -D__KERNEL__. We could hack distcc to pass through -D__KERNEL__, but since kernel modules are small and quickly compiled anyway, I think it's probably better just to disable distcc when building kernel stuff. Another possibility, where a package _only_ builds a kernel module, adding 'filter-flags -fPIE' might be appropriate - but where a package builds other stuff as well, that solution would not be appropriate. Lisa - up to you; if you'll take a patch to distcc to pass through -D__KERNEL__, I'll make one - if you'd rather not, just resolve/cantfix or whatever.
*** Bug 171553 has been marked as a duplicate of this bug. ***
It is INVALID by upstream. FYI: http://code.google.com/p/distcc/issues/detail?id=25 I think that it should be set 'RESTRICT="distcc"'
I have entertained that thought many times, as kernel modules often is (and I think should be) small packages, not taking that much time to compile (or recomile when updating you kernel). I have also had failures with kernel modules not wanting to be loaded when mixing arch and ~arch with non-hardened toolchains. So considering the fragility of this maybe the idea about RESTRICT is a good one.
CCing eclass/linux-mod.eclass maintainer I suggest to add 'RESTRICT="distcc"' to eclass/linux-mod.
(In reply to comment #17) > CCing eclass/linux-mod.eclass maintainer > > I suggest to add 'RESTRICT="distcc"' to eclass/linux-mod. > Good idea, done. Thanks!
Reopening as there is no support for RESTRICT="distcc".
Fixed via patch to distcc adding USE="hardened". emerge --sync and rebuild distcc.
