Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 200376

Summary: app-emulation/open-vm-tools fails to compile PIC/PIE (hardened default)
Product: Gentoo Linux Reporter: Tim Taubert <ttmails2>
Component: New packagesAssignee: Gentoo VMWare Bug Squashers [disabled] <vmware+disabled>
Status: RESOLVED FIXED    
Severity: normal CC: amlabs, blueness, chris.lieb.hotmail, hardened, sonix, steffen.bergner, vmware+disabled
Priority: High    
Version: unspecified   
Hardware: All   
OS: Linux   
URL: http://sourceforge.net/tracker/index.php?func=detail&aid=1813024&group_id=204462&atid=989708
Whiteboard:
Package list:
Runtime testing required: ---
Attachments: vmware-esx-tools-3.0.2.ebuild
checkvm-pie-safety.patch
open-vm-tools-0.0.20090824.187411.ebuild.patch
checkvm-pie-safety.patch - revision 2
checkvm-pie-safety.patch - revision 3
checkvm-pie-safety.patch - revision 3 no damage
checkvm-pie-safety.patch - revision 4
checkvm-pie-safety.patch - revision 4 correct whitespace

Description Tim Taubert 2007-11-26 09:55:10 UTC
VMware ESX tools 3.0.2 are out.

Reproducible: Always
Comment 1 Tim Taubert 2007-11-26 09:55:36 UTC
Created attachment 137029 [details]
vmware-esx-tools-3.0.2.ebuild
Comment 2 Mike Auty (RETIRED) gentoo-dev 2007-11-26 10:25:31 UTC
Tim,

I'd be very interested to know if the open-vm-tools ebuild works under ESX.  Any chance you could give the ebuild a go?  It's currently in the vmware overlay (which you can get with layman).  If it all works out, then I think I'll suggest it's use as a replacement for all vmware-tools packages.

It currently seems to handle server/player/workstation without issue, and gets updated more frequently than other tools, I'm just not certain if the ESX version will have different code within it.

Thanks...  5:)
Comment 3 Tim Taubert 2007-11-26 11:39:41 UTC
Hey Mike,

I just forgot about the open source vmware tools, it's a good point to try. Unfortunately the emerge fails with this error message:

checkvm.c: In function `getVersion':
checkvm.c:80: error: can't find a register in class `BREG' while reloading `asm'
make[1]: *** [checkvm.o] Error 1
make[1]: Leaving directory `/var/tmp/portage/app-emulation/open-vm-tools-0.0.20071121.64693/work/open-vm-tools-2007.11.21-64693/checkvm'
make: *** [all-recursive] Error 1

I'm running a hardened profile on x86-hardened-2.6.23-r2. Do you have any idea how to fix this? My guess is that there are problems with the PIE/SSP-patched gcc.
Comment 4 Mike Auty (RETIRED) gentoo-dev 2007-11-26 11:47:16 UTC
Hiya Tim,

Thanks for giving that a try so quickly.  I'm afraid I don't know enough to offer any advice, but it might be worth mentioning on an upstream bug tracker and working with them to fix the problem.  I'm sure they'd appreciate having a tester who could check it against ESX?

If you do report it upstream, please post a link to the bug report/mailing list entry on their sourceforge site (http://open-vm-tools.sf.net/).  Thanks...  5:)
Comment 5 Mike Auty (RETIRED) gentoo-dev 2007-11-26 11:54:56 UTC
My mistake, it turns out this has already been reported upstream:

http://sourceforge.net/tracker/index.php?func=detail&aid=1813024&group_id=204462&atid=989708

Your best bet is to start tracking that bug.  If you can add your issues or help narrow down the diagnosis, I'm sure it'll be greatly appreciated.  5:)
Comment 6 Tim Taubert 2007-11-26 12:19:01 UTC
Ok, the project's bug tracker is not very helpful until now :). I tried to solve it my own and patched the Makefile.in in './checkvm/' so that it's CFLAGS+="-nopie".

With this added to the ebuild it compiles fine and runs fine. The ESX server displays that the tools are running "OK". Unfortunately trying to restart/halt the guest operating system fails (nothing happens).
Comment 7 Mike Auty (RETIRED) gentoo-dev 2007-11-26 12:28:31 UTC
Hmmm, this might be because of the scripts files not being executable.  They just got added in this release, and I haven't had time to check the functionality.

Actually, I just took a look through the ebuild and realized I made a stupid mistake.  Gimme a few minutes to correct it and then I'll post again.  Sorry about this...  5:)
Comment 8 Mike Auty (RETIRED) gentoo-dev 2007-11-26 12:57:15 UTC
Ok, it turns out, I hadn't checked my installation of the default soft-power option scripts.  I've updated the ebuild so they're now installed and given the executable bit (please update the overlay and reinstall).

Unfortunately I'm not sure if they're supposed to be renamed from x-vm-default to x-vm, so that might be worth a test as well.  Anyway, if you could please check out whether this helps, we can try and narrow down the problem and hopefully get it solved.  Thanks...  5:)
Comment 9 Tim Taubert 2007-11-26 13:24:47 UTC
Ok, well done :). Controlling the guest operating system now works for me and the tools compile cleanly (remember the nopie-Makefile.in patch).

The last thing I wonder is: shouldn't the vmxnet and vmmemctl modules automatically be loaded with the vmware-tools init script?
Comment 10 Mike Auty (RETIRED) gentoo-dev 2007-12-22 17:59:13 UTC
Hiya Tim, the open-vm-tools has just been committed to the main tree, but I'm afraid doesn't include the nopie patch.  I'm going to change the title of this bug, since we've mostly been discussing the failure of open-vm-tools under a hardened profile, but if you're rather we take that to a separate bug, just say and I'll convert it all back and start the new bug.

In the meantime, I'll try and create the appropriate patch in the coming month and get it added into the overlay, followed by the main tree after a bit of testing.

Finally, I haven't decided whether the modules should be loaded by the init script or not.  Certainly there are occasions where vmxnet doesn't work or isn't wanted, but pcnet32 does, so I'm a little reticent to force them on the user, but I'll have a think about it.
Comment 11 Tim Taubert 2007-12-23 09:05:19 UTC
Hey Mike,

I think renaming this thread is totally good. I'll just wait until you have the patched ebuild released and will test it then. Your thoughts about the autoload of modules maybe right because giving the user a choice is better I think.
Comment 12 Mike Auty (RETIRED) gentoo-dev 2007-12-26 14:27:10 UTC
Hiya Tim, 

I've added a new ebuild that uses eautoreconf to regenerate the various configure files (including those for checkvm).  I'm hoping this will cure your problem, but if not, I can still go back to the original patch.  Please give it a go and let me know how it works out.  Thanks...  5:)
Comment 13 Tim Taubert 2007-12-28 08:08:24 UTC
Hey Mike,
unfortunately the ebuild doesn't compile giving the same error as before:

checkvm.c: In function `getVersion':
checkvm.c:80: error: can't find a register in class `BREG' while reloading `asm'
make[1]: *** [checkvm.o] Error 1
Comment 14 Mike Auty (RETIRED) gentoo-dev 2007-12-28 22:00:51 UTC
Thanks Tim,

I've checked in a proper patch now that applies to both hardened and unhardened systems.  Please give it a try ans if it all works out I'll commit it to the main tree in a couple of weeks.  Thanks very much!  5:)
Comment 15 Tim Taubert 2008-01-11 08:16:53 UTC
Unfortunately emerge fails with the following error message:

 * Preparing vmblock module
Using standalone build system.
./autoconf/geninclude.c:19:28: linux/autoconf.h: No such file or directory
./autoconf/geninclude.c:19:28: linux/autoconf.h: No such file or directory
.././autoconf/geninclude.c:19:28: linux/autoconf.h: No such file or directory
In file included from .././linux/filesystem.c:26:
.././include/driver-config.h:40:2: #error "No Module support in this kernel.  Please configure with CONFIG_MODULES"
In file included from .././linux/module.c:26In file included from .././linux/control.c:26:
.././include/driver-config.h:40:2: #error "No Module support in this kernel.  Please configure with CONFIG_MODULES"
:
.././include/driver-config.h:40:2: #error "No Module support in this kernel.  Please configure with CONFIG_MODULES"
make[2]: *** [filesystem.d] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: *** [module.d] Error 1
make[2]: *** [control.d] Error 1
make[1]: *** [deps] Error 2
make: *** [auto-build] Error 2

I _do_ have module support built into the kernel. Maybe it's because "linux/autoconf.h" is not found? I have no clue.
Comment 16 Mike Auty (RETIRED) gentoo-dev 2008-01-11 09:11:05 UTC
Hmmmm, so, the difference between your patch and mine is that I patched the underlying Makefile.am that later gets filled out by autotools into a Makefile.in for the configure script to work with.  It suggests that I possibly need to use a different version of autotools, but I don't know enough about them at the moment to offer a fix.

I'll ask around a bit and see if I can get that fixed, otherwise we'll go with no autotooling and just directly patching the Makefile.in.  Thanks for your patience and testing in getting this fixed...  5:)
Comment 17 Gordon Malm (RETIRED) gentoo-dev 2008-01-27 00:48:19 UTC
In regards to the error in Comment #15:

I just hit this error as well.  I had compiled open-vm-tools from the portage tree successfully before.  I checked the revision history of the ebuild and associated files, but there are no relevant changes.

Looked at my portage logs...

Previously, when the modules compiled correctly I would get the message:
 * Preparing vmblock module
Using 2.6.x kernel build system.

Now I get:
 * Preparing vmblock module
Using standalone build system.

The Makefile for each module reads:

VM_UNAME = $(shell uname -r)
HEADER_DIR = /lib/modules/$(VM_UNAME)/build/include
BUILD_DIR = $(HEADER_DIR)/..

ifndef VM_KBUILD
VM_KBUILD := no
ifeq ($(call vm_check_file,$(BUILD_DIR)/Makefile), yes)
ifneq ($(call vm_check_file,$(BUILD_DIR)/Rules.make, yes)
VM_KBUILD := 26
endif
endif
export VM_KBUILD
endif

ifndef VM_KBUILD_SHOWN
ifeq ($(VM_KBUILD), no)
VM_DUMMY := $(shell echo >&2 "Using standalone build system.")
else
ifeq ($(VM_KBUILD, 24)
VM_DUMMY := $(shell echo >&2 "Using 2.4.x kernel build system.")
else
VM_DUMMY := $(shell echo >&2 "Using 2.6.x kernel build system.")
endif
endif
VM_KBUILD_SHOWN := yes
export VM_KBUILD_SHOWN
endif

The line:
VM_UNAME = $(shell uname -r)
is of particular interest.  This returns the running kernel, not the target kernel we are trying to compile the modules for.

Going back to the logs of the previous successful build of open-vm-tools, I notice:
 * Preparing vmblock module
Using 2.6.x kernel build system.
make -C /lib/modules/2.6.20-hardened-r10-2007092501/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. modules

This particular build target was actually kernel 2.6.23-hardened-r5-2008010701, but I had left the sources for 2.6.20-hardened-r10-2007092501 in place in /usr/src.  Thus, open-vm-tools successfully chose the kernel 2.6.x build system.  The manner in which open-vm-tools came to this conclusion was incorrect, but worked to allow it to select the kernel 2.6.x build system.

This time around I am compiling open-vm-tools for 2.6.23-hardened-r6-2008012601, but I moved the running kernel's (2.6.23-hardened-r5-2008010701) sources before attempting to compile open-vm-tools.  So vm_check_file checks are looking for Makefile and Rules.make under "/lib/modules/2.6.23-hardened-r5-2007010701/build/include/.." (instead of the target kernel), which does not exist.

If the source directory for the _running_ kernel doesn't exist, open-vm-tools trys to use the standalone build system.  This results in failures as open-vm-tools looks in /usr/include/linux for the header files (like autoconf.h) instead of the headers included in the kernel source, etc.

I fixed this by adding the following line to src_unpack() in the ebuild:
for i in ${S}/${VMWARE_MOD_DIR}/*; do sed -i s/"shell uname -r"/KV/ $i/Makefile; done

The VMware guest in question is set USE="pam -X -xinerama".  I'm not sure this is the "most correct" way to fix the error, but I haven't found a downside so far.  A quick "grep -r uname" of the open-vm-tools sources shows lots of hits, but I don't have the time to go through them all right now to see if they are actually used/valid/etc.


As to the original problem: 

checkvm.c: In function `getVersion':
checkvm.c:80: error: can't find a register in class `BREG' while reloading `asm'
make[1]: *** [checkvm.o] Error 1

Well.. it is still a problem with the ebuild in the portage tree.  It can be worked around by setting the -hardenednopie compiler using gcc-config or setting the GCC_SPECS environment variable to point to the -hardenednopie gcc specs file.  For example:
GCC_SPECS="/usr/lib/gcc/i686-pc-linux-gnu/3.4.6/hardenednopie.specs" emerge open-vm-tools
Comment 18 Mike Auty (RETIRED) gentoo-dev 2008-01-28 17:46:10 UTC
Hi guys, I've just committed a fix for the kernel source detection issues. and build errors with 2.6.24.  I've added a warning in pkg_setup for people to see this comment concerning compiling with a hardened toolchain.  I may still add the -fnopie patch at some point, but this will need pushing out sooner.  In the meantime, I'll requote the relevant paragraph from comment 17, as it seemed very handy.  5:)

"Well.. it is still a problem with the ebuild in the portage tree.  It can be
worked around by setting the -hardenednopie compiler using gcc-config or
setting the GCC_SPECS environment variable to point to the -hardenednopie gcc
specs file.  For example:
GCC_SPECS="/usr/lib/gcc/i686-pc-linux-gnu/3.4.6/hardenednopie.specs" emerge
open-vm-tools"
Comment 19 Tim Taubert 2008-01-29 06:22:54 UTC
> GCC_SPECS="/usr/lib/gcc/i686-pc-linux-gnu/3.4.6/hardenednopie.specs" emerge
> open-vm-tools"

Using this command line I was finally able to compile open-vm-tools. As far as I can see they are running without any problems.
Comment 20 Steffen Bergner 2008-01-30 09:18:29 UTC
Just for info about new compilation problem with standard 2.6.24 kernel: 
http://bugs.gentoo.org/show_bug.cgi?id=200376
Comment 21 Steffen Bergner 2008-01-30 09:38:20 UTC
Sorry, correct link/incident:

http://bugs.gentoo.org/show_bug.cgi?id=208154
Comment 22 Mike Auty (RETIRED) gentoo-dev 2008-01-30 22:19:51 UTC
Ok, there's now a warning during package setup for users using hardened, and a new build that builds off the correct kernel sources and works with 2.6.24.  5:)

I'm gonna mark this as FIXED, please feel free to re-open if there are further issues...  5:)
Comment 23 Chris Lieb 2008-11-18 20:32:41 UTC
Was a patch for nopie ever added to the tree?  I am trying to compile open-vm-tools-20080808.108361-r1 against hardened-sources-2.6.25-r10 and getting the same register error from comment #3.  The GCC_SPECS command line hack worked, but is not optimal since I will need to document its use for future upgrades and administrators.
Comment 24 Mike Auty (RETIRED) gentoo-dev 2008-11-18 22:30:24 UTC
Hiya Chris, there was no patch added, just the warning for hardened users to see this bug.  Your best bet is to file an upstream bug at http://open-vm-tools.sf.net/ and get it fixed at the source.  I don't run a hardened system and so can't test or maintain a nopie patch, sorry...
Comment 25 Chris Lieb 2008-11-18 22:34:35 UTC
Where was the warning that you are referring to?  I never saw one.  I just googled the compiler error and found this bug.
Comment 26 Mike Auty (RETIRED) gentoo-dev 2008-11-18 22:37:35 UTC
It happens during package setup (so right at the start, although more recent portages will recap all the messages at the end):

http://sources.gentoo.org/viewcvs.py/gentoo-x86/app-emulation/open-vm-tools/open-vm-tools-0.0.20080808.109361-r1.ebuild?rev=1.1&view=markup

ewarn "If you're compiling for a hardened target, please use the hardened"
ewarn "toolchain (see bug #200376, comment 18)."
Comment 27 Chris Lieb 2008-11-18 22:50:51 UTC
I recall seeing that.  I wasn't sure what to make of it, since I figured I was already compiling with a hardened toolchain.  Wouldn't it be clearer to say that you need to compile with a "nopie" gcc?
Comment 28 Mike Auty (RETIRED) gentoo-dev 2008-11-19 01:38:06 UTC
You're quite right, I've now updated the error message to read "If you're compiling with a hardened toolchain, please use the hardened gcc profile."

The hardened gcc profiles have the nopie flag as far as I recall.  Just so you know, if you ever have a hardened toolchain but *aren't* using the hardened profile, you're unlikely to get much support from devs.  The best you'll get if it breaks is to keep the two pieces...
Comment 29 Chris Lieb 2008-11-19 02:24:02 UTC
I have the following gcc profiles listed by `gcc-config -l` (from my memory):

* i686-pc-linux-gnu-3.4.6
* i686-pc-linux-gnu-3.4.6-hardenednopie
* i686-pc-linux-gnu-3.4.6-hardenednopiessp
* i686-pc-linux-gnu-3.4.6-hardenednossp
* i686-pc-linux-gnu-3.4.6-vanilla

It is my understanding that the first one is the default hardened GCC that adds SSP and PIC to all code, which is what I normally have enabled.  To get open-vm-tools to compile, I used the GCC_SPECS for the hardenednopie profile.  The message that you currently display would lead me to believe that I should activate the first profile listed, not the second, which is the one that would be optimal (though the third and fifth profiles should work too since they also exclude PIC).
Comment 30 Gordon Malm (RETIRED) gentoo-dev 2008-11-19 09:33:49 UTC
Comment #29 is quite right, hardenednopie are the gcc specs you want.  However, this needs to be patched properly so re-opening and CCing hardened herd.
Comment 31 Mike Auty (RETIRED) gentoo-dev 2008-11-19 10:15:15 UTC
I'll be glad to apply any patches you guys produce.  You should also please be checking this against the latest version in the vmware overlay, not just the main tree (we're up to 20081010 there, and I believe 20081118 will come out soon).  Thanks for offering to fix it...  5:)
Comment 32 Gordon Malm (RETIRED) gentoo-dev 2009-01-25 08:51:51 UTC
Hi Mike,

Please drop the ewarns for open-vm-tools-0.0.20081223.137496 and feel free to close this bug.  It compiles fine w/o any need to switch gcc specs.  Didn't test any earlier versions in the tree to know if they are fixed or not, so might as well leave the message in them and those ebuilds will just expire over time.
Comment 33 Mike Auty (RETIRED) gentoo-dev 2009-01-25 13:21:05 UTC
Ok, thanks for letting me know.  Marking as FIXED!  (I wish there were a little noise or animation with cheering people and fireworks and stuff when things get marked as fixed.  Yay!  Woo!)  5:)
Comment 34 Aurelien Minet 2009-02-08 03:06:13 UTC
(In reply to comment #32)
 
> Please drop the ewarns for open-vm-tools-0.0.20081223.137496 and feel free to
> close this bug.  It compiles fine w/o any need to switch gcc specs.

Sorry it doesn't, at least in my case :

Portage 2.1.6.4 (hardened/x86/2.6, gcc-3.4.6, glibc-2.6.1-r0, 2.6.24-gentoo-r8 i686)
=================================================================
System uname: Linux-2.6.24-gentoo-r8-i686-Intel-R-_Xeon-R-_CPU_E5420_@_2.50GHz-with-glibc2.3.2
Timestamp of tree: Fri, 06 Feb 2009 07:45:01 +0000
ccache version 2.4 [disabled]
app-shells/bash:     3.2_p39
dev-java/java-config: 1.3.7-r1, 2.1.6-r1
dev-lang/python:     2.5.2-r7
dev-python/pycrypto: 2.0.1-r6
dev-util/ccache:     2.4-r7
sys-apps/baselayout: 1.12.11.1
sys-apps/sandbox:    1.2.18.1-r2
sys-devel/autoconf:  2.13, 2.63
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10.2
sys-devel/binutils:  2.18-r3
sys-devel/gcc-config: 1.4.0-r4
sys-devel/libtool:   1.5.26
virtual/os-headers:  2.6.27-r2
ACCEPT_KEYWORDS="x86"
CBUILD="i686-pc-linux-gnu"
CFLAGS=" -O2 -mtune=i686 -fomit-frame-pointer -pipe "
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /var/bind"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/gconf /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/splash /etc/terminfo /etc/udev/rules.d"
CXXFLAGS=" -O2 -mtune=i686 -fomit-frame-pointer -pipe "
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--with-bdeps y"
FEATURES="distlocks fixpackages parallel-fetch protect-owned sandbox sfperms strict unmerge-orphans userfetch"
GENTOO_MIRRORS="http://ftp.belnet.be/mirror/rsync.gentoo.org/gentoo/ ftp://pandemonium.tiscali.de/pub/gentoo/ ftp://sunsite.informatik.rwth-aachen.de/pub/Linux/gentoo ftp://ftp.tu-clausthal.de/pub/linux/gentoo/"
LANG="fr_FR@euro"
LC_ALL="fr_FR@euro"
LDFLAGS=""
LINGUAS="fr fr_FR"
MAKEOPTS="-j3"
PKGDIR="/usr/portage/packages"
PORTAGE_RSYNC_EXTRA_OPTS="--timeout=200"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/portage/local/layman/vmware /usr/local/portage"
SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage"
USE="acl acpi apache apache2 bash-completion berkdb bzlib chroot cracklib crypt curl hardened hardenedphp iconv java ldap md5sum memlimit midi mmap mmx mpi mysql ncurses nls nptl nptlonly openntpd pam perl php pic pie pnp posix python quotas readline ruby samba sasl shared sharedmem snmp sockets ssl sysvipc threads unicode urandom usb vim-syntax virus-scan x86 xml xml2 zlib" 
Unset:  CPPFLAGS, CTARGET, FFLAGS, INSTALL_MASK, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS


I still need to add GCC_SPECS in the command line for 0.0.20081223.137496 and 0.0.20090121.142982  as in checkvm/checkvm.c , getVersion() hasn't change.

thanks in advance for reopening
Comment 35 Gordon Malm (RETIRED) gentoo-dev 2009-02-09 21:36:32 UTC
Indeed this is not fixed.  I had a long-term workaround in place locally and forgot about it.  Re-opening.
Comment 36 Vadim Kuznetsov (RETIRED) gentoo-dev 2009-07-09 21:35:19 UTC

*** This bug has been marked as a duplicate of bug 276759 ***
Comment 37 Gordon Malm (RETIRED) gentoo-dev 2009-07-13 02:52:33 UTC
Not a duplicate of bug 276759.  The error is different, re-opening.
Comment 38 Gordon Malm (RETIRED) gentoo-dev 2009-10-24 12:03:42 UTC
Created attachment 208136 [details, diff]
checkvm-pie-safety.patch
Comment 39 Gordon Malm (RETIRED) gentoo-dev 2009-10-24 12:04:14 UTC
Created attachment 208138 [details, diff]
open-vm-tools-0.0.20090824.187411.ebuild.patch
Comment 40 Gordon Malm (RETIRED) gentoo-dev 2009-10-24 12:08:50 UTC
Hi VMware herd.

Attached patches should resolve the issue.  checkvm patch applies cleanly to all versions in the tree.  --with-pic configuration option is also available in all versions in the tree.

Please test:
vmware-checkvm
vmware-checkvm -h
vmware-checkvm -p
vmware-checkvm -r

Output should not change after the patch vs. without the patch.  Works for me on hardened/x86.
Comment 41 Anthony Basile gentoo-dev 2009-10-25 16:40:36 UTC
I tested checkvm-pie-safety.patch on hardened/amd64.  I checked most of the features of vmware-tools.  Works fine for me.
Comment 42 Anthony Basile gentoo-dev 2009-10-25 19:30:26 UTC
Let me follow up on my previous comment.  On 32-bit I got the same results before and after applying the patch for the tests in Comment #40:

thirtytwo-testing app-emulation # vmware-checkvm
VMware software version 6 (good)
thirtytwo-testing app-emulation # vmware-checkvm -h
VM's hw version is 4
VMware software version 6 (good)
thirtytwo-testing app-emulation # vmware-checkvm -p
Workstation
thirtytwo-testing app-emulation # vmware-checkvm -r
1024 768

However, for 64-bit I got a minor compile time error because pushl/popl %%ebx is 32-bit specific and we need pushq/popq %%rbx instead.  So I applied the following:

--- checkvm.c.orig	2009-10-25 19:09:57.000000000 +0000
+++ checkvm.c	2009-10-25 19:10:55.000000000 +0000
@@ -79,10 +79,10 @@
 {
    uint32 eax, ebx, ecx, edx;
    
-   __asm__ volatile("pushl %%ebx      \n\t"
+   __asm__ volatile("pushq %%rbx      \n\t"
                     "inl (%%dx)       \n\t"
                     "movl %%ebx, %3   \n\t"
-                    "popl %%ebx       \n\t" :
+                    "popq %%rbx       \n\t" :
    	            "=a"(eax), "=c"(ecx), "=d"(edx), "=r"(ebx) :
 		    "0"(BDOOR_MAGIC), "1"(BDOOR_CMD_GETVERSION),
 		    "2"(BDOOR_PORT) : "memory");


Afterwords, it compiled and gave the same results as before applying the patch:

sixtyfour-testing ~ # vmware-checkvm
VMware software version 6 (good)
sixtyfour-testing ~ # vmware-checkvm -h
VM's hw version is 4
VMware software version 6 (good)
sixtyfour-testing ~ # vmware-checkvm -p
Workstation
sixtyfour-testing ~ # vmware-checkvm -r
1024 768

Comment 43 Gordon Malm (RETIRED) gentoo-dev 2009-10-25 21:32:39 UTC
Created attachment 208253 [details, diff]
checkvm-pie-safety.patch - revision 2

Updated patch incorporates fixes for x86-64 architecture.  Passes tests and works for me on hardened/x86.

Many thanks to Anthony Basile for testing & fixes!
Comment 44 Anthony Basile gentoo-dev 2009-10-25 22:07:20 UTC
Version 2 of the patch passes the tests and works on hardened 64-bit.  I test with both hardened and hardenednopie.
Comment 45 Gordon Malm (RETIRED) gentoo-dev 2009-10-26 00:57:01 UTC
Hold the phone.  The patch breaks when compiled non-PIE on x86 sometimes.  I say sometimes because it depends on optimization level.  gcc-4.3.4 @ -O2 re-orders the popl ahead of the movl.  -O1 is fine, -O0 ends up with what edx would normally have.  I didn't catch it before because I had some debugging fprintfs which kept that gcc opt from happening.

Removing the pushl/popl completely and leaving the rest works, so we could ifdef that on __PIC__ if necessary.  Trying to find a more elegant solution though.

Status summary:

x86/PIE - works
x86/no-PIE - fails on -O0 and -O2, works on -O1.
amd64/PIE - works
amd64/no-PIE - works
Comment 46 Gordon Malm (RETIRED) gentoo-dev 2009-10-26 08:13:03 UTC
Created attachment 208290 [details, diff]
checkvm-pie-safety.patch - revision 3

(In reply to comment #45)
> Hold the phone.  The patch breaks when compiled non-PIE on x86 sometimes.  I
> say sometimes because it depends on optimization level.  gcc-4.3.4 @ -O2
> re-orders the popl ahead of the movl.  -O1 is fine, -O0 ends up with what edx
> would normally have.  I didn't catch it before because I had some debugging
> fprintfs which kept that gcc opt from happening.

Ok, so after looking @ the disasm it wasn't the push/mov/pop that was being re-ordered, but some other stuff ahead of it.  The function was actually being *fixed* by happenstance at by certain compiler + -O[0-3] optimization combinations.

> 
> Removing the pushl/popl completely and leaving the rest works, so we could
> ifdef that on __PIC__ if necessary.  Trying to find a more elegant solution
> though.
> 

I did some re-ordering of push/mov/pop stuff and looked @ the disasm.  afaict the backdoor code actually relies on & uses the value in ebx as part of the magic.  So that pushl, which would normally be a virtual noop in the non-PIE scenario, breaks things in this case.  Then did some poking around (probably should've before going disasm crazy, oh well) and discovered ifdefing on __PIC__ is actually what *many* parts of the open-vm-tools code does; probably for that very reason.

That determined, here is a new patch.  Tested on x86 under the following scenarios:
gcc-3.4.6, -O[0-3], no-PIE - all tests pass
gcc-3.4.6, -O[0-3], PIE - all tests pass
gcc-4.3.4, -O[0-3], no-PIE - all tests pass
gcc-4.3.4, -O[0-3], PIE - all tests pass

Unlike previous patches, in the case of x86-64 the resultant compiled code remains unchanged. So 3rd time is a charm right? ;)
Comment 47 Gordon Malm (RETIRED) gentoo-dev 2009-10-26 09:43:21 UTC
Created attachment 208293 [details, diff]
checkvm-pie-safety.patch - revision 3 no damage

still rev3, no changes.  last patch had ---/+++ header damage, sorry.
Comment 48 Gordon Malm (RETIRED) gentoo-dev 2009-10-26 11:04:04 UTC
Created attachment 208297 [details, diff]
checkvm-pie-safety.patch - revision 4

minor optimization, first xchg replaced with a mov.

x86-32:
gcc-3.4.6, -O[0-3s], no-PIE - all tests pass
gcc-3.4.6, -O[0-3s], PIE - all tests pass
gcc-4.3.4, -O[0-3s], no-PIE - all tests pass
gcc-4.3.4, -O[0-3s], PIE - all tests pass

x86/non-PIE and x86-64 codepaths are untouched by the patch.
Comment 49 Gordon Malm (RETIRED) gentoo-dev 2009-10-26 11:13:29 UTC
Created attachment 208305 [details, diff]
checkvm-pie-safety.patch - revision 4 correct whitespace

lol, make it stop. whitespace, no real change - sorry.
Comment 50 Anthony Basile gentoo-dev 2009-10-27 20:23:27 UTC
I've tested the latest patch on 64-bits and found:

gcc-4.4.2, -O[0-3s], no-PIE - all tests pass
gcc-4.4.2, -O[0-3s], PIE - all tests pass

This is gcc from the hardened-dev overlay with espf patch.  I am in the process of testing hardened gentoo's stock gcc-4.3.4.

Comment 51 Anthony Basile gentoo-dev 2009-10-30 00:17:53 UTC
I finished testing hardened gentoo's stock gcc-4.3.4 on 64-bits.  All looks good:

gcc-4.3.4, -O[0-3s], no-PIE - all tests pass
gcc-4.3.4, -O[0-3s], PIE - all tests pass

Comment 52 Gordon Malm (RETIRED) gentoo-dev 2009-10-30 17:07:28 UTC
VMware herd, please add patch to open-vm-tools.

Also, I do not have an account on upstream's tracker in comment #5 and would prefer not to open one.  I suspect someone in VMware team has one already, mind attaching the last checkvm-pie-safety.patch upstream please?

Thank you Anthony Basile for all your help and testing.  And thanks Gentoo VMware team.
Comment 53 Vadim Kuznetsov (RETIRED) gentoo-dev 2009-11-07 13:48:24 UTC
committed to the tree.