Bug 570684

Summary:	x11-base/xorg-server-1.17.4 crashes on startup when run within virtualbox 5.0.12
Product:	Gentoo Linux	Reporter:	Jochen Trumpf <Jochen.Trumpf>
Component:	Current packages	Assignee:	Gentoo X packagers <x11>
Status:	RESOLVED NEEDINFO
Severity:	normal	CC:	bkohler
Priority:	Normal
Version:	unspecified
Hardware:	AMD64
OS:	Linux
Whiteboard:
Package list:		Runtime testing required:	---

Description Jochen Trumpf 2016-01-03 06:28:00 UTC

When starting an xorg-server 1.17.4 session that runs within a virtual machine (virtualbox 5.0.12) I get the following reproducible crash:

(EE) Backtrace:
(EE) 0: /usr/bin/X (xorg_backtrace+0x56) [0x5da016]
(EE) 1: /usr/bin/X (0x400000+0x1df1b9) [0x5df1b9]
(EE) 2: /lib64/libc.so.6 (0x7f027ff72000+0x333f0) [0x7f027ffa53f0]
(EE) 3: /usr/bin/X (xf86SlowBcopy+0xc9) [0x4baa19]
(EE) 4: /usr/lib64/xorg/modules/libvgahw.so (vgaHWSaveFonts+0x16b) [0x7f027b457adb]
(EE) 5: /usr/lib64/xorg/modules/drivers/vboxvideo_drv.so (0x7f027ba85000+0x5682) [0x7f027ba8a682]
(EE) 6: /usr/bin/X (AddScreen+0x1c5) [0x43b285]
(EE) 7: /usr/bin/X (InitOutput+0x497) [0x490007]
(EE) 8: /usr/bin/X (0x400000+0x3f724) [0x43f724]
(EE) 9: /lib64/libc.so.6 (__libc_start_main+0xf0) [0x7f027ff927b0]
(EE) 10: /usr/bin/X (_start+0x29) [0x426f29]
(EE) 
(EE) Illegal instruction at address 0x4baa19
(EE) 
Fatal server error:
(EE) Caught signal 4 (Illegal instruction). Server aborting

The culprit are SSE instructions in xf86SlowBcopy (point 3 of the above backtrace). More information is here:

http://docs.freebsd.org/cgi/getmsg.cgi?fetch=3723422+0+/usr/local/www/mailindex/archive/2015/svn-ports-all/20150906.svn-ports-all

I adapted the patch in the above FreeBSD commit and it solves the problem for me:

--- hw/xfree86/os-support/misc/Makefile.in.orig 2016-01-03 16:06:39.058077040 +1100
+++ hw/xfree86/os-support/misc/Makefile.in      2016-01-03 16:10:46.004479596 +1100
@@ -530,7 +530,8 @@
 
 #AM_LDFLAGS = -r
 AM_CPPFLAGS = $(XORG_INCS)
-AM_CFLAGS = $(XORG_CFLAGS) $(DIX_CFLAGS)
+@I386_VIDEO_TRUE@I386_VIDEO_CFLAGS = -mno-sse
+AM_CFLAGS = $(I386_VIDEO_CFLAGS) $(XORG_CFLAGS) $(DIX_CFLAGS)
 EXTRA_DIST = $(I386_SRCS) $(PORTIO_SRCS)
 all: all-am



Reproducible: Always

Steps to Reproduce:
1. boot up your gentoo Linux virtualbox
2. start an Xsession
3. 
Actual Results:  
xorg-server crashes

Expected Results:  
don't crash ;-)

Comment 1 Ben Kohler gentoo-dev

2017-08-31 19:22:26 UTC

Are you applying CFLAGS that imply sse? Does vbox not pass through sse instructions?