Bug 379753 – app-emulation/xen-tools-4.1.1: Cannot start HVM virtual machines gracefully wtih bridged networking

• Status: RESOLVED WORKSFORME
• Product: Gentoo Linux

[Description Richard 2011-08-18 16:31:50 UTC]

If you try to start a HVM virtual machine with bridged networking on Xen 4.1.1, it will fail after a time-out. Here is an example:

# xm create hvm.cfg
Using config file "./hvm.cfg".
Error: Device 0 (vif) could not be connected. Hotplug scripts not working.

I can workaround this problem by sending SIGINT to xm and then unpausing the virtual machine. It starts without a problem if I do that, but that is a hack.

Here is my hvm.cfg:

kernel = '/usr/lib/xen/boot/hvmloader'
builder = 'hvm'
memory = '2048'
device_model='/root/qemu-dm-custom'

# Disks
disk = [ 'phy:vg/mediacenter,hda,w',
'file:/mnt/backup/isos/win7_64/en_windows_7_professional_x64_dvd_X15-65805.iso,ioemu:hdc:cdrom,r' ]

# Hostname
name = 'mediacenter'

# Networking
vif = ['type=ioemu,mac=52:54:00:00:ee:00,bridge=br0']
vfb = [ 'vnc=1, vnclisten=0.0.0.0,vncunused=1']

vcpus=2
usbdevice='tablet'

#pci = [ '03:08.0' ]

# Behaviour
boot='d'
vnc=1
vncviewer=1
sdl=0

I followed the instructions here to do networking, except I named my bridge br0 instead of xenbr0:

http://forums.gentoo.org/viewtopic-t-689293-highlight-xen+bridge.html

This issue is being discussed on the Xen mailing list:

http://www.gossamer-threads.com/lists/xen/users/213049
http://www.gossamer-threads.com/lists/xen/devel/214149
http://www.gossamer-threads.com/lists/xen/devel/214629

For the record, the patch mentioned in conversation 214629 does not fix the problem. I did some troubleshooting via the log files and I managed to produce a stack trace via gdb. I tried eyeballing the code, but I really don't understand Python, so I am at a loss as to how to proceed:

(gdb) thread 2
[Switching to thread 2 (Thread 0x7f70dd7f4700 (LWP 24198))]#0  0x00007f70e9597be3 in select () from /lib64/libc.so.6
(gdb) pystack
/usr/lib64/python2.7/threading.py (258): wait
/usr/lib64/python2.7/threading.py (395): wait
/usr/lib64/python2.7/site-packages/xen/xend/server/DevController.py (559): waitForBackend
/usr/lib64/python2.7/site-packages/xen/xend/server/DevController.py (151): waitForDevice
/usr/lib64/python2.7/site-packages/xen/xend/server/DevController.py (140): waitForDevices
/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py (1237): waitForDevices
/usr/lib64/python2.7/site-packages/xen/xend/server/XMLRPCServer.py (56): dispatch
<string> (1): <lambda>
/usr/lib64/python2.7/SimpleXMLRPCServer.py (422): _dispatch
/usr/lib64/python2.7/site-packages/xen/util/xmlrpclib2.py (136): _marshaled_dispatch
/usr/lib64/python2.7/site-packages/xen/util/xmlrpclib2.py (85): do_POST
/usr/lib64/python2.7/BaseHTTPServer.py (326): handle_one_request
/usr/lib64/python2.7/BaseHTTPServer.py (338): handle
/usr/lib64/python2.7/SocketServer.py (641): __init__
/usr/lib64/python2.7/site-packages/xen/util/xmlrpclib2.py (68): __init__
/usr/lib64/python2.7/site-packages/xen/util/xmlrpclib2.py (109): <lambda>
/usr/lib64/python2.7/SocketServer.py (323): finish_request
/usr/lib64/python2.7/SocketServer.py (583): process_request_thread
/usr/lib64/python2.7/threading.py (487): run
/usr/lib64/python2.7/threading.py (531): __bootstrap_inner
/usr/lib64/python2.7/threading.py (504): __bootstrap

I am using Gentoo amd64 stable with some exceptions:

# cat /etc/portage/package.keywords 
app-emulation/libvirt ~amd64
app-emulation/qemu-kvm ~amd64
app-emulation/spice ~amd64
app-emulation/spice-protocol ~amd64
app-emulation/virt-manager ~amd64
app-emulation/virt-viewer ~amd64
app-emulation/xen ~amd64
app-emulation/xen-tools ~amd64
media-libs/celt ~amd64
sys-kernel/amd-ucode ~amd64
sys-kernel/dracut ~amd64
sys-kernel/vanilla-sources ~amd64

I originally used KVM on this system, but the pci-passthrough did not work, so I switched to Xen. Here is my system information:

# emerge --info
Portage 2.1.10.3 (default/linux/amd64/10.0, gcc-4.4.5, glibc-2.11.3-r0, 3.0.3 x86_64)
=================================================================
System uname: Linux-3.0.3-x86_64-AMD_Phenom-tm-_II_X2_555_Processor-with-gentoo-2.0.3
Timestamp of tree: Wed, 17 Aug 2011 23:30:01 +0000
ccache version 2.4 [disabled]
app-shells/bash:          4.1_p9
dev-lang/python:          2.7.1-r1, 3.1.3-r1
dev-util/ccache:          2.4-r9
dev-util/cmake:           2.8.4-r1
dev-util/pkgconfig:       0.26
sys-apps/baselayout:      2.0.3
sys-apps/openrc:          0.8.3-r1
sys-apps/sandbox:         2.4
sys-devel/autoconf:       2.68
sys-devel/automake:       1.11.1
sys-devel/binutils:       2.20.1-r1
sys-devel/gcc:            4.4.5
sys-devel/gcc-config:     1.4.1-r1
sys-devel/libtool:        2.4-r1
sys-devel/make:           3.82
sys-kernel/linux-headers: 2.6.36.1 (virtual/os-headers)
sys-libs/glibc:           2.11.3
Repositories: gentoo
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe -march=amdfam10 -mcx16 -msahf -mpopcnt --param l1-cache-size=64 --param l1-cache-line-size=64 --param l2-cache-size=512 -mtune=amdfam10"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -pipe -march=amdfam10 -mcx16 -msahf -mpopcnt --param l1-cache-size=64 --param l1-cache-line-size=64 --param l2-cache-size=512 -mtune=amdfam10"
DISTDIR="/usr/portage/distfiles"
FEATURES="assume-digests binpkg-logs distlocks ebuild-locks fixlafiles fixpackages news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"
FFLAGS="-O2 -pipe -march=amdfam10 -mcx16 -msahf -mpopcnt --param l1-cache-size=64 --param l1-cache-line-size=64 --param l2-cache-size=512 -mtune=amdfam10"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j3"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY=""
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="3dnow 3dnowext acl acpi amd64 berkdb bzip2 cli cracklib crypt cups cxx dri fortran gdbm gpm iconv ipv6 lvm2 mmap mmx modules mudflap multilib ncurses nls nptl nptlonly openmp pam pcre perl pppd python readline session sse sse2 sse3 sse4a ssl sysfs tcpd unicode xorg zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="braindump flow karbon kexi kpresenter krita tables words" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="fbdev glint intel mach64 mga neomagic nouveau nv r128 radeon savage sis tdfx trident vesa via vmware dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" 
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, LINGUAS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

[Comment 1 Richard 2011-08-23 15:08:22 UTC]

It appears that upstream might have fixed this in commit 3c05c4bed4ccce3f22f6d7899b308faae24ad198:

http://www.kernel.org/pub/linux/kernel/v3.0/testing/ChangeLog-3.1-rc3

I won't have time to verify the fix for a while.

[Comment 2 Ian Delaney (RETIRED) 2011-09-02 04:46:55 UTC]

ditto
most recent emerge of 4.1.1 seems fixed

[Comment 3 Vitaliy V. Osypenko 2011-09-02 21:28:23 UTC]

I have the same error. And if I comment out vif configuration from cfg it will show next error:

Error: Device 51712 (tap) could not be connected. Hotplug scripts not working.

So I guess hotplug scripts don't work at all in xen-tools 4.1.1. Could it be udevd?

kernel = "/VM/vmlinuz"
ramdisk = "/VM/initrd.img"
extra = 
name = "SL6"
memory = "512"
disk = [ 'tap:aio:/home/data/VM/CentOS6OVZ.img,xvda,w', ]
vif = [ 'bridge=xenbr0', ]
vcpus=1
on_reboot = 'destroy'
on_crash = 'destroy'

#last line

Portage 2.2.0_alpha51 (default/linux/amd64/10.0/desktop, gcc-4.5.3, glibc-2.13-r4, 3.0.4-gentoo x86_64)
=================================================================
System uname: Linux-3.0.4-gentoo-x86_64-AMD_Athlon-tm-_64_X2_Dual_Core_Processor_5000+-with-gentoo-2.0.3
Timestamp of tree: Fri, 02 Sep 2011 00:45:01 +0000
distcc 3.1 x86_64-pc-linux-gnu [disabled]
ccache version 3.1.6 [disabled]
app-shells/bash:          4.2_p10
dev-java/java-config:     2.1.11-r3
dev-lang/python:          2.7.2-r2, 3.2-r2
dev-util/ccache:          3.1.6
dev-util/cmake:           2.8.5-r2
dev-util/pkgconfig:       0.26
sys-apps/baselayout:      2.0.3
sys-apps/openrc:          0.9.1
sys-apps/sandbox:         2.5
sys-devel/autoconf:       2.13, 2.68
sys-devel/automake:       1.6.3-r1, 1.9.6-r3, 1.10.3, 1.11.1-r1
sys-devel/binutils:       2.21.1-r1
sys-devel/gcc:            4.4.6-r1, 4.5.3-r1
sys-devel/gcc-config:     1.4.1-r1
sys-devel/libtool:        2.4-r1
sys-devel/make:           3.82-r1
sys-kernel/linux-headers: 2.6.39 (virtual/os-headers)
sys-libs/glibc:           2.13-r4
Repositories: zugaina rion kde-sunset kde sunrise local gentoo
Installed sets: @kdeadmin-4.7, @kdeartwork-4.7, @kdebase-4.7, @kdegames-4.7, @kdegraphics-4.7, @kdelibs-4.7, @kdemultimedia-4.7, @kdenetwork-4.7, @kdeutils-4.7, @kdewebdev-4.7, @system
ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -march=native -msse3 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/openvpn/easy-rsa /usr/share/themes/oxygen-gtk/gtk-2.0"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/splash /etc/terminfo"
CXXFLAGS="-O2 -march=native -msse3 -pipe"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--with-bdeps y --keep-going"
FEATURES="assume-digests binpkg-logs buildpkg distlocks ebuild-locks fixlafiles fixpackages metadata-transfer news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"
FFLAGS=""
GENTOO_MIRRORS="ftp://ftp.linux.cz/pub/linux/gentoo"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en cs ru"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp/portage"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/var/lib/layman/zugaina /var/lib/layman/rion /var/lib/layman/kde-sunset /var/lib/layman/kde /var/lib/layman/sunrise /system/portage-local /usr/portage"
SYNC="rsync://rsync.de.gentoo.org/gentoo-portage"
USE="3dnow 3dnowext 64bit 7zip X a52 aac acl acpi alsa amd64 avi bash-completion bitmap-fonts bluetooth branding bzip2 cairo cdda cddb cdr cli consolekit cpudetection cracklib crypt css cups cxx dbus devmap dlloader dri dts dv dvd dvdr dvdread emboss emul-linux-x86 encode exif extensions fam fbcon ffmpeg firefox flac fortran ftp fuse gdbm gdu gif glut gnutls gpm gstreamer gtk gtk2 htmlhandbook iconv ieee1394 ipv6 jpeg kde lame lcms libg++ libnotify lm_sensors logitech-mouse mad mikmod mmx mng modules mp3 mp4 mpeg mplayer mudflap multilib musepack musicbrainz ncurses nls nptl nptlonly nvidia ogg opengl openmp oss pam pango pcre pdf pdflib perl png policykit ppds pppd python qt3 qt3support qt4 quicktime readline reflection sdl session spell spl sse sse-filters sse2 sse3 ssl startup-notification svg symlink sysfs tcpd threads tiff tk truetype truetype-fonts type1-fonts udev unicode usb utf8 vcd vorbis x264 xcb xcomposite xml xorg xscreensaver xulrunner xv xvid zlib" ALSA_CARDS="intel8x0" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="cgi cgid actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="evdev mouse keyboard" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en cs ru" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="nvidia nouveau vesa" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

[Comment 4 Vitaliy V. Osypenko 2011-09-03 17:36:47 UTC]

I'm sorry, I don't know what happened but after restart everything is OK.

[Comment 5 Ian Delaney (RETIRED) 2011-09-05 14:04:21 UTC]

wonder if you improved the kernel config by a setting, of got tun modules loaded.