I am currently testing xen 4.1.1 with Gentoo-sources 3.0.3. (Inside a Virtualbox VM) Dom0 will start correctly. When trying to start a DomU, I found that it fails when trying to access qemu. After installing qemu, I was able to start a PV-domain. Reproducible: Always Steps to Reproduce: 1. Install clean Gentoo with Xen 4.1.1 and gentoo-sources 3.0.3 2. Try to start PV-domain Actual Results: PV fails to start, logs in /var/log/xen indicate issue finding qemu-dm Expected Results: Using Xen 3 and xen-sources, I was able to start PV-domains without qemu. For qemu, I only had "ncurses" in the USE-flags (disabled "sdl") And both TARGET-lists only had "i386 x86_64" included. I used the app-emulation/qemu version that is currently marked stable for amd64 (0.11.1)
Further testing, /etc/init.d/xendomains doesn't work even with qemu installed. Changing this init-script to use "xm" instead of "xl" solves this. Error message for this (found in /var/log/xen/qemu-dm-domU.log): libxl: cannot execute /usr/lib/xen/bin/qemu-dm: No such file or directory.
that's good testing. Seems ? a gap in xl can you try xl and list any other cmds it fails
also check /etc/xen/xl.conf. Ensure the settings are uncommented
I tried some more commands. From the way it fails, I think it only goes wrong when trying to create a Domain. Even with a PV-domain, "xl" wants to access qemu. I just tried it with debugging: # xl create domU -d Parsing config file domU (domain (domid -1) (domain_create_info) (hvm 0) (hap 1) (oos 1) (ssidref 0) (name domU) (uuid <unknown>) (cpupool Pool-0) (xsdata (null)) (platformdata (null)) (domain_build_info) (max_vcpus 1) (tsc_mode 0) (max_memkb 1048576) (target_memkb 524288) (nomigrate 0) (image (linux 0) (kernel /boot/linux_3.0.3) (cmdline root=/dev/xvda1) (ramdisk (null)) ) ) (device (tap (backend_domid 0) (domid 0) (physpath /dev/vg_host/domu_1_disk) (phystype 1) (virtpath xvda1) (unpluggable 0) (readwrite 1) (is_cdrom 0) ) ) (device (tap (backend_domid 0) (domid 0) (physpath /dev/vg_host/domu_1_swap) (phystype 1) (virtpath xvda2) (unpluggable 0) (readwrite 1) (is_cdrom 0) ) ) (device (vif (backend_domid 0) (domid 0) (devid 0) (mtu 1492) (model rtl8139) (mac 00163e1601bb) ) ) ) libxl: error: libxl_device.c:476:libxl__wait_for_device_model Device Model not ready xl: fatal error: libxl_create.c:535, rc=-1: libxl__confirm_device_model_startup I had a quick look at the code, but am not familiar with this one yet. Perhaps someone who knows this code can find out why it wants to have qemu for a PV-domain. The config I use is: # cat /etc/xen/auto/domU kernel = "/boot/linux_3.0.3" memory = 512 maxmem = 1024 name = "domU" vif = ['mac=00:16:3E:16:01:BB,bridge=eth0'] disk = ['phy:/dev/vg_host/domu_1_disk,xvda1,w', 'phy:/dev/vg_host/domu_1_swap,xvda2,w'] extra = 'root=/dev/xvda1'
Ok, wasn't aware of this file yet. As I'm coming from Xen2.x and Xen3.x, I never had use of "xl". *** # cat /etc/xen/xl.conf ## Global XL config file ## # automatically balloon down dom0 when xen doesn't have enough free # memory to create a domain #autoballoon=1 # full path of the lockfile used by xl during domain creation #lockfile="/var/lock/xl" # default vif script #vifscript="vif-bridge" *** All settings commented, lets try this: *** # cat /etc/xen/xl.conf ## Global XL config file ## # automatically balloon down dom0 when xen doesn't have enough free # memory to create a domain autoballoon=1 # full path of the lockfile used by xl during domain creation lockfile="/var/lock/xl" # default vif script vifscript="vif-bridge" *** Am afraid this has the same result.
Error message for this (found in /var/log/xen/qemu-dm-domU.log): libxl: cannot execute /usr/lib/xen/bin/qemu-dm: No such file or directory Does this conclusively tie the failure to find /usr/lib/xen/bin/qemu-dm to xl? I must assume that /usr/lib/xen/bin/qemu-dm exists because xm found it.
xm doesn't require "qemu". xl does. Prior to todays tests, I unmerged qemu: # emerge -C qemu # emerge --depclean I just tested with "xm": ** xentoo ~ # xm create /etc/xen/auto/domU Using config file "/etc/xen/auto/domU". Started domain domU (id=3) xentoo ~ # ls -lsa /usr/lib/xen/bin/qemu-dm ls: cannot access /usr/lib/xen/bin/qemu-dm: No such file or directory xentoo ~ # xm list Name ID Mem VCPUs State Time(s) Domain-0 0 512 1 r----- 39.7 domU 3 512 1 -b---- 4.7 ** After this, I can use "xl" to connect to the console with: # xl console domU My suspicion is that "xl", for whatever reason, thinks "qemu-dm" is needed for a PV-domain and fails if it can't find it. As "xm" is able to create the domain, I feel it is clear that "qemu-dm" is not really needed.
ok, now double check You are using your host kernel kernel = "/boot/linux_3.0.3". examine your config. Confirm whether it is a pvops config or a XenPVonHVM config. Is that's new, it's outlined at xensource.org or xen.org whichever. So if you have a XenPVonHVM config, I s
Ian, I think your last comment was cut off. I checked the kernel config and did find the following: CONFIG_XEN_PVHVM=y A search (make menuconfig .... /pvhvm ) didn't give anything useful. Via google, I can't find anything quickly either. I don't know how to disable that option without editing the .config-file. I'm wondering why a kernel-config-setting would make "xl" want to load qemu-dm, where "xm" doesn't appear to care at all. I would prefer not to have to install qemu, but if it's a hard dependency then I'll have to. One of the reasons why I'd rather not install qemu is that I'd prefer to keep the dom0 as small as possible and I have no need for HVM-domains. If you have any other ideas on things I can try, please let me know. If it helps, I can make the virtualbox VM available for download. Joost
(In reply to comment #9) > Ian, > > I think your last comment was cut off. > > I checked the kernel config and did find the following: > CONFIG_XEN_PVHVM=y > > A search (make menuconfig .... /pvhvm ) didn't give anything useful. > Via google, I can't find anything quickly either. > > I don't know how to disable that option without editing the .config-file. > > > If you have any other ideas on things I can try, please let me know. > If it helps, I can make the virtualbox VM available for download. > > Joost yes I did cut it off a bit. Little loss CONFIG_XEN_PVHVM=y is the key. this is indeed a XenPVonHVM kernel. That means it requires qemu-dm, & to my best understanding, this is in xen-tools gentoo64 xen-4.1.1 # equery b qemu-dm * Searching for qemu-dm ... app-emulation/xen-tools-4.1.1-r1 (/usr/lib/xen/bin/qemu-dm). It seems to me there is no place for qemu package in this scenario in dom0. The problem is that xl you say is not finding qemu-dm, when it is there to be found. You can relieve it of this by redoing your kernel as straight pvops. I thought you would have find this in xensource. http://wiki.xensource.com/xenwiki/XenLinuxPVonHVMdrivers related links also describe pvops. To revert, you need deselect a number of settings. Secondly, need find a way to inform xl of its presence, which is daunting in light of the line that you cite saying it's just not finding it.
"qemu-dm" is not, necessarily, part of "xen-tools". Am wondering if it's related to some USE-flags: ** xentoo ~ # equery b qemu-dm * Searching for qemu-dm ... xentoo ~ # eix xen-tools [I] app-emulation/xen-tools Available versions: 3.4.2 ~4.1.0-r1 (~)4.1.1 **9999 {acm api custom-cflags debug doc flask hvm ioemu pygrub screen} Installed versions: 4.1.1(12:52:25 08/16/11)(screen -acm -api -custom-cflags -debug -doc -flask -hvm -ioemu -pygrub) Homepage: http://xen.org/ Description: Xend daemon and tools ** I did originally look for the file "qemu-dm" using: find / -iname "*qemu-dm*" Apart from files in /var/log/xen, it didn't return any. I will check the kernel-config and the links on that page you provided to see how to disable that kernel option and get back to you on that. Is there no way to tell "xl" not to try to use qemu-dm when it's clearly not necessary?
Ok, to test without PVHVM, I'm running into a problem. I can't figure out how to disable this option. When I manually delete it from the .config, it reappears as enabled. When doing a search for it in menuconfig, I get: Symbol: XEN_PVHVM [=y] Type :boolean Also trying to grep for it in the kernel-source doesn't bring me any further. How does "xl" determine wether or not to have to use "qemu-dm"? Or does it always want to use it? As I put in my previous entry, "qemu-dm" is not always included when installing "xen-tools". My guess, it's one of the USE-flags I didn't select. I didn't de-select any USE-flags. And only "screen" is added to xen-tools.
As I put in my previous entry, "qemu-dm" is not always included when installing "xen-tools". My guess, it's one of the USE-flags I didn't select > I didn't de-select any USE-flags. And only "screen" is added to xen-tools. grief gentoo64 temp # eix xen-tools [I] app-emulation/xen-tools Available versions: *3.4.2 (~)4.1.1-r1 **9999 {acm api custom-cflags debug doc flask hvm ioemu pygrub screen} Installed versions: 4.1.1-r1(23:50:21 09/06/11)(api custom-cflags debug flask hvm ioemu pygrub screen -doc) ok now to the heart of the matter You have been installing xen-tools a number of times. without use hvm, it won't install qemu-dm Now re-emerge xen-tools with the above use flags, take a couple out if you want, but I'd suggest them as they are. a hvm boot uses qemu-dm. a xen pvops boot can use the host kernel or the guest's own kernel depending on if you're using pygrub or pvgrub. I suggest your vm can be happily booted just like my current gentoo vm guest with xm or al. xm or xl is ireelevant. a hvm boot calls qemu-dm. Leave your kernel as is. It's good boot your XenPVonHVM capable kernel with a hvm boot after you emerge it with the required use flags. I tend to enable most to cover any possibility re-post
Ian, That will be a work-around. Adding HVM and simply ignore it. I have been checking the source-code for "xl". From that, I noticed there are a few reasons why it will try to access "qemu-dm": *** int libxl__need_xenpv_qemu(libxl_ctx *ctx, int nr_consoles, libxl_device_console *consoles, int nr_vfbs, libxl_device_vfb *vfbs, int nr_disks, libxl_device_disk *disks) { int i, ret = 0; libxl__gc gc = LIBXL_INIT_GC(ctx); if (nr_consoles > 1) { ret = 1; goto out; } for (i = 0; i < nr_consoles; i++) { if (consoles[i].consback == LIBXL_CONSBACK_IOEMU) { ret = 1; goto out; } } if (nr_vfbs > 0) { ret = 1; goto out; } if (nr_disks > 0 && !libxl__blktap_enabled(&gc)) ret = 1; out: libxl__free_all(&gc); return ret; } *** (File = libxl_dm.c) in the xen-tools.tar.gz package. Number of consoles should be 1. Not sure what the "LIBXL_CONSBACK_IOEMU" is. Still investigating that one. Am guessing: "nr_vfbs" = 0. (Not using VNC) This leaves nr_disks (=2) and the blktab_enabled. I'm using "phy:....." to specify my disks. Am wondering if this should be going upstream?
possibly for upstream. libxl: cannot execute /usr/lib/xen/bin/qemu-dm: No such file or directory is the line that concerns me. Still now clear whether the qemu-dm was in place at the time xl went looking for it. qemu-dm is the 'kernel' used by xen to boot a hvm guest.If you cross check doing a manual build in the xensource package, xen-tools makes and owns qenu-dm. xl is new to me too. It is the new xm. It is supposed to do what xm does. It however does not use xend, and that is all. To me it's not a workaround, it's supplying the system with the required qemu-dm. I still see the problem is that xl cannot find qemu-dm. I do not see that xl should not seek and find it. To demo this further, boot the guest by whichever, do a dumpxml of the guest as it is running by libvirt. virsh -c xen dumpxml vm-guest It will output a libvirt xml file that represents the booting of the guest. a hvm boot calls qemu-dm. It 'will' cite qemu-dm as the booting source. To return to the task of converting your kernel, just remove all your xen settings, enter the few cited in the xensource wiki to equip it with pvops, only about 3 or 4 required, and that should remove your PVONHVM setting. If not, get a previous gentoo xen kernel of a jeremy xensource git kernel xen kernel, and retry. Once you get that, you can claim the guest boot does not require qemu-dm, and can build up more xen settings if desried
(In reply to comment #15) > possibly for upstream. > > libxl: cannot execute /usr/lib/xen/bin/qemu-dm: No such file or directory > > is the line that concerns me. Still now clear whether the qemu-dm > was in place at the time xl went looking for it. It wasn't. As it's either included when using "hvm"-USE-flag (as you found out) Or when installing "qemu" At least, "xl" didn't complain anymore after I installed "qemu" > qemu-dm is the 'kernel' used by xen to boot a hvm guest.If you cross check > doing a manual build in the xensource package, xen-tools makes and owns > qenu-dm. Yes, but it shouldn't bother with hvm as the config used is for a PV-domain. > xl is new to me too. It is the new xm. It is supposed to do what xm does. In this respect, it doesn't. Unless upstream says, we always use "qemu-dm" > It however does not use xend, and that is all. To me it's not a workaround, > it's supplying the system with the required qemu-dm. I still see the problem > is that xl cannot find qemu-dm. I do not see that xl should not seek and find > it. What is the new way? Not start "xend"? I am in the process of planning an upgrade of Xen to 4.x. Currently happily using Xen 3. It's unfortunate that the howto's are all either based on HVM or on Xen3.x. There is no clear guide on how to set up PV-domains using Xen4.x (Or I missed it) The Xen-wiki isn't too useful. Too much focus on the file-based storage and HVM.
I feel your pain. I am currently testing xen 4.1.1 with Gentoo-sources 3.0.3. I am in the process of planning an upgrade of Xen to 4.x. Currently happily using Xen 3. Well, Gentoo-sources 3.0.3 for a start demands a recent xen-4.0 or 4.1 minimal. Mix them up. and ... extra = 'root=/dev/xvda1' Does you guest boot and create /dev/xvda? I am perplexed by the lack of docs for xl and xen4. I am picking up the threads too. xensource are simply behind. It seems xl functions independently from xend. I have never emerged qemu to boot a xen domain.. You are booting the guest from the host kernel, pv style. It is a XenPVonHVM equipped kernel. There are the other kernels, you can alternately equip the guest with its own kernel, as I have. There is pygrub and pvgrub. xen offers a generosity of options. You problem may be your kernel is too new. xensource stable is ?? 2.6.32, which they just update.
I spend some hours in our server room last weekend to fiddle around with new xen versions and hit exactly this bug (among others). I searched the xen-devel mailing list archives and found some things. At first some things to clearify: * with Xen 4.1 xm and xend are deprecated but can still be used * the replacement is the xl tool, written in c (so no more python blah) with libxl as interface to the hypervisor (doesn't need xend running) * libxl's behavior of handle storage backend configurations differs from xm (see http://article.gmane.org/gmane.comp.emulators.xen.devel/97877) Relevant for my puporses is the handling of file (disk='file://...') backends. xm call a hotplug-script to loop mount the image and use the blkback backend from dom0 (now included in vanilla kernel). libxl only uses blkback for block devices, for raw image and vhd files it uses blktap2 (not included in vanilla kernel). And there was a bug in handling block devices: http://article.gmane.org/gmane.comp.emulators.xen.devel/110962 Which probably fix your issue. Hopefully included in upcoming 4.1.2 release. Until blktap2 support reaches vanilla I will stuck to xm and xend. The current ebuild installs both, xm/xend and xl. This reminds me to figure out why I have to start xend to start domains with xl (like /etc/init.d/xendomains does through service dependency) Many things to test. My last stable configuration is still Xen 4.0.2 (builds with python 2.7, yeah) and xen-sources-2.6.34-r4.
Thanks Peter, That patch looks like it would work. When specifying "phy:..." for the disks, it won't try starting qemu. (If that is the only part wanting it) Now I just need to figure out how to get this patch into the xen-tools ebuild... -- Joost
Uhm, until 4.1.2 you could try: 1. mkdir -p /etc/portage/patches/app-emulation/xen-tools-4.1.1 2. wget http://xenbits.xen.org/hg/xen-unstable.hg/raw-rev/d4ca456c0c25 -O /etc/portage/patches/app-emulation/xen-tools-4.1.1/xen-tools-4.1.1-libxl-tap.patch 3. add the following to /etc/portage/bashrc: post_src_unpack() { epatch_user }
(In reply to comment #20) > Uhm, until 4.1.2 you could try: > > 1. mkdir -p /etc/portage/patches/app-emulation/xen-tools-4.1.1 > 2. wget http://xenbits.xen.org/hg/xen-unstable.hg/raw-rev/d4ca456c0c25 -O > /etc/portage/patches/app-emulation/xen-tools-4.1.1/xen-tools-4.1.1-libxl-tap.patch > 3. add the following to /etc/portage/bashrc: > > post_src_unpack() { > epatch_user > } Needed some more work, the patch-file needed some modification for this to work. I will upload my copy in a second. The PV-domU does start, but now complains about something else. But I'll try to get that to work. Either way, I think this bug should be solvable using the patch to the "xl" tool.
Created attachment 286331 [details, diff] libxl-patch (amended for gentoo) Please do following: 1) mkdir -p /etc/portage/patches/app-emulation/xen-tools-4.1.1 2) Place this file at "/etc/portage/patches/app-emulation/xen-tools-4.1.1" 3) Add the following to the file: /etc/portage/bashrc (Create it if it doesn't exist yet) ## Import custom patches ## post_src_unpack() { epatch_user } ## End of Import custom patches ## 4) rebuild the "xen-tools" package: emerge -va --oneshot xen-tools
Joost do as follows, save making another patch copy the patch content into $PORTDIR/app-emulation/xen-tools/files/xen-tools-4.1.1-xllib.patch # cd $PORTDIR/app-emulation/xen-tools/ # nano xen-tools-4.1.1-r4.ebuild add " # Patch to libxl epatch "${FILESDIR}/${PN}-4.1.1-xllib.patch" " after # Do not strip binaries epatch "${FILESDIR}/${PN}-3.3.0-nostrip.patch" # repoman manifest # ebuild xen-tools-4.1.1-r4.ebuild unpack prepare compile you should get to make: Leaving directory `/var/tmp/portage/portage/app-emulation/xen-tools-4.1.1-r4/work/xen-4.1.1/docs' >>> Source compiled. repost
and forgot, this is step 1 change --- xen-4.1.1.orig/tools/libxl/libxl_dm.c.orig Tue Mar 15 10:14:27 2011 +0000 to --- xen-4.1.1.orig/tools/libxl/libxl_dm.c.orig.orig Tue Mar 15 10:14:27 2011 +0000 then as above
Fixed in -r5 for xen-tools
