Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 493246 - app-emulation/libvirt-1.1.3 with sys-apps/systemd-208-r2: VMs not starting
Summary: app-emulation/libvirt-1.1.3 with sys-apps/systemd-208-r2: VMs not starting
Status: RESOLVED NEEDINFO
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Gentoo systemd Team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-12-03 17:58 UTC by Stefan G. Weichinger
Modified: 2014-11-20 12:35 UTC (History)
3 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Stefan G. Weichinger 2013-12-03 17:58:42 UTC
When I use "virsh start myvm" it does not work.
I tried upgrade to libvirt-1.1.4 without success.

Downgrading systemd seems a bit scary to me right now ...


Reproducible: Always

Steps to Reproduce:
1. virsh start vm180

Actual Results:  
# virsh start vm180
error: Failed to start domain vm180
error: Input/output error

Expected Results:  
VM should simply start up.


# journalctl -f -u libvirtd.service

shows:

Dec 03 17:32:56 jupiter libvirtd[24020]: Input/output error
Dec 03 17:33:47 jupiter libvirtd[24020]: Input/output error

The log for the specific VM shows:

2013-12-03 16:33:47.004+0000: starting up
LC_ALL=C
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -name vm180 -S -machine
pc-i440fx-1.5,accel=kvm,usb=off -cpu
Opteron_G5,+bmi1,+perfctr_nb,+perfctr_core,+topoext,+nodeid_msr,+tce,+lwp,+wdt,+skinit,+ibs,+osvw,+cr8legacy,+extapic,+cmp_legacy,+fxsr_opt,+mmxext,+osxsave,+monitor,+ht,+vme
-m 3954 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid
646f757f-2ab7-4fdf-2140-a79396200c6f -no-user-config -nodefaults
-chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/vm180.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=localtime,clock=vm,driftfix=slew -no-kvm-pit-reinjection -no-hpet
-no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/dev/VG01/vm180_disk0,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=threads
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-drive
file=/mnt/tests/grml64-full_2013.02.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw
-device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev
tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:8d:3b:af,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:5 -vga cirrus
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
2013-12-03 16:33:47.013+0000: shutting down
libvirt:  error : libvirtd quit during handshake: Input/output error


# journalctl -f

lists

Dec 03 18:19:31 jupiter systemd-machined[23995]: Failed to start machine
scope: Method "StartTransientUnit" with signature "ssa(sv)" on interface
"org.freedeskt...sn't exist
Dec 03 18:19:31 jupiter libvirtd[2039]: Input/output error


when I issue "virsh start vm180"

---

# journalctl -f -u systemd-machined.service
-- Logs begin at Tue 2013-08-27 23:38:28 CEST. --
Dec 03 18:02:55 jupiter systemd-machined[23995]: Failed to start machine
scope: Method "StartTransientUnit" with signature "ssa(sv)" on interface
"org.freedesktop.systemd1.Manager" doesn't exist
Dec 03 18:04:31 jupiter systemd-machined[23995]: Failed to start machine
scope: Method "StartTransientUnit" with signature "ssa(sv)" on interface
"org.freedesktop.systemd1.Manager" doesn't exist
Dec 03 18:06:49 jupiter systemd-machined[23995]: Failed to start machine
scope: Method "StartTransientUnit" with signature "ssa(sv)" on interface
"org.freedesktop.systemd1.Manager" doesn't exist
Dec 03 18:07:22 jupiter systemd-machined[23995]: Failed to start machine
scope: Method "StartTransientUnit" with signature "ssa(sv)" on interface
"org.freedesktop.systemd1.Manager" doesn't exist
Dec 03 18:08:11 jupiter systemd-machined[23995]: Failed to start machine
scope: Method "StartTransientUnit" with signature "ssa(sv)" on interface
"org.freedesktop.systemd1.Manager" doesn't exist
Dec 03 18:14:02 jupiter systemd[1]:
[/usr/lib64/systemd/system/systemd-machined.service:12] Failed to add
dependency on machine.slice, ignoring: Invalid argument
Dec 03 18:14:02 jupiter systemd[1]:
[/usr/lib64/systemd/system/systemd-machined.service:13] Failed to add
dependency on machine.slice, ignoring: Invalid argument
Dec 03 18:15:27 jupiter systemd-machined[23995]: Failed to start machine
scope: Method "StartTransientUnit" with signature "ssa(sv)" on interface
"org.freedesktop.systemd1.Manager" doesn't exist
Dec 03 18:19:23 jupiter systemd-machined[23995]: Failed to start machine
scope: Method "StartTransientUnit" with signature "ssa(sv)" on interface
"org.freedesktop.systemd1.Manager" doesn't exist
Dec 03 18:19:31 jupiter systemd-machined[23995]: Failed to start machine
scope: Method "StartTransientUnit" with signature "ssa(sv)" on interface
"org.freedesktop.systemd1.Manager" doesn't exist
Comment 1 Stefan G. Weichinger 2013-12-03 18:19:49 UTC
Downgrading systemd to 204-r1 helped.
Seems to be related to the new systemd-machined.service ?
Comment 2 Doug Goldstein (RETIRED) gentoo-dev 2013-12-03 18:32:36 UTC
While I doubt there's anything specific in there, peek in /var/log/libvirt/qemu/<vm>.log

I wouldn't really recommend libvirt + systemd on Gentoo. Things change fast in systemd land and the depends aren't explicitly called out in libvirt for which versions of systemd they need as they're usually tied to whatever Fedora is shipping at the time so there's nothing documented on what the min and max version numbers need to be.
Comment 3 Stefan G. Weichinger 2013-12-03 18:45:20 UTC
(In reply to Doug Goldstein from comment #2)
> While I doubt there's anything specific in there, peek in
> /var/log/libvirt/qemu/<vm>.log
> 
> I wouldn't really recommend libvirt + systemd on Gentoo. Things change fast
> in systemd land and the depends aren't explicitly called out in libvirt for
> which versions of systemd they need as they're usually tied to whatever
> Fedora is shipping at the time so there's nothing documented on what the min
> and max version numbers need to be.

I perfectly understand but in this case I won't be easily able to migrate back to openrc. The server is ~600kms from me and although I have a remote console with "hp IlO" I somehow don't like the idea of doing that (maybe because I am not sure if it's hard to do ...).

For now I stay with the mentioned working versions .. thanks!
Comment 4 Pacho Ramos gentoo-dev 2013-12-09 10:10:45 UTC
Still the same with 1.2.0? (I see it's the same version currently in fedora that uses systemd-208 too):
http://pkgs.fedoraproject.org/cgit/libvirt.git/tree/
Comment 5 Stefan G. Weichinger 2013-12-11 16:23:59 UTC
(In reply to Pacho Ramos from comment #4)
> Still the same with 1.2.0? (I see it's the same version currently in fedora
> that uses systemd-208 too):
> http://pkgs.fedoraproject.org/cgit/libvirt.git/tree/

I am on vacation and can't test that right now. Only after dec 21st.
Comment 6 Stefan G. Weichinger 2013-12-23 21:33:57 UTC
app-emulation/libvirt-1.2.0 and app-emulation/virt-manager-0.10.0-r2 start VMs correctly under systemd-208-r2.

--

other bug, yes ->

virt-manager only shows 2 of my ~10 configured QEMU/KVM-hosts with this release ... with 0.9.5 it shows all of them.
Comment 7 Pacho Ramos gentoo-dev 2013-12-23 22:21:30 UTC
(In reply to Stefan G. Weichinger from comment #6)
> app-emulation/libvirt-1.2.0 and app-emulation/virt-manager-0.10.0-r2 start
> VMs correctly under systemd-208-r2.
> 
> --
> 
> other bug, yes ->
> 
> virt-manager only shows 2 of my ~10 configured QEMU/KVM-hosts with this
> release ... with 0.9.5 it shows all of them.

I would think it's a different problem :|
Comment 8 Stefan G. Weichinger 2013-12-30 11:23:45 UTC
(In reply to Pacho Ramos from comment #7)
> (In reply to Stefan G. Weichinger from comment #6)
> > app-emulation/libvirt-1.2.0 and app-emulation/virt-manager-0.10.0-r2 start
> > VMs correctly under systemd-208-r2.
> > 
> > --
> > 
> > other bug, yes ->
> > 
> > virt-manager only shows 2 of my ~10 configured QEMU/KVM-hosts with this
> > release ... with 0.9.5 it shows all of them.
> 
> I would think it's a different problem :|

Yes. I am not sure if libvirt or virt-manager is the problem and which package to file the bug for ... downgrading VMM also leads to an older libvirt here. pls advise. Thanks.
Comment 9 Alexander Tsoy 2013-12-30 13:32:47 UTC
JFYI: I have no problems with systemd-208-r2 + libvirt-1.1.3.1
Comment 10 Stefan G. Weichinger 2013-12-30 14:42:36 UTC
(In reply to Alexander Tsoy from comment #9)
> JFYI: I have no problems with systemd-208-r2 + libvirt-1.1.3.1

thanks. Do you use virt-manager? If yes, which version?
I am downgrading stuff right now.
Comment 11 Alexander Tsoy 2013-12-30 14:55:22 UTC
(In reply to Stefan G. Weichinger from comment #10)
> (In reply to Alexander Tsoy from comment #9)
> > JFYI: I have no problems with systemd-208-r2 + libvirt-1.1.3.1
> 
> thanks. Do you use virt-manager? If yes, which version?
> I am downgrading stuff right now.

$ qlist -ICv virt-manager
app-emulation/virt-manager-0.10.0
Comment 12 Stefan G. Weichinger 2013-12-30 15:14:22 UTC
(In reply to Alexander Tsoy from comment #11)
> (In reply to Stefan G. Weichinger from comment #10)
> > (In reply to Alexander Tsoy from comment #9)
> > > JFYI: I have no problems with systemd-208-r2 + libvirt-1.1.3.1
> > 
> > thanks. Do you use virt-manager? If yes, which version?
> > I am downgrading stuff right now.
> 
> $ qlist -ICv virt-manager
> app-emulation/virt-manager-0.10.0

Thanks.

My issue:

With virt-manager >= 0.10.0 I only see 2 of my connections to various QEMU/KVM-servers.

With virt-manager-0.9.5 I see all of my connections.
Comment 13 Alexander Tsoy 2013-12-30 15:38:58 UTC
Hmm.. Have a look at ~/.virt-manager/virt-manager.log, may be you'll find something here.
Comment 14 Stefan G. Weichinger 2013-12-30 15:51:29 UTC
(In reply to Alexander Tsoy from comment #13)
> Hmm.. Have a look at ~/.virt-manager/virt-manager.log, may be you'll find
> something here.

I see lines with 2 uris and others with all my URIs in there.

Before the lines with only 2 uris there is this:

ERROR (importer:51) Could not find any typelib for AppIndicator3

Helpful? Thanks!
Comment 15 Alexander Tsoy 2013-12-30 17:21:40 UTC
(In reply to Stefan G. Weichinger from comment #14)
> Helpful? Thanks!

No :)

This may be a result of incomplete settings migration from gconf to gsettings. Try running "gsettings-data-convert".
Comment 16 Stefan G. Weichinger 2013-12-30 18:26:42 UTC
(In reply to Alexander Tsoy from comment #15)
> (In reply to Stefan G. Weichinger from comment #14)
> > Helpful? Thanks!
> 
> No :)
> 
> This may be a result of incomplete settings migration from gconf to
> gsettings. Try running "gsettings-data-convert".

thanks for the suggestion, didn't change the issue.
Comment 17 Alexander Tsoy 2013-12-30 20:54:54 UTC
Compare the output of the following 2 commands:
$ gsettings list-recursively org.virt-manager.virt-manager.connections
$ gconftool-2 -R /apps/virt-manager/connections
If they show different uri lists, then you must recreate all missing connections manually.
Comment 18 Stefan G. Weichinger 2013-12-31 16:33:43 UTC
(In reply to Alexander Tsoy from comment #17)
> Compare the output of the following 2 commands:
> $ gsettings list-recursively org.virt-manager.virt-manager.connections
> $ gconftool-2 -R /apps/virt-manager/connections
> If they show different uri lists, then you must recreate all missing
> connections manually.

*sigh*

Thanks for the pointer anyway ... the 2 commands really show 2 vs more connections. No way to somehow export/import stuff?
Comment 19 Stefan G. Weichinger 2013-12-31 16:49:04 UTC
(In reply to Stefan G. Weichinger from comment #18)
> (In reply to Alexander Tsoy from comment #17)
> > Compare the output of the following 2 commands:
> > $ gsettings list-recursively org.virt-manager.virt-manager.connections
> > $ gconftool-2 -R /apps/virt-manager/connections
> > If they show different uri lists, then you must recreate all missing
> > connections manually.
> 
> *sigh*
> 
> Thanks for the pointer anyway ... the 2 commands really show 2 vs more
> connections. No way to somehow export/import stuff?

forget it, I already recreated them manually. Thanks!
Comment 20 Doug Goldstein (RETIRED) gentoo-dev 2014-01-02 16:04:20 UTC
(In reply to Stefan G. Weichinger from comment #14)
> (In reply to Alexander Tsoy from comment #13)
> > Hmm.. Have a look at ~/.virt-manager/virt-manager.log, may be you'll find
> > something here.
> 
> I see lines with 2 uris and others with all my URIs in there.
> 
> Before the lines with only 2 uris there is this:
> 
> ERROR (importer:51) Could not find any typelib for AppIndicator3
> 
> Helpful? Thanks!

Try 0.10.0-r1 or -r2, I believe I worked around that in one of those bumps.
Comment 21 Marat Nepomnyashy 2014-01-23 02:28:55 UTC
I could not get any of my VMs to start with libvirt-1.1.3.1, virt-manager-0.10.0-r1, and systemd-208-r2.

The VMs started working again when I masked systemd-208-r2, and now I'm running on systemd-204-r1
Comment 22 Marat Nepomnyashy 2014-01-23 02:36:00 UTC
The actual exception I'm getting inside virt-manager (with systemd-208-r2) is:

Error starting domain: Launch helper exited with unknown return code 1

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 100, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 122, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1210, in startup
    self._backend.create()
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 708, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: Launch helper exited with unknown return code 1



Logs under '/var/log/libvirt':

==> ./libvirtd.log <==
2014-01-23 02:34:37.890+0000: 16887: warning : qemuOpenVhostNet:521 : Unable to open vhost-net. Opened so far 0, requested 1

==> ./qemu/mercury6.log <==
2014-01-23 02:34:37.890+0000: starting up
LC_ALL=C PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.7.3:/opt/nvidia-cg-toolkit/bin:/usr/games/bin HOME=/root USER=root QEMU_AUDIO_DRV=spice /usr/bin/qemu-kvm -name mercury6 -S -machine pc-1.1,accel=kvm,usb=off -m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid a7771234-f218-34bb-fab1-ab8713698ff4 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/mercury6.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/var/lib/libvirt/images/mercury6.qcow2.img,if=none,id=drive-ide0-0-0,format=qcow2 -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=19,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:ff:42:00,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -vga vmware -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7

==> ./libvirtd.log <==
2014-01-23 02:34:37.894+0000: 16887: error : virDBusCallMethod:1173 : Launch helper exited with unknown return code 1

==> ./qemu/mercury6.log <==
2014-01-23 02:34:37.894+0000: shutting down
libvirt:  error : libvirtd quit during handshake: Input/output error
Comment 23 Marat Nepomnyashy 2014-01-24 02:17:24 UTC
Unfortunately just masking 'systemd-208-r2' to run on 'systemd-204-r1' made it impossible to upgrade or emerge other packages, as emerge kept complaining about all kinds of conflicts between 'systemd' and 'udev'.

Ultimately what I ended up doing is:

emerge --unmerge systemd
emerge --unmerge gentoo-systemd-integration


I'm now able to run 'libvirt' virtual machines, and I believe my system is now in a relatively stable state.

Some issues I'm still having is that I have to manually 'modprobe tun', but I'm not sure if that's related to this specific problem.

Hope this turns out to be useful for whoever else is having problem with 'systemd'.  It took me several days to figure this out.  But damn, I just can't help by wonder if this really has to be so counter-intuitive.
Comment 24 Pavel Šimerda 2014-04-08 08:31:50 UTC
Removing the systemd use flag from libvirt seems to do the job for me:

euse -D systemd -p app-emulation/libvirt

Is there any fix on the way so that I don't have to use hacks for intrinsic linux developer's tools like libvirt?
Comment 25 Pavel Šimerda 2014-04-08 21:51:54 UTC
(In reply to Pavel Šimerda (pavlix) from comment #24)
> Removing the systemd use flag from libvirt seems to do the job for me:
> 
> euse -D systemd -p app-emulation/libvirt

This was a misinterpretation. It's necessary to explicitly uninstall the systemd package to get libvirt and virt-manager working again.

> Is there any fix on the way so that I don't have to use hacks for intrinsic
> linux developer's tools like libvirt?
Comment 26 Pavel Šimerda 2014-04-10 18:33:09 UTC
(In reply to Marat Nepomnyashy from comment #21)
> The VMs started working again when I masked systemd-208-r2, and now I'm
> running on systemd-204-r1

Btw, I see that systemd-204-r1 is no longer in the portage and this bug doesn't seem likely to be processed any time soon. Would it be possible to put back systemd-204-r1 to allow for an easy step back to a working version?
Comment 27 Marat Nepomnyashy 2014-04-10 18:39:00 UTC
(In reply to Pavel Šimerda (pavlix) from comment #26)
> (In reply to Marat Nepomnyashy from comment #21)
> > The VMs started working again when I masked systemd-208-r2, and now I'm
> > running on systemd-204-r1
> 
> Btw, I see that systemd-204-r1 is no longer in the portage and this bug
> doesn't seem likely to be processed any time soon. Would it be possible to
> put back systemd-204-r1 to allow for an easy step back to a working version?

Take a look at my follow-up comment on 2014-01-24 02:17:24 UTC -- retaining systemd will still cause all kinds of problems, the simplest thing to do is to remove it entirely.
Comment 28 Pavel Šimerda 2014-04-11 11:22:39 UTC
(In reply to Marat Nepomnyashy from comment #27)
> Take a look at my follow-up comment on 2014-01-24 02:17:24 UTC -- retaining
> systemd will still cause all kinds of problems, the simplest thing to do is
> to remove it entirely.

All recent resources including Gentoo Wiki make the impression that systemd is a supported alternative on Gentoo. While I value your well meant and friendly advice, I wouldn't consider it a solution. In fact I would like to have at least a patch to fix libvirt's systemd detection or what else causes the reported problem and the problem other users have.

I'm not 100% sure that my issue is the same as the one described in the bug report, but my goal is to have systemd, openrc and libvirt installed and have a fully working system whether I boot to systemd or openrc. I've been searching on the web with no result, this bug also doesn't bring any, I'm trying to contact some libvirt developers now.
Comment 29 Richard Freeman gentoo-dev 2014-04-11 13:16:22 UTC
(In reply to Pavel Šimerda (pavlix) from comment #28)
> 
> All recent resources including Gentoo Wiki make the impression that systemd
> is a supported alternative on Gentoo. While I value your well meant and
> friendly advice, I wouldn't consider it a solution. 

I doubt it was intended as one.  I suggest avoiding debates in bugs about what is/isn't supported.  Instead focus on the fact that for whatever reason libvirt works if systemd isn't installed, and doesn't work if it is, and we can try to figure out why that is.  Nobody has closed the bug.
Comment 30 Matthias Maier gentoo-dev 2014-10-31 06:41:34 UTC
Currently, I'm running sys-apps/systemd-215-r3 with app-emulation/libvirt-1.2.9.

Is this regression reproducible with these versions? If yes, what user configuration with respect to libvirt is installed under /etc/systemd/system (i.e. content of all files with libvirt in name ignoring everything under /etc/systemd/system/multi-user.target.wants)
Comment 31 Matthias Maier gentoo-dev 2014-11-20 12:35:11 UTC
I cannot reproduce with current versions in tree.