Triggering soft reboot from within a guest freezes qemu on shutdown/restart. Issue was replicated on multiple hosts and guests using both regular and hardened profile. This is a critical issue for all productive systems exposing guests to users, leaving them with a dead vm as soon as a reboot is triggered. Reproducible: Always Steps to Reproduce: 1. Start guest 2. Soft reboot guest (by eg. invoking `reboot` within guest) Actual Results: Guest freezes on shutdown/restart. `virsh list` shows the vm as paused in this state. trying to resume the vm using `virsh resume` yields the following error: ```error: internal error: unable to execute QEMU command 'cont': Resetting the Virtual Machine is required``` Expected Results: Guest should restart normally. Portage 2.3.3 (python 3.4.5-final-0, default/linux/amd64/13.0, gcc-4.9.4, glibc-2.23-r3, 4.9.16-gentoo x86_64) ================================================================= System uname: Linux-4.9.16-gentoo-x86_64-Intel-R-_Xeon-R-_CPU_E3-1275L_v3_@_2.70GHz-with-gentoo-2.3 KiB Mem: 8137632 total, 3898992 free KiB Swap: 16777212 total, 16768372 free Timestamp of repository gentoo: Tue, 02 May 2017 02:15:01 +0000 sh bash 4.3_p48-r1 ld GNU ld (Gentoo 2.26.1 p1.0) 2.26.1 ccache version 3.2.4 [enabled] app-shells/bash: 4.3_p48-r1::gentoo dev-lang/perl: 5.22.3_rc4::gentoo dev-lang/python: 2.7.12::gentoo, 3.4.5::gentoo dev-util/ccache: 3.2.4::gentoo dev-util/cmake: 3.7.2::gentoo dev-util/pkgconfig: 0.28-r2::gentoo sys-apps/baselayout: 2.3::gentoo sys-apps/openrc: 0.23.2::gentoo sys-apps/sandbox: 2.10-r3::gentoo sys-devel/autoconf: 2.69::gentoo sys-devel/automake: 1.15::gentoo sys-devel/binutils: 2.26.1::gentoo sys-devel/gcc: 4.9.4::gentoo sys-devel/gcc-config: 1.7.3::gentoo sys-devel/libtool: 2.4.6-r3::gentoo sys-devel/make: 4.2.1::gentoo sys-kernel/linux-headers: 4.4::gentoo (virtual/os-headers) sys-libs/glibc: 2.23-r3::gentoo Repositories: gentoo location: /usr/portage sync-type: rsync sync-uri: rsync://rsync.europe.gentoo.org/gentoo-portage priority: -1000 chutzpah location: /var/lib/layman/chutzpah masters: gentoo priority: 50 ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -pipe -march=native -O2 -pipe -fomit-frame-pointer" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -pipe -march=native -O2 -pipe -fomit-frame-pointer" DISTDIR="/usr/portage/distfiles" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs ccache config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://ftp.halifax.rwth-aachen.de/gentoo/ ftp://sunsite.informatik.rwth-aachen.de/pub/Linux/gentoo ftp://ftp.uni-erlangen.de/pub/mirrors/gentoo http://mirrors.evowise.com/gentoo/ ftp://ftp.wh2.tu-dresden.de/pub/mirrors/gentoo http://mirror.qubenet.net/mirror/gentoo/ http://ftp.uni-erlangen.de/pub/mirrors/gentoo rsync://mirror.netcologne.de/gentoo/ http://mirror.netcologne.de/gentoo/ rsync://ftp.snt.utwente.nl/gentoo" LANG="en_US.UTF-8" LC_ALL="en_US.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j2" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" USE="acl acpi amd64 berkdb btrfs bzip2 cli cracklib crypt cxx device-mapper dhcp dri ext4 fortran gdbm hddtemp hvm iconv ipv6 kvm lm_sensors logrotate loop-aes lvm lvm2 lvm2create_initrd modules multilib ncurses nls nptl openmp pam pcre readline sasl seccomp session ssl tcpd unicode virt-network xattr zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 fma3 mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6" PYTHON_SINGLE_TARGET="python3_4" PYTHON_TARGETS="python2_7 python3_4" QEMU_SOFTMMU_TARGETS="x86_64" QEMU_USER_TARGETS="x86_64" RUBY_TARGETS="ruby21 ruby22" USERLAND="GNU" VIDEO_CARDS="amdgpu fbdev intel nouveau radeon radeonsi vesa dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
> Steps to Reproduce: > 1. Start guest > 2. Soft reboot guest (by eg. invoking `reboot` within guest) > > Actual Results: > Guest freezes on shutdown/restart. > > `virsh list` shows the vm as paused in this state. > trying to resume the vm using `virsh resume` yields the following error: > > ```error: internal error: unable to execute QEMU command 'cont': Resetting > the Virtual Machine is required``` > > Expected Results: > Guest should restart normally. I'm sorry - I cannot reproduce with my VMs. Soft reboot works just fine. Can you provide a minimal qemu configuration for example for some known live cd/dvd image, that triggers this issue?
that's odd. I'm seeing this with a gentoo livecd (2017-03), multiple other gentoo guests, sabayon & debian. so I presume it's not guest related. qemu-2.7.0 works fine in all configurations. find attached one of the hosts kernel config.
Created attachment 471578 [details] Host kernel config
issue persists with qemu-2.9.0-r2
My hardened (stable) Gentoo also was affected by that or a very similar problem. It simply was impossible to reboot or really shut down any of my guests (all Gentoo VMS also) without forceful shutdown. Switching to qemu-2.9.0-r56 solved that issue. Now the VMS are no longer hanging with a black screen, instead seabios boot messages appear again as they should. Unfortunately the reboot still failed with the new qemu (and the pulled in newer seabios version): The VMS were now only able to find the boot disks after a power on. Resetting a VM even prior to booting Linux reproducible did not find the boot disks. I was able to fix that by getting the got version of seabios and replacing bios.bin with my self compiled git version. So my system is now again working normally.
Same problem here. ----------- app-emulation/qemu ~amd64 # required by app-emulation/qemu-2.9.0-r56::gentoo[pin-upstream-blobs,qemu_softmmu_targets_x86_64,-qemu_softmmu_targets_i386] # required by app-emulation/libvirt-3.3.0::gentoo[qemu] # required by @selected # required by @world (argument) =sys-firmware/edk2-ovmf-2017_pre20170505 ~amd64 # required by app-emulation/qemu-2.9.0-r56::gentoo[pin-upstream-blobs,qemu_softmmu_targets_x86_64,-qemu_softmmu_targets_i386] # required by app-emulation/libvirt-3.3.0::gentoo[qemu] # required by @selected # required by @world (argument) =sys-firmware/seabios-1.10.2 ~amd64 --------------- These versions fixing it for me without further modifications.
(In reply to Alexander Wetzel from comment #5) > I was able to fix that by getting the got version of seabios and replacing > bios.bin with my self compiled git version. Do we have to stabilize a different firmware version?
with 2.10.0 instead of freezing, soft reboot makes guests always boot from cd instead of hd.
I get the same issue using qemu-2.9.0-r2. Changing the guest's machine type from "pc-1.2" to "pc-i440fx-2.8" works as expected. Here's a clue: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=845964 Here's the qemu log on guest reboot: KVM internal error. Suberror: 1 emulation failure EAX=000e2960 EBX=0000db40 ECX=1ffbfeec EDX=1fece800 ESI=000f16c0 EDI=1ffbff60 EBP=1ffb2420 ESP=00006fbc EIP=c09208cb EFL=00010006 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA] CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA] SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA] DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA] FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA] GS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA] LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy GDT= 000f76a0 00000037 IDT= 000f76de 00000000 CR0=00000011 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000000 Code=00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 `emerge --info`: Portage 2.3.6 (python 3.4.5-final-0, hardened/linux/amd64/no-multilib, gcc-5.4.0, glibc-2.23-r4, 4.7.10-hardened-hypervisor x86_64) ================================================================= System Settings ================================================================= System uname: Linux-4.7.10-hardened-hypervisor-x86_64-Intel-R-_Xeon-R-_CPU_E5-2650_0_@_2.00GHz-with-gentoo-2.3 KiB Mem: 99052584 total, 2099940 free KiB Swap: 31450364 total, 15162428 free Timestamp of repository gentoo: Mon, 13 Nov 2017 02:15:01 +0000 sh bash 4.3_p48-r1 ld GNU ld (Gentoo 2.28 p1.2) 2.28 app-shells/bash: 4.3_p48-r1::gentoo dev-lang/perl: 5.24.1-r2::gentoo dev-lang/python: 2.7.12::gentoo, 3.4.5::gentoo dev-util/cmake: 3.7.2::gentoo dev-util/pkgconfig: 0.28-r2::gentoo sys-apps/baselayout: 2.3::gentoo sys-apps/openrc: 0.26.3::gentoo sys-apps/sandbox: 2.10-r3::gentoo sys-devel/autoconf: 2.69::gentoo sys-devel/automake: 1.15-r2::gentoo sys-devel/binutils: 2.28-r2::gentoo sys-devel/gcc: 5.4.0-r3::gentoo sys-devel/gcc-config: 1.7.3::gentoo sys-devel/libtool: 2.4.6-r3::gentoo sys-devel/make: 4.2.1::gentoo sys-kernel/linux-headers: 4.4::gentoo (virtual/os-headers) sys-libs/glibc: 2.23-r4::gentoo Repositories: gentoo location: /usr/portage sync-type: rsync sync-uri: rsync://ftp.aaaa/gentoo priority: -1000 aaaa location: /usr/aaaa-overlay masters: gentoo ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="*" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -pipe -march=native" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -pipe -march=native" DISTDIR="/usr/distfiles" EMERGE_DEFAULT_OPTS="--quiet-build --jobs=1 --ask-enter-invalid" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="ftp://ftp.aaaa/distfiles" LANG="en_US.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j8" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" USE="amd64 bash-completion berkdb bzip2 caps cli cracklib crypt cxx dri gdbm hardened hardware iconv ipv6 justify logrotate modules ncurses nls nptl openmp pam pcre pie readline seccomp session ssl ssp tcpd unicode urandom vim-syntax xattr xtpax zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2 sse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="pc" INPUT_DEVICES="libinput keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6" POSTGRES_TARGETS="postgres9_5" PYTHON_SINGLE_TARGET="python3_4" PYTHON_TARGETS="python3_4" RUBY_TARGETS="ruby22" USERLAND="GNU" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS ================================================================= Package Settings ================================================================= app-emulation/qemu-2.9.0-r2::gentoo was built with the following: USE="aio bzip2 curl fdt filecaps jpeg ncurses nls pin-upstream-blobs png seccomp spice vhost-net vnc xattr -accessibility -alsa -bluetooth -caps -debug (-glusterfs) -gnutls -gtk -gtk2 -infiniband -iscsi -lzo -nfs -numa -opengl -pulseaudio -python -rbd -sasl -sdl -sdl2 (-selinux) -smartcard -snappy -ssh -static -static-user -systemtap -tci -test -usb -usbredir -vde -virgl -virtfs -vte -xen -xfs" LINGUAS="-bg -de_DE -fr_FR -hu -it -tr -zh_CN" PYTHON_TARGETS="python2_7" QEMU_SOFTMMU_TARGETS="x86_64 -aarch64 -alpha -arm -cris -i386 -lm32 -m68k -microblaze -microblazeel -mips -mips64 -mips64el -mipsel -moxie -nios2 -or1k -ppc -ppc64 -ppcemb -s390x -sh4 -sh4eb -sparc -sparc64 -tricore -unicore32 -xtensa -xtensaeb" QEMU_USER_TARGETS="-aarch64 -alpha -arm -armeb -cris -hppa -i386 -m68k -microblaze -microblazeel -mips -mips64 -mips64el -mipsel -mipsn32 -mipsn32el -nios2 -or1k -ppc -ppc64 -ppc64abi32 -ppc64le -s390x -sh4 -sh4eb -sparc -sparc32plus -sparc64 -tilegx -x86_64" >>> Attempting to run pkg_info() for 'app-emulation/qemu-2.9.0-r2' Using: app-emulation/spice-protocol-0.12.12 sys-firmware/ipxe-1.0.0_p20160620 sys-firmware/seabios-1.10.1 USE=binary sys-firmware/vgabios-0.7a
The issue is with SeaBIOS 1.10.2, upon soft reboot (initiated from within the guest) the bios hangs with "boot failed could not read the boot disk". Installing the latest git (rel-1.11.0-4) resolved this issue for me, that bios rev now presents virtio pci as a valid boot option. git clone git://git.seabios.org/seabios.git seabios cd seabios make sudo cp out/bios.bin /usr/share/seabios I believe this is the relevant upstream commit enhancement. https://review.coreboot.org/cgit/seabios.git/commit/?id=dd9bba5b9c1d5175a2757f3fdc9d554b4c8eea3a (This behavior can be observed from monitoring the guest console, vnc in my instances. My hunch is you can workaround the issue by not using virtio bus in your guest configs, have not tested)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=24e0876ca53199dcde6ee2e8384e9f9c6319edff commit 24e0876ca53199dcde6ee2e8384e9f9c6319edff Author: Matthias Maier <tamiko@gentoo.org> AuthorDate: 2018-02-11 21:15:07 +0000 Commit: Matthias Maier <tamiko@gentoo.org> CommitDate: 2018-02-11 21:30:27 +0000 sys-firmware/seabios: version bump to 1.11.0 Bug: https://bugs.gentoo.org/617232 Package-Manager: Portage-2.3.24, Repoman-2.3.6 sys-firmware/seabios/Manifest | 2 + sys-firmware/seabios/seabios-1.11.0.ebuild | 137 +++++++++++++++++++++++++++++ 2 files changed, 139 insertions(+)}
Please check with 1.11.0 (which will be stabilized soon) and reopen if problem persits.
