libvirt 0.9.13-r1 works fine. Starting libvirt 0.10.2-r3 or 1.0.0 succeeds and using virsh for, e.g. editing domains does work. As soon as a qemu domain is started, libvirt segfaults. Reproducible: Always Steps to Reproduce: 1. /etc/init.d/libvirtd start 2. virsh start myQemuDomain Actual Results: output of virsh: error: Failed to start domain myQemuDomain error: End of file while reading data: Input/output error error: Failed to reconnect to the hypervisor Expected Results: starting the domain working: ============ [ebuild R ] app-emulation/qemu-1.1.2-r2 USE="aio caps curl jpeg ncurses png python threads uuid vhost-net vnc -alsa -bluetooth -brltty -debug -doc -fdt -mixemu -opengl -pulseaudio -rbd -sasl -sdl -smartcard -spice -static -systemtap -tci -tls -usbredir -vde -virtfs -xattr -xen -xfs" QEMU_SOFTMMU_TARGETS="x86_64 -alpha -arm -cris -i386 -lm32 -m68k -microblaze -microblazeel -mips -mips64 -mips64el -mipsel -ppc -ppc64 -ppcemb -s390x -sh4 -sh4eb -sparc -sparc64 -xtensa -xtensaeb" QEMU_USER_TARGETS="x86_64 -alpha -arm -armeb -cris -i386 -m68k -microblaze -microblazeel -mips -mipsel -ppc -ppc64 -ppc64abi32 -s390x -sh4 -sh4eb -sparc -sparc32plus -sparc64 -unicore32" 0 kB [ebuild R ] app-emulation/libvirt-0.9.13-r1 USE="caps libvirtd nls numa python qemu (selinux) udev virt-network -audit -avahi -debug -iscsi -lvm -lxc -macvtap -nfs -openvz -parted -pcap -phyp -policykit -rbd -sasl -uml -vepa -virtualbox -xen" 0 kB not working: ============ [ebuild U ~] app-emulation/libvirt-1.0.0 [0.9.13-r1] USE="caps libvirtd nls numa python qemu (selinux) udev virt-network -audit -avahi -debug -firewalld% -iscsi -lvm -lxc -macvtap -nfs -openvz -parted -pcap -phyp -policykit -rbd -sasl -uml -vepa -virtualbox -xen" 0 kB [ebuild U ] app-emulation/libvirt-0.10.2-r3 [0.9.13-r1] USE="caps libvirtd nls numa python qemu (selinux) udev virt-network -audit -avahi -debug -iscsi -lvm -lxc -macvtap -nfs -openvz -parted -pcap -phyp -policykit -rbd -sasl -uml -vepa -virtualbox -xen" 0 kB
Portage 2.1.11.31 (hardened/linux/amd64/no-multilib/selinux, gcc-4.5.4, glibc-2.15-r3, 3.5.4-hardened-r1 x86_64) ================================================================= System uname: Linux-3.5.4-hardened-r1-x86_64-Intel-R-_Core-TM-_i7-3770_CPU_@_3.40GHz-with-gentoo-2.1 Timestamp of tree: Sat, 17 Nov 2012 01:45:01 +0000 ld GNU ld (GNU Binutils) 2.22 app-shells/bash: 4.2_p37 dev-lang/python: 2.7.3-r2, 3.2.3 dev-util/cmake: 2.8.9 dev-util/pkgconfig: 0.27.1 sys-apps/baselayout: 2.1-r1 sys-apps/openrc: 0.9.8.4 sys-apps/sandbox: 2.5 sys-devel/autoconf: 2.68 sys-devel/automake: 1.11.6 sys-devel/binutils: 2.22-r1 sys-devel/gcc: 4.5.4 sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4-r1 sys-devel/make: 3.82-r3 sys-kernel/linux-headers: 3.4-r2 (virtual/os-headers) sys-libs/glibc: 2.15-r3 Repositories: gentoo ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="*" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=native -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-march=native -O2 -pipe" DISTDIR="/usr/portage/distfiles" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch protect-owned sandbox selinux sesandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="rsync://de-mirror.org/gentoo/ http://distfiles.gentoo.org http://www.ibiblio.org/pub/Linux/distributions/gentoo" LANG="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j9" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="" SYNC="rsync://rsync.de.gentoo.org/gentoo-portage" USE="amd64 berkdb bzip2 cli cracklib crypt cups cxx dbus dri fuse gdbm gnutls gpm hardened iconv ipv6 justify mmx modules mudflap ncurses nls nptl open_perms openmp pam pax_kernel pcre perl pppd python readline selinux session sse sse2 ssl tcpd unicode urandom zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" PHP_TARGETS="php5-3" PYTHON_TARGETS="python2_7 python3_2" RUBY_TARGETS="ruby18 ruby19" USERLAND="GNU" VIDEO_CARDS="fbdev glint intel mach64 mga nouveau nv r128 radeon savage sis tdfx trident vesa via vmware dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
Created attachment 329710 [details] backtrace of libvirt libvirt with -ggdb
Created attachment 329720 [details] strace -f libvirtd
This is a SELinux issue with marking a file or a task with a specific permission. I'm unfortunately not very familiar with how to debug or track this down so you'll have to get the Gentoo SELinux guys to help. I also don't run any Gentoo SELinux machines so I won't really be able to help debug this.
Do you notice any errors in the dmesg output and in the SELinux denial logs?
There are quite some denies in avc.log, but that's true for 0.9.13-r1 as well. The system runs in permissive mode. I'm going to fetch the logs as soon as I can stop the 'productive' domains and switch to 0.10.2-r3 again.
I don't see anything in the trace that would give me a "yes, this is a SELinux issue" feeling. On the contrary, you are running in permissive mode, and the application looks like it is just checking information. But since the application is SELinux-aware, I cannot guarantee that it /isn't/ due to SELinux either. Anything in dmesg that looks like a grSecurity/PaX enforcement being the culprit?
Oh, but if it is SELinux, then it is probably because the application doesn't expect to be in sysadm_t domain. There is a virsh_t domain and a few others. Care to look what the domain ought to be? Once we know that, then we can see how to get there.
-- grsecurity is not enabled -- pax is ... not existent anymore? Did I miss anything? In my last kernel 3.4.2-h-r1 pax was enabled; in the currently running 3.5.4-h-r1, the options for pax are not available at all?? Anyhow, pax seems to be not enabled either. -- selinux-virt and selinux-qemu are installed and used. -- I attached the dmesg for run_init libvirt 0.10.2. Starting a qemu domain afterwards does not produce any further messages before libvrit segfaults. -- I also attached the dmesg for starting a qemu domain using libvirt 0.9.2. It complains about the image file being unlabeled which I don't really understand, because I added the context virt_image_t to the respective folder and libvirt does dynamically relabel it forth to svirt and back to virt each time.
Created attachment 330398 [details] dmesg of run_init libvirt 0.10.2-r3
Created attachment 330402 [details] dmesg of starting qemu with libvirt 0.9.13-r1
Given that the whole application is SELinux-aware, I wouldn't be surprised that the engineers didn't put in the correct checks for permissive mode. Can you switch to enforcing mode right before you start libvirtd, and then start it and report back with the failure and denials? Many of the denials shown are results of earlier denials, and we need to filter those out. Running in enforcing mode should help us with that. Make sure that you have a console open where you are sysadm_t and, if that is through SSH, make sure the daemon runs in the sshd_t domain (otherwise switching to enforcing mode might crash the daemon and leave you out of the system).
I used audit2allow for cleaning up the avc.log. However checkmodule gives: libsepol.check_assertion_helper: neverallow violated by allow virtd_t memory_device_t:chr_file { read }; Installing the resulting module (without this memory_device_t), setting enforce to 1 and starting libvirtd 0.10.2-r3 gives: $> run_init libvirtd Authenticating root. Password: 2012-11-30 22:02:25.869+0000: 17169: info : libvirt version: 0.10.2 2012-11-30 22:02:25.869+0000: 17169: error : virSysinfoRead:763 : internal error Failed to find path for dmidecode binary 2012-11-30 22:02:25.870+0000: 17169: error : virSecurityDriverLookup:78 : unsupported configuration: Security driver selinux not enabled 2012-11-30 22:02:25.870+0000: 17169: error : qemuSecurityInit:316 : Failed to initialize security drivers 2012-11-30 22:02:25.870+0000: 17169: error : virStateInitialize:798 : Initialization of QEMU state driver failed 2012-11-30 22:02:25.870+0000: 17169: error : daemonRunStateInit:766 : Driver state initialization failed and exit
You can enable that too through "dev_read_raw_memory(virtd_t)" but I can't add that in the gentoo policy until I also know what the other denials were that you got fixed through audit2allow :-(
I added the dev_read_raw_memory(virtd_t) (as you see below), but without luck for 0.10.2. However, libvirt 0.9.13 is now running in enforced strict without issues. $> echo 0 > /selinux/enforce §> run_init libvirtd 2012-12-04 21:22:38.481+0000: 4379: info : libvirt version: 0.10.2 2012-12-04 21:22:38.481+0000: 4379: error : virSecurityDriverLookup:78 : unsupported configuration: Security driver selinux not enabled 2012-12-04 21:22:38.481+0000: 4379: error : qemuSecurityInit:316 : Failed to initialize security drivers 2012-12-04 21:22:38.481+0000: 4379: error : virStateInitialize:798 : Initialization of QEMU state driver failed 2012-12-04 21:22:38.481+0000: 4379: error : daemonRunStateInit:766 : Driver state initialization failed and no entries in avc.log or dmesg. I added the following rules, all but the last gathered with audit2allow. #============= qemu_t ============== allow qemu_t svirt_var_run_t:dir { write add_name }; allow qemu_t svirt_var_run_t:sock_file create; allow qemu_t sysfs_t:file { read open }; allow qemu_t unlabeled_t:file { write read getattr open }; allow qemu_t vhost_device_t:chr_file { read write ioctl }; #============= sysadm_dbusd_t ============== allow sysadm_dbusd_t self:capability sys_resource; allow sysadm_dbusd_t self:capability2 block_suspend; allow sysadm_dbusd_t self:process setrlimit; allow sysadm_dbusd_t sysfs_t:file { read open }; #============= virtd_t ============== allow virtd_t default_t:dir read; allow virtd_t dhcpd_port_t:udp_socket name_bind; allow virtd_t dmidecode_exec_t:file { read getattr open execute execute_no_trans }; allow virtd_t dns_port_t:tcp_socket name_bind; allow virtd_t dns_port_t:udp_socket name_bind; allow virtd_t file_context_t:file { read getattr open }; allow virtd_t initrc_tmp_t:file append; allow virtd_t initrc_var_run_t:file getattr; allow virtd_t initrc_var_run_t:file { read write open }; allow virtd_t memory_device_t:chr_file { read open }; allow virtd_t node_t:udp_socket node_bind; allow virtd_t self:capability net_bind_service; allow virtd_t self:capability sys_rawio; allow virtd_t self:capability2 mac_admin; allow virtd_t self:packet_socket { create ioctl }; allow virtd_t self:process setsockcreate; allow virtd_t sysctl_kernel_t:dir search; allow virtd_t sysctl_kernel_t:file { read open }; allow virtd_t unlabeled_t:file { relabelfrom getattr setattr read relabelto open }; allow virtd_t var_run_t:sock_file { create unlink }; allow virtd_t vhost_device_t:chr_file { read write open }; dev_read_raw_memory(virtd_t) and the following local context for my images: /data/r1/images(/.*)? system_u:object_r:virt_image_t
(In reply to comment #15) > $> echo 0 > /selinux/enforce typo: echo 1, of course
Looks like in the virt code, the following function returns a SECURITY_DRIVER_DISABLE state: """ static int virSecuritySELinuxSecurityDriverProbe(const char *virtDriver) { if (!is_selinux_enabled()) return SECURITY_DRIVER_DISABLE; if (virtDriver && STREQ(virtDriver, "LXC")) { #if HAVE_SELINUX_LXC_CONTEXTS_PATH if (!virFileExists(selinux_lxc_contexts_path())) #endif return SECURITY_DRIVER_DISABLE; } return SECURITY_DRIVER_ENABLE; } """ As is_selinux_enabled() will return true, this isn't the culprit. Your USE flags also tell USE=-lxc so I assume you are not using LXC, but can you confirm this? You don't have any domains using the libvirt_lxc emulator do you?
Yes, use flag lxc is not (was never) set. I only use qemu. [ebuild U ] app-emulation/libvirt-0.10.2-r3 [0.9.13-r1] USE="caps libvirtd nls numa python qemu (selinux) udev virt-network -audit -avahi -debug -iscsi -lvm -lxc -macvtap -nfs -openvz -parted -pcap -phyp -policykit -rbd -sasl -uml -vepa -virtualbox -xen" 0 kB
Yes, but can you also confirm that your domains (the xml files that define your virtual guests and such) do not refer to LXC?
$> grep -ri "[^-]lxc" /etc $> grep -ri "lxc" /var/lib/ both show nothing relevant $> eix -Ic --installed-with-use lxc No matches found. Anything else I could check?
In that case, is_selinux_enabled() is probably not returning 1, which is very weird, since even in permissive mode it should be returning 1. """ hpl ~ # sestatus [...] Current mode: enforcing hpl ~ # ~swift/Development/build/tmp/test is_selinux_enabled() = 1 hpl ~ # setenforce 0 hpl ~ # sestatus [...] Current mode: permissive hpl ~ # ~swift/Development/build/tmp/test is_selinux_enabled() = 1 """ From a simple strace, it looks like it checks if /sys/fs/selinux exists: """ statfs("/sys/fs/selinux", {f_type=0xf97cff8c, f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=4096}) = 0 statfs("/sys/fs/selinux", {f_type=0xf97cff8c, f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=4096}) = 0 stat("/sys/fs/selinux", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0 """ Can you check if /sys/fs/selinux does exist? Perhaps somewhere along the route, the libvirt cannot check /sys/fs/selinux (be it due to permissions or because it is mounted elsewhere)?
$> sestatus SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: strict Current mode: enforcing .. $> mount | grep selinux selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime) none on /selinux type selinuxfs (rw) dr-xr-xr-x. 11 root root system_u:object_r:sysfs_t 0 Nov 17 14:08 /sys drwxr-xr-x. 5 root root system_u:object_r:sysfs_t 0 Nov 17 14:08 /sys/fs drwxr-xr-x. 7 root root system_u:object_r:security_t 0 Nov 17 14:08 /sys/fs/selinux While in permissive mode, libvirtd starts and reads /sys/fs/selinux, but dies once qemu starts. How can I start strace libvirtd in enforcing mode?
Oh I thought the failures were also in permissive mode. If this occurs in enforcing mode, then we need to check is there are denials somewhere that SELinux is enforcing the domain to get information from the SELinux state. Try running with dontaudit's disabled (semodule -DB), clear your avc log and reproduce. What does the avc log contain then?
updated to current stable selinux libs and policies 2.20120725-r8 and used: libvirt 1.0.0 enforce=1 semodule -DB increased /proc/sys/kernel/printk_ratelimit(_burst) dmesg output attached for: §> run_init libvirt $> virsh start DOMAIN with terminating segfault
Created attachment 333934 [details] libvirt 1.0.0, semodule -DB, enforce=1
Just a side note: libvirt 0.10.2-r3 (and upwards, I guess) works without any further changes if(f) selinux is disabled: - disabled selinux in /etc/selinux/config - removed security_driver = "selinux" from /etc/libvirt/qemu.conf - rebooted However, that was not what I was looking for. I would like to have the host running selinux strict.
I'm going to ask some help on this on our online meeting today (hopefully some other developers with SELinux have or can use libvirt). I'm not able to use libvirt on my system so it's harder for me to work on this :-(
Sadly no SELinux developer with libvirt (or they're afraid of saying it). In the strace you showed, there is the following output as well: [pid 29965] --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0} --- [pid 29965] write(4, "Caught Segmentation violation", 29) = 29 [pid 29965] write(4, " dumping internal log buffer:\n", 30) = 30 [pid 29965] write(4, "\n\n ====== start of log =====\n"..., 33) = 33 [pid 29965] write(4, "2012-11-16 18:43:35.006+00002995"..., 3648) = 3648 [pid 29965] write(4, "\n\n ====== end of log =====\n\n", 32) = 32 Do you have the full output on this somewhere?
(In reply to comment #26) > Just a side note: libvirt 0.10.2-r3 (and upwards, I guess) works without any > further changes if(f) selinux is disabled: > > - disabled selinux in /etc/selinux/config > - removed security_driver = "selinux" from /etc/libvirt/qemu.conf > - rebooted > > However, that was not what I was looking for. I would like to have the host > running selinux strict. +1
I don't use SELinux so I don't really have experience with it. You might find some help on the libvirt-users ML. Any suggestions people on there give I'll be glad to implement.
(In reply to Sven Vermeulen from comment #28) > Sadly no SELinux developer with libvirt (or they're afraid of saying it). > > In the strace you showed, there is the following output as well: > > [pid 29965] --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0} > --- > [pid 29965] write(4, "Caught Segmentation violation", 29) = 29 > [pid 29965] write(4, " dumping internal log buffer:\n", 30) = 30 > [pid 29965] write(4, "\n\n ====== start of log =====\n"..., 33) = 33 > [pid 29965] write(4, "2012-11-16 18:43:35.006+00002995"..., 3648) = 3648 > [pid 29965] write(4, "\n\n ====== end of log =====\n\n", 32) = 32 > > Do you have the full output on this somewhere? Going to resolve this as NEEDINFO until the above can be provided.
