Because of the mentioned issue in the summary, one cannot use that feature, when he wants to use qemu-kvm. Since UDEREF is a nice security related feature, it would be nice to get it working with qemu-kvm.
This issue affects VirtualBox and kqemu as well. I'm willing to bet it affects VMWare as well so this seems like you're going to need some changes to some of the interfaces to make this work.
is this the same as http://forums.grsecurity.net/viewtopic.php?f=1&t=2324 ? i guess it's the per-CPU PGD stuff that's used by both KERNEXEC and UDEREF on amd64, and i don't think there's much we can fix there... at most you could remove it from KERNEXEC (at the expense of security), but it's really mandatory for UDEREF. what may be possible is perhaps to reduce the performance impact if someone can give me kernel profiles (i won't have time for this in the foreseeable future myself) so that i can see where the virtualization code is hit most.
Back when the UDEREF option was introduced, only this function caused the slowdown, while KERNEXEC did not produce a slowdown. But recent tests (h-s-2.6.32-r10 and h-s-2.6.34) show me, that this has changed and KERNEXEC does now produce the same slowdown as originally only UDEREF did. So there is a good chance, that it is the same as the mentioned thread. If you can give me a link or tell me, how to create the needed kernel profiles, i can try to produce them.
(In reply to comment #2) > what may be possible is perhaps to reduce the performance impact if > someone can give me kernel profiles Would oprofile do the trick?
(In reply to comment #4) > Would oprofile do the trick? that or these days there's perf as well (http://perf.wiki.kernel.org/index.php/Main_Page). basically what i'm interested most is the called function profile to see if there's anything in particular that stands out (as compared to the !PAX case), and then if possible, which part of the culprit function is particularly slow. my guess is that it'll be some page table manipulation code as that's where the per-CPU PGD changes have impact, but i need to see which exact caller/user is affected in KVM.
(In reply to comment #5) > (In reply to comment #4) > > Would oprofile do the trick? > > that or these days there's perf as well Here's the top of perf's report. I just booted, logged in and shut down a kvm guest under a hardened host with both UNDEREF and KERNEXEC # Events: 251K cycles # # Overhead Command Shared Object Symbol # ........ ............... .................... ...... # 41.82% qemu-system-x86 [kernel.kallsyms] [k] page_fault 13.01% qemu-system-x86 [kernel.kallsyms] [k] copy_user_generic_string 12.88% qemu-system-x86 [kernel.kallsyms] [k] vmx_vcpu_run 3.56% qemu-system-x86 [kernel.kallsyms] [k] handle_mm_fault 2.64% qemu-system-x86 [kernel.kallsyms] [k] __vmx_load_host_state 2.43% qemu-system-x86 [kernel.kallsyms] [k] do_page_fault 1.86% qemu-system-x86 [kernel.kallsyms] [k] kvm_arch_vcpu_ioctl_run 1.85% qemu-system-x86 [kernel.kallsyms] [k] _raw_spin_lock 1.77% qemu-system-x86 [kernel.kallsyms] [k] up_read 1.33% qemu-system-x86 qemu-system-x86_64 [.] 0x00000000075741 1.24% qemu-system-x86 [kernel.kallsyms] [k] down_read_trylock 1.16% qemu-system-x86 [kernel.kallsyms] [k] vmx_flush_tlb 1.14% qemu-system-x86 [kernel.kallsyms] [k] error_entry 0.69% qemu-system-x86 [kernel.kallsyms] [k] find_highest_vector 0.66% qemu-system-x86 [kernel.kallsyms] [k] vmcs_writel 0.58% qemu-system-x86 [kernel.kallsyms] [k] pax_enter_kernel 0.57% qemu-system-x86 libc-2.11.2.so [.] ioctl 0.55% qemu-system-x86 [kernel.kallsyms] [k] gfn_to_hva 0.52% qemu-system-x86 [kernel.kallsyms] [k] error_exit 0.46% qemu-system-x86 [kernel.kallsyms] [k] mark_page_dirty
(In reply to comment #6) > Here's the top of perf's report. I just booted, logged in and shut down a kvm > guest under a hardened host with both UNDEREF and KERNEXEC > > # Events: 251K cycles > # > # Overhead Command Shared Object Symbol > # ........ ............... .................... ...... > # > 41.82% qemu-system-x86 [kernel.kallsyms] [k] page_fault ok, i guess it's the per-CPU PGD stuff playing with the page tables. is it possible to get per-insn coverage from perf? also, can you test a normal kernel or at least without KERNEXEC/UDEREF please?
i forgot to ask, but what kernel is running in the guest itself?
The host is running gentoo's h-s-2.6.34-r2 = vanilla 2.6.34.4 + grsecurity-2.2.0-2.6.34.4-201008131840.patch The guest is also running hardened sources, but h-s-2.6.32-r9 = vanilla 2.6.32.15 + grsecurity-2.1.14-2.6.32.15-20100601150.patch What I'll do is produce systematic perf stats for all the 4 possibilities of vanilla vs hardened on guest vs host of the *latest* kernels rather than mixing as above so we have a clearer comparison.
(In reply to comment #9) > What I'll do is produce systematic perf stats for all the 4 possibilities of > vanilla vs hardened on guest vs host of the *latest* kernels rather than mixing > as above so we have a clearer comparison. ok, i was going to suggest something like that next, but what you should test is not hardened vs. vanilla but more like hardened with or without KERNEXEC/UDEREF as i'm pretty sure it's them that introduce the performance impact. now to simplify your life you can just create a single hardened kernel image with only UDEREF enabled (i.e., KERNEXEC disabled) and boot it with or without pax_nouderef in the host/guest.
This is the same host kernel, but with KERNEXEC disabled. UNDEREF is still enabled. The guest booted noticeably faster. # Events: 29K cycles # # Overhead Command Shared Object Symbol # ........ ............... .................... ...... # 58.79% qemu-system-x86 [kernel.kallsyms] [k] vmx_vcpu_run 5.21% qemu-system-x86 [kernel.kallsyms] [k] kvm_arch_vcpu_ioctl_run 4.77% qemu-system-x86 [kernel.kallsyms] [k] vmx_flush_tlb 2.21% qemu-system-x86 qemu-system-x86_64 [.] 0x000000000a7684 1.74% qemu-system-x86 [kernel.kallsyms] [k] copy_user_generic_string 1.70% qemu-system-x86 [kernel.kallsyms] [k] vmcs_writel 1.68% qemu-system-x86 [kernel.kallsyms] [k] find_highest_vector 1.13% qemu-system-x86 [kernel.kallsyms] [k] gfn_to_hva 1.12% qemu-system-x86 [kernel.kallsyms] [k] _raw_spin_lock 1.00% qemu-system-x86 [kernel.kallsyms] [k] kvm_find_cpuid_entry 0.95% qemu-system-x86 [kernel.kallsyms] [k] __vmx_load_host_state 0.81% qemu-system-x86 [kernel.kallsyms] [k] vmx_save_host_state 0.80% qemu-system-x86 [kernel.kallsyms] [k] mark_page_dirty 0.74% qemu-system-x86 [kernel.kallsyms] [k] __srcu_read_lock 0.68% qemu-system-x86 [kernel.kallsyms] [k] x86_decode_insn 0.62% qemu-system-x86 [kernel.kallsyms] [k] check_object_size 0.56% qemu-system-x86 [kernel.kallsyms] [k] mmu_free_roots 0.47% qemu-system-x86 [kernel.kallsyms] [k] vmx_set_cr0
(In reply to comment #11) > This is the same host kernel, but with KERNEXEC disabled. UNDEREF is still > enabled. The guest booted noticeably faster. Correction, that one actually had UDEREF *not* enabled.
Okay, I tested all 4 possibilities on the host. 1) KERNEXEC=n UDEREF=n 2) KERNEXEC=y UDEREF=n 3) KERNEXEC=n UDEREF=y 4) KERNEXEC=y UDEREF=y While keeping KERNEXEC=y UDEREF=y on the client. Host kernel is h-s-2.6.34-r2 = vanilla 2.6.34.4 + grsecurity-2.2.0-2.6.34.4-201008131840.patch Client kernel is same. I've pretty much confirmed Tommy[D]'s conclusion that both have to be off on the host to avoid the bug. Also, #1 is the only configuration that has CONFIG_PAX_PER_CPU_PGD not set. I have not found a way of getting per-insn from perf.
(In reply to comment #1) > > ..I'm willing to bet it affects VMWare as well.. > Confirmed - There is roughly a magnitude increase in compilation times for any packages when UDEREF and KERNEXEC are enabled inside a virtual machine running under VMware vSphere 4.1U1.
I have not seen this effect vmware, I am running with KERNEXEC and UDEREF enabled on esxi 4.0 on intel nahalem hosts. MattM, I seem to remember testing with you before. I'd like to run some more tests with you to compare later.
(In reply to comment #15) > I have not seen this effect vmware, I am running with KERNEXEC and UDEREF > enabled on esxi 4.0 on intel nahalem hosts. MattM, I seem to remember testing > with you before. I'd like to run some more tests with you to compare later. Hardened guest or hardened host? A hardened guest in vmware with KERNEXEC and/or UDEREF enabled should be fine.
Yes, guest. I have not been able to get hardened and vmware workstation to work. I documented my progress here. http://git.overlays.gentoo.org/gitweb/?p=proj/hardened-docs.git;a=blob_plain;f=html/hardened-virtualization.html;hb=f3d78ad276f953bb4305f700d7ba4f15c422275b
genmaster portage # grep UDEREF /boot/config-2.6.36-hardened-r9c # CONFIG_PAX_MEMORY_UDEREF is not set Yep - we disabled UDEREF on our vmware esx 4.1U1 virtual machines running the kernel version above because of the substantial impact on performance when enabled. A simple recompile of kernel w/ it disabled restored performance to normal. Note that the exact kernel configuration we are using is already uploaded and stored in version control on github: https://github.com/deploylinux/Gentoo-VM-Kernel-Configs-Plus-Utiliities-for-VMware-vSphere-Cluster As for qemu-kvm, we're running our new kernel config on it w/o any issues. I haven't verified that enabling UDEREF on qemu-kvm generates the same slow downs as under vmware, but I wouldn't be surprised. Actually, if you read the kernel config notes for UDEREF it pretty much warns about the possible performance impact on virtualized guests. Unfortunate, but I'm not really sure how this can be handled outside of upstream kernel devs. Note that all our testing has been performed w/ 2.6.36.
(In reply to comment #18) > genmaster portage # grep UDEREF /boot/config-2.6.36-hardened-r9c > # CONFIG_PAX_MEMORY_UDEREF is not set you could have just booted with pax_nouderef ;). > As for qemu-kvm, we're running our new kernel config on it w/o any issues. I > haven't verified that enabling UDEREF on qemu-kvm generates the same slow downs > as under vmware, but I wouldn't be surprised. it does, so does KERNEXEC/amd64. > Actually, if you read the kernel config notes for UDEREF it pretty much warns > about the possible performance impact on virtualized guests. well, some impact is expected but not as much as people are seeing on EPT enabled CPUs so i still think there's some bad interaction between kvm and this per-CPU PGD feature that PaX relies on. > Unfortunate, but I'm not really sure how this can be handled outside of > upstream kernel devs. yes, some kvm guru's help would be needed to resolve this i think...
I wish to confirm that this is still an issue. I will be without a test machine for a couple of more weeks. I'm pretty sure that this has been fixed though.
(In reply to comment #20) > I wish to confirm that this is still an issue. "still" = what kernel and what options? Can you give me your kernel config for both host and guest. I'd like to confirm at my end.
I believe that it was fixed back in 2.6.39. I have not been able to confirm though. On intel, enable uderef and keep kernexec disabled. I expect the same on AMD. I'll attach a guest config (3.0.4-r1) to the bug.
Created attachment 292639 [details] kvm guest config
(In reply to comment #20) > I wish to confirm that this is still an issue. I will be without a test > machine for a couple of more weeks. I'm pretty sure that this has been fixed > though. i can only confirm that i certainly haven't made any conscious fix since as i still don't know the root cause.
The issue still exists, though is better with a host that supports nested pages. Uderef enabled When the host caches the HDD my speed in doing a dd from within the guest is about 2MBps. When the host does not cache the HDD the dd speed within the guest is about 20MBps. Uderef disabled When the host caches the HDD my speed in doing a dd from within the guest is about 340MBps. When the host does not cache the HDD the dd speed within the guest is about 40MBps. As is already known this confirms that the issue is in copying mem not just the HDD.
this one should be fixed now, can you please confirm/close?
Confirmed fixed for me, can someone else (preferably with an old amd chip or something) confirm this works for them as well?
Closing as it is fixed in hardened-sources-3.5.4
