Created attachment 317834 [details] config from hardened sources causing the issue This one took a while to track down. linux-3.4.2-hardened-r1 <--problem kernel Running a hardened server (toolchain, profile, kernel) and KVM/QEMU. GRSecurity set to Virtualization. (Tried with GRSecurity and PAX disabled too) When I start qemu-system-x86_64 even with a minimalistic command line like so: "/usr/bin/qemu-system-x86_64 -m 8G -S" which will allocate 8G for the VM and not start execution, qemu immediately occupies 8G of memory (RSS) as if something has initialized all of it. It is eventually merged by KSM but that's it not the point. It is supposed to allocate but not actually occupy memory until it is touched for the first time. This makes a bad scenario for over committing. I have since put in gentoo-sources (left toolchain hardened as well as profile) after importing my config from the hardened kernel, recompiled and rebooted. Problem was gone. I can start a VM and it will occupy as much memory as it has actually started to use instead of all of it immediately. Maybe I am misunderstanding the way the hardened-sources work (maybe this is a feature not a bug?), but this is not expected behavior to me. I have attached the config from the hardened sources in case I'm just stupid and someone can point out the option that is enabled that did this to me. I'd like to be able to run a hardened kernel, but the memory issue it more of a concern to me than the extra security. Also, if you think that using hardening on a KVM host system is stupid, please let me know, and why. Thanks
Don't CC arch teams on your own
my bad, meant to just add it to the architecture, haven't used this system much yet
> It is supposed to > allocate but not actually occupy memory until it is touched for the first > time. This makes a bad scenario for over committing. Correct. This is an issue. > I have since put in gentoo-sources (left toolchain hardened as well as > profile) after importing my config from the hardened kernel, recompiled and > rebooted. Problem was gone. That's the first thing I aways ask. The second is, can you bracket which hardened kernel *introduces* or *relieves* the problem. So can you try hardened-sources-3.4.4-r2 which I just added to the tree. Note that it introduces a very different menu system than 3.4.2-hardened-r1 and you should go through it again. Pay particular attention to the following settings under [*] Grsecurity: Usage Type (Server) ---> Virtualization Type (Host) ---> Virtualization Hardware (EPT/RVI Processor Support) ---> Virtualization Software (KVM) ---> > Maybe I am misunderstanding the way the hardened-sources work (maybe this is > a feature not a bug?), but this is not expected behavior to me. If it is a feature, then its a bad feature. Overcommit is an important part of virtualization. Now, it may be the case that we can't get overcommit + hardening to work, but that's not the same as whether we want it or not. > Also, if you think that using > hardening on a KVM host system is stupid, please let me know, and why. Not at all. We've been working to get this right.
I can confirm that this is happening in 3.4.4. (booting just to bios). Don't see either kernexec or uderef (can cause oddness), so that's good.
Found something interesting: with problem alleviated by runnung gentoo-sources instead of hardened on host, I noticed something else, a guest I have running hardened-sources-3.4.2-r1 is exhibiting the same problem note: this is only for that guest; other guests (non-hardened) on the host are behaving as expected and to clarify, the problem is that this guest commits all 8G of memory allotted to it as it boots as if the kernel is touching all memory when it starts I am going to try the new kernel suggested here (hardened-sources-3.4.4-r2) on host and guest and report back
Created attachment 317946 [details] Config from host hardened-sources-3.4.4-r2 on host: issue seems to persist in hardened-sources-3.4.4-r2 set to suggested settings with no modification (HOST, KVM, EPT/RVI Processor Support) VM immediately takes up all memory allotted to it testing on guest.....
try turning memory sanitization off in pax settings
SANITIZE (in the guest) would touch all memory assigned to the guest on boot (the kernel's early init code where the kernel transfers ownership from the boot allocator to the main allocator is simulated by freeing said pages to main allocator and that invokes the sanitize logic). see some interesting numbers here: http://staff.aist.go.jp/k.suzaki/EuroSec12-SUZAKI-revised2.pdf now why you observed the same with SANITIZE disabled is another question, i have no idea about that.
on Guest: All seems well after putting hardened-sources-3.4.4-r2 in and disabling CONFIG_PAX_MEMORY_SANITIZE I'm going to try again in the hardened-sources-3.4.2-r1 as those are the currently marked "stable" sources. The update might not have been necessary had I known about CONFIG_PAX_MEMORY_SANITIZE before trying the new kernel.... That being said, my above post was confirming that on the *HOST* system, the update to hardened-sources-3.4.4-r2 did not fix the issue.....two separate issues here with very similar effects: hardened-sources-3.4.4-r2 on *HOST*: all allotted memory is committed *PRIOR* to the guest even starting execution qemu-system-x86_64 -m 8G -S this command took up 8G of memory immediately This issue, the originally noticed and posted issue, *still remains* hardened-sources on the *GUEST*: if CONFIG_PAX_MEMORY_SANITIZE is set, then all allotted memory is committed *AFTER* the guest starts execution, this is fixed by disabling CONFIG_PAX_MEMORY_SANITIZE. I also confirmed that on the hardened-sources-3.4.2-r1, after disabling CONFIG_PAX_MEMORY_SANITIZE, the *GUEST* does not cause the memory issue so, I guess we still need to figure out what is causing this on the host kernel
> hardened-sources-3.4.4-r2 on *HOST*: all allotted memory is committed > *PRIOR* to the guest even starting execution > qemu-system-x86_64 -m 8G -S > this command took up 8G of memory immediately > This issue, the originally noticed and posted issue, *still remains* > I assume this is irrespective of whether or not you are using a hardened guest or a vanilla guest. Correct? ie. hardened HOST + vanilla GUEST = 8G mem allocated immediately on startup. > hardened-sources on the *GUEST*: if CONFIG_PAX_MEMORY_SANITIZE is set, then > all allotted memory is committed *AFTER* the guest starts execution, this is > fixed by disabling CONFIG_PAX_MEMORY_SANITIZE. I assume this is for a vanille HOST only, given the the previous problem would eclipse this one. Correct?
(In reply to comment #10) > > > hardened-sources-3.4.4-r2 on *HOST*: all allotted memory is committed > > *PRIOR* to the guest even starting execution > > qemu-system-x86_64 -m 8G -S > > this command took up 8G of memory immediately > > This issue, the originally noticed and posted issue, *still remains* > > > > I assume this is irrespective of whether or not you are using a hardened > guest or a vanilla guest. Correct? ie. hardened HOST + vanilla GUEST = 8G > mem allocated immediately on startup. That is correct. The memory is allocated before the guest even starts running. > > hardened-sources on the *GUEST*: if CONFIG_PAX_MEMORY_SANITIZE is set, then > > all allotted memory is committed *AFTER* the guest starts execution, this is > > fixed by disabling CONFIG_PAX_MEMORY_SANITIZE. > > I assume this is for a vanille HOST only, given the the previous problem > would eclipse this one. Correct? This is also correct. Didn't notice the second issue until putting in a vanilla kernel to bypas the first issue.
what's the value of /proc/sys/vm/overcommit_memory under the working/failing kernels?
Created attachment 317966 [details] kernel.config This also occurs if the guest kernel is hardened but has both grsec and pax disabled.
(In reply to comment #12) > what's the value of /proc/sys/vm/overcommit_memory under the working/failing > kernels? 0 in both cases on host 0 in both cases on guest (In reply to comment #13) > Created attachment 317966 [details] > kernel.config > > This also occurs if the guest kernel is hardened but has both grsec and pax > disabled. huh.....thats funny. Wouldn't disabling pax disable CONFIG_PAX_MEMORY_SANITIZE? I had found that hardened guest kernel works with CONFIG_PAX_MEMORY_SANITIZE disabled. I found that the host kernel, however, has the issue with both pax and grsec disabled.
just to clarify things the issue occurs with a hardened host with nopax and nogrsec enabled with ANY guest. This leads me to believe that pax constifications or something is the cause. any thoughts pipacs?
(In reply to comment #15) > This leads me to believe that pax constifications or something is the cause. > > any thoughts pipacs? not a lot, i still can't reproduce this here with most of PaX enabled. e.g., when i use -m 14G, the kernel only uses about 700M rss when it gets to init. maybe it's a difference in the qemu version you guys are using? i'm sort of stuck with app-emulation/qemu-kvm-0.15.1-r1, so maybe you should try that too and also give me an strace -f output for your qemu.
Steven, do you have KSM and/or TRANSPARENT_HUGEPAGE enabled?
I have both enabled and reproduced it with last nights patch vanilla + grsecurity-2.9.1-3.5.4-201209171824.patch and qemu-kvm-1.1.1-r1
never mind, i figured it out, i'll fix this in the next patch.
(In reply to comment #17) > Steven, do you have KSM and/or TRANSPARENT_HUGEPAGE enabled? though this was already confirmed, yes, I have both enabled (In reply to comment #19) > never mind, i figured it out, i'll fix this in the next patch. fantastic, do tell?
(In reply to comment #20) > (In reply to comment #19) > > never mind, i figured it out, i'll fix this in the next patch. > > fantastic, do tell? it's a PaX feature that interfered with kvm. basically, when qemu (userland) tells the kernel about the simulated guest memory mapping, kvm (kernel) checks that the given range does actually fall into the userland address range (such checks are done every time the kernel accesses userland, it's the access_ok macro). in PaX this check is modified to also pre-fault all the pages in the range to reduce the effectiveness of a certain race-based exploit technique. you can guess that pre-faulting the GBs of guest memory didn't do too well on the performance/memory consumption side ;). fortunately i kept the original non pre-faulting version around for such circumstances, so the fix was to simply convert kvm to use that.
bug can be closed when grsecurity-2.9.1-3.5.4-201209192118.patch is being used (3.5.4-r1 please :D).
(In reply to comment #22) > bug can be closed when grsecurity-2.9.1-3.5.4-201209192118.patch is being > used (3.5.4-r1 please :D). Its in the tree. Please test the full patchset, not just the vanilla + grsec patch ... just in case. Reopen if there's still a problem. Nice work pipacs.
