When running kernel 2.6.32-r7 (amd64) under KVM 0.12.4 (installed as part of proxmox virtual environment - http://pve.proxmox.com) there appears to be a severe memory leak condition in kernel mode... After running for a short time, especially if the system is under load (such as compilation) This was reproduced on a HP DL140 G3 running proxmox pve 1.5 with the latest updates. I am unsure how to debug kernel memory allocation, if anyone can provide details how to debug this i will post appropriate output. Reproducible: Always Steps to Reproduce: 1. Install a gentoo amd64 image under kvm or proxmox using virtio disk/net, 512mb ram, using the attached kernel config 2. attempt to emerge mysql under the booted kernel Actual Results: System will consume an increasing amount of memory, aparrently in kernel mode... After the OOM killer has killed all the userland processes, the memory usage will still be high. Expected Results: Mysql should compile successfully, and the memory used by the compile should be freed again after completion.
Created attachment 235557 [details] Configuration file for 2.6.32-hardened-r7 Example configuration file that causes memory leak issue
Kernel 2.6.32-hardened-r6 (from hardened-development overlay) does work correctly with the same configuration.
Output from a test system.. "free" showing memory usage, ps aux showing that not a huge amount is running (certainly not enough to justify the level of memory usage) dbmail ~ # free total used free shared buffers cached Mem: 506740 501844 4896 0 80 1676 -/+ buffers/cache: 500088 6652 Swap: 1048568 1928 1046640 dbmail ~ # w 11:21:21 up 35 min, 1 user, load average: 0.32, 1.31, 1.18 USER TTY LOGIN@ IDLE JCPU PCPU WHAT root ttyS0 11:21 0.00s 0.20s 0.02s w dbmail ~ # ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 3984 8 ? Ss 10:45 0:00 init [3] root 2 0.0 0.0 0 0 ? S 10:45 0:00 [kthreadd] root 3 0.0 0.0 0 0 ? S 10:45 0:00 [migration/0] root 4 0.0 0.0 0 0 ? S 10:45 0:00 [ksoftirqd/0] root 5 0.0 0.0 0 0 ? S 10:45 0:00 [events/0] root 6 0.0 0.0 0 0 ? S 10:45 0:00 [khelper] root 16 0.0 0.0 0 0 ? S 10:45 0:00 [async/mgr] root 59 0.0 0.0 0 0 ? S 10:45 0:00 [sync_supers] root 61 0.0 0.0 0 0 ? S 10:45 0:00 [bdi-default] root 63 0.0 0.0 0 0 ? S 10:45 0:00 [kblockd/0] root 64 0.0 0.0 0 0 ? S 10:45 0:00 [kacpid] root 65 0.0 0.0 0 0 ? S 10:45 0:00 [kacpi_notify] root 66 0.0 0.0 0 0 ? S 10:45 0:00 [kacpi_hotplug] root 136 0.0 0.0 0 0 ? S 10:45 0:00 [kseriod] root 178 0.0 0.0 0 0 ? D 10:45 0:01 [kswapd0] root 179 0.0 0.0 0 0 ? S 10:45 0:00 [aio/0] root 180 0.0 0.0 0 0 ? S 10:45 0:00 [crypto/0] root 814 0.0 0.0 0 0 ? S 10:45 0:00 [jbd2/vda1-8] root 815 0.0 0.0 0 0 ? S 10:45 0:00 [ext4-dio-unwr] root 904 0.0 0.0 12600 0 ? S<s 10:45 0:00 /sbin/udevd --d root 3866 0.0 0.0 12596 0 ? S< 10:46 0:00 /sbin/udevd --d root 3921 0.0 0.0 6076 4 tty1 Ss+ 10:46 0:00 /sbin/agetty 38 root 3922 0.0 0.0 6064 4 tty2 Ss+ 10:46 0:00 /sbin/agetty 38 root 3923 0.0 0.0 6116 4 tty3 Ss+ 10:46 0:00 /sbin/agetty 38 root 3924 0.0 0.0 6096 4 tty4 Ss+ 10:46 0:00 /sbin/agetty 38 root 3925 0.0 0.0 6104 4 tty5 Ss+ 10:46 0:00 /sbin/agetty 38 root 3926 0.0 0.0 6120 4 tty6 Ss+ 10:46 0:00 /sbin/agetty 38 root 28395 0.0 0.0 56192 4 ttyS0 Ss 11:15 0:00 /bin/login -- root 28440 0.0 0.0 0 0 ? S 11:21 0:00 [flush-254:0] root 28441 0.8 0.0 17920 216 ttyS0 S 11:21 0:00 -bash root 28448 0.0 0.2 14908 1052 ttyS0 R+ 11:21 0:00 ps aux dbmail ~ # free total used free shared buffers cached Mem: 506740 501844 4896 0 76 1716 -/+ buffers/cache: 500052 6688 Swap: 1048568 1960 1046608 dbmail ~ # uname -a Linux dbmail 2.6.32-hardened-r9-katyusha #3 SMP Tue Jun 15 16:36:13 BST 2010 x86_64 QEMU Virtual CPU version 0.12.4 GenuineIntel GNU/Linux
Can the reporter please test the in tree hardened-sources-2.6.32-r9 and see if this is still a problem?
(In reply to comment #4) > Can the reporter please test the in tree hardened-sources-2.6.32-r9 and see if > this is still a problem? Still occurs on hardened-sources-2.6.32-r9 with the same configuration.
same issue here running kvm on ~amd64 other VMs is working fine besides the gentoo hardened vm (amd64-nomultilib) with virtio disk/net (maybe that could be the problem? since i vaguely remember it not happening when i used emulated pata and the e1000 driver (will test that when i have the time). host kernel: 2.6.35-gentoo-r1 VM kernel 2.6.32-hardened-r9
ok tested it it happens even when i used emulated hardware instead of virtio disk/net
Created attachment 243361 [details] emerge --info for hardened VM
I think that bug #332355 is a special instance of this one, but I'm not 100% sure. There the memory leak was triggered when using 1. kvm 2. hardened-sources-2.6.32-r9 3. virtio for hardware devices rather than emulating The workaround is either emulate the hardware (eg use -nic,model=e1000 and compile in e1000 support in the guest kernel) or use kernels later than hs-2.6.32-r9. -r9 was the last in the grsec-2.1.4 series which was also used in -r7. Beginning from -r10 on, grsec-2.2.0 was used. It seems to be working for me. Can users try -r14 and let me know.
(In reply to comment #9) > I think that bug #332355 is a special instance of this one, but I'm not 100% > sure. There the memory leak was triggered when using > > 1. kvm > 2. hardened-sources-2.6.32-r9 > 3. virtio for hardware devices rather than emulating > > The workaround is either emulate the hardware (eg use -nic,model=e1000 and > compile in e1000 support in the guest kernel) or use kernels later than > hs-2.6.32-r9. -r9 was the last in the grsec-2.1.4 series which was also used > in -r7. Beginning from -r10 on, grsec-2.2.0 was used. > > It seems to be working for me. Can users try -r14 and let me know. > again i have just disproved "3" so only 1 and 2 could be the cause ill get back when i have tested -r14
I have tested 2.6.34-hardened-r2 and 2.6.32-hardened-r14 with the same configuration under KVM and both seem to work fine so far, tested with a loop of emerge mysql commands with no problems (previously this same vm would crash during a single emerge mysql process). However, enabling the new "Prevent invalid userland pointer dereference" option present in these kernels causes a crash on bootup.
Created attachment 245177 [details] hardened kvm guest kernel config attached the kernel config for the guest that is hitting this bug (2.6.32-r9).
disabling CONFIG_PAX_KERNEXEC (by bluness`s advice) fixed it so it seems like that is the problem with this specific pax/kernel version.
can you guys try the latest .32 and .35 PaX test patches (http://www.grsecurity.net/~paxguy1/) and let me know if you still see the memory leak? if you do, can you post /proc/slabinfo in the hope that the leaking structure will show up there?
(In reply to comment #14) > can you guys try the latest .32 and .35 PaX test patches > (http://www.grsecurity.net/~paxguy1/) and let me know if you still see the > memory leak? if you do, can you post /proc/slabinfo in the hope that the > leaking structure will show up there? > Tested and neither show the leak. slabtop shows very little change as the vm boots. Again, I'll close this as soon as we have stabilization of kernels which do not exhibit the leak. Thanks pipacs :)
Just stabilized hardened-sources-2.6.32-r31.ebuild and hardened-sources-2.6.36-r6.ebuild which include the fix. Closing.
