After having switched the hole system to ~x86 for a bunch of very severe and spurious mouse related problems (pointer movement was constrained vertically), running "valgrind /usr/ls" triggered a kernel oops in mm/mmap.c:1730. The EIP is related to "<pax_find_mirror_vma+97/a7>" (see attachment). Valgrind did not run too with linux-2.6.25-hardened-r8, but then I did not investigate the circumstances. Valgrind and all the system was compiled by "gcc (GCC) 4.2.4 (Gentoo Hardened 4.2.4-r1 p1.0, builtin ssp,fortify, pie-9.0.11)", which in turn was obtained from "https://hardened.gentooexperimental.org/trac/secure/" Reproducible: Always Steps to Reproduce: 1. run valgrind /bin/ls (or with any other program) 2. look at dmesg output Actual Results: Kernel oops Expected Results: say ... valgrind may crash, this is no surprise with pax enabled
Created attachment 171630 [details] ksymoops transformation It's certainly somewhat contradictory to setup pax and grsecurity, and then make kernel addresses publicly available ...
Created attachment 171631 [details] Kernel configuration as obtained from /proc/config.gz
(In reply to comment #1) > Created an attachment (id=171630) [edit] > ksymoops transformation i'll need your mm/mmap.o file as well to see which check triggered. fwiw, i couldn't trigger it here on the same setup. which valgrind version was this? also can you try to compile your kernel with a normal gentoo gcc version? > It's certainly somewhat contradictory to setup pax and grsecurity, and then > make kernel addresses publicly available ... if it bothers you, you can always directly email me or spender ;). also for bugreports we prefer the actual kernel messages with enabled KALLSYMS, then you won't have to bother with ksymoops.
(In reply to comment #3) > i'll need your mm/mmap.o file as well to see which check triggered. I'll send you the mm/mmap.o file and also dmesg output (with KALLSYMS) via email (to pageexec@freemail.hu) > couldn't trigger it here on the same setup. which valgrind version was this? dev-util/valgrind-3.3.1 > also can you try to compile your kernel with a normal gentoo gcc version? Exactly which one is normal? Previously, I used gcc-3.4.6-r2 to compile vtk demo programs, but then I observed (next to other, probably related problems) that this version has issues keeping track of local variables properly, inspite of being the only one officially supported.
(In reply to comment #4) > > also can you try to compile your kernel with a normal gentoo gcc version? > > Exactly which one is normal? anything non-hardened. ssp used to produce incorrect code, so i'd like to make sure we're looking at something sane first ;).
Can you try to use the gcc 3.4.6 version for hardened and see if it still error out?
(In reply to comment #6) > Can you try to use the gcc 3.4.6 version for hardened and see if it still error > out? > Noo need for that sorry.
Current status: - Recompiling sys-kernel/hardened-sources-2.6.27-r1 by a non-hardened gcc-4.3.2 does not change the situation (mmbug persists). Due to this, getting a post mortem dump is impossible, as is inspecting the situation using gdb-6.8 directly. - Recompiling valgrind by non-hardened gcc-4.3.2 does change the situation: no mmbug any more, whereas - "valgrind /bin/ls" still gets aborted by pax for some reasons, most probably due to a destroyed stack. PAX: execution attempt in: <anonymous mapping>, 2fec3000-31ff8000 2fec3000 PAX: terminating task: /usr/lib/valgrind/x86-linux/memcheck(memcheck):5152, uid/euid: 0/0, PC: 2fec3000, SP: 2fec0ed0 PAX: bytes at PC: 8b 9d 50 01 00 00 8b 75 10 89 9d 40 01 00 00 89 75 00 c7 45 PAX: bytes at SP-4: gdb --core=./core /usr/lib/valgrind/x86-linux/memcheck Core was generated by `valgrind /bin/ls'. Program terminated with signal 9, Killed. [New process 5152] #0 0x2fec3000 in ?? () (gdb) bt #0 0x2fec3000 in ?? () #1 0x2fec0f2c in ?? () #2 0x00000001 in ?? () #3 0x00000025 in ?? () #4 0x3873ee5c in vgPlain_threads () #5 0x2fec0f20 in ?? () #6 0x04000980 in ?? () #7 0x380344c6 in run_thread_for_a_while (tid=0) at m_scheduler/scheduler.c:636 #8 0x00000000 in ?? ()
(In reply to comment #8) > - Recompiling valgrind by non-hardened gcc-4.3.2 does change the situation: no > mmbug any more, whereas this only masks the problem, apparently it's something to do with valgrind being compiled as PIE (that the hardened toolchain does by default). > - "valgrind /bin/ls" still gets aborted by pax for some reasons, most > probably due to a destroyed stack. no, it's more about valgrind needing to generate code at runtime, so all its binaries will need MPROTECT removed (paxctl -m), i thought the ebuild did that already, but apparently not... that's a bug on its own.
(In reply to comment #9) > this only masks the problem, apparently it's something to do with valgrind > being compiled as PIE (that the hardened toolchain does by default). I did elaborate on this phenomen to shed some light to the circumstances making the mmap code fail. > > - "valgrind /bin/ls" still gets aborted by pax for some reasons, most > > probably due to a destroyed stack. > > no, it's more about valgrind needing to generate code at runtime, so all its > binaries will need MPROTECT removed (paxctl -m), i thought the ebuild > did that already, but apparently not... that's a bug on its own. Actually, it's not only valgrind, but all valgrind programs located in "/usr/lib/valgrind/x86-linux", which do need this flag. Thus "scanelf /usr/lib/valgrind/x86-linux | grep ET_EXEC | cut -d" " -f2 | xargs sudo paxctl -m" would fix the valgrind related problem, but only if valgrind was compiled with non-hardened gcc-4.3.2 (else it still get's aborted, and still triggers "kernel BUG at mm/mmap.c:1730")
According to one of the PaX authors, the following patch would fix the issue: --- linux-2.6.27.7-pax/mm/mmap.c 2008-10-10 09:20:24.000000000 +0200 +++ linux-2.6.27.7-pax/mm/mmap.c 2008-11-25 00:11:17.000000000 +0100 @@ -1721,7 +1721,7 @@ BUG_ON(vma->vm_mirror); return NULL; } - BUG_ON(vma->vm_end - SEGMEXEC_TASK_SIZE - 1 < vma->vm_start - SEGMEXEC_TASK_SIZE - 1); + BUG_ON(vma->vm_start < SEGMEXEC_TASK_SIZE && SEGMEXEC_TASK_SIZE < vma->vm_end); vma_m = vma->vm_mirror; BUG_ON(!vma_m || vma_m->vm_mirror != vma); BUG_ON(vma->vm_file != vma_m->vm_file); The changed line maps exactly to mm/mmap.c +1730 of my local source tree. Having applied it, the assertion actually does not trigger a kernel bug message any more. Also core dumps work again, since no kernel bug is triggered. Valgrind is still crashing, but this is another, probably much different problem. However, looking closer at the changed line, stripping off the part of the expression being equal on both sides of the inequation, the triggering condition essentially was: BUG_ON( vma->vm_end < vma->vm_start ); which appears to be a very sane condition to trigger an assertion. Since the new condition doesn't trigger anymore within an else unchanged environment, I would suspect an underlying, related bug to be still there, which could be, for example, vma->vm_end == 0 or something.
(In reply to comment #11) > Valgrind is still crashing, but this is another, probably much different > problem. does valgrind actually crash or does it get killed by PaX? if it's the latter, it's the lack of paxctl -m markings, otherwise it's worth another investigation (in a new bug). > However, looking closer at the changed line, stripping off the part of the > expression being equal on both sides of the inequation, the triggering > condition essentially was: > > BUG_ON( vma->vm_end < vma->vm_start ); > > which appears to be a very sane condition to trigger an assertion. Since the > new condition doesn't trigger anymore within an else unchanged environment, I > would suspect an underlying, related bug to be still there, which could be, for > example, vma->vm_end == 0 or something. nope, the original line and what you simplified it to are *not* equivalent, there's a trick in there ;). consider what happens for vm_start = SEGMEXEC_TASK_SIZE and vm_end=SEGMEXEC_TASK_SIZE+0x1000. hint: they're all 32 bit unsigned entities and can happily (and in this case, intentionally) wrap around. this is an old trick to save on one comparision when checking that a value is within a range (here i wanted to ensure that the vma doesn't cross SEGMEXEC_TASK_SIZE).
(In reply to comment #12) > > Valgrind is still crashing, but this is another, probably much different > > problem. > > does valgrind actually crash or does it get killed by PaX? if it's the latter, > it's the lack of paxctl -m markings, otherwise it's worth another >investigation> (in a new bug). No, this is another problem. While the non-PIE version compiled by gcc-4.3.2 is running fine, the PIE version compiled by gcc-4.2.4-r1 gets killed by signal 11 if MPROTECT is active. But after paxctl -m had been applied to valgrind and it's deputees in /usr/lib/valgrind/x86-linux, the message is this: $ valgrind /bin/ls valgrind: mmap(0x0, 102400) failed in UME with error 22 (Invalid argument). valgrind: this can be caused by executables with very large text, data or bss segments
(In reply to comment #13) > But after paxctl -m had been applied to valgrind and > it's deputees in /usr/lib/valgrind/x86-linux, the message is this: > > $ valgrind /bin/ls > valgrind: mmap(0x0, 102400) failed in UME with error 22 (Invalid argument). > valgrind: this can be caused by executables with very large text, data or bss > segments if you strace the above command we'll see what that failing mmmap looks like. even then it may be some design issue in valgrind that makes it misbehave when compiled as a PIE, probably it's a question that upstream should be asked about...
(In reply to comment #14) > if you strace the above command we'll see what that failing mmmap looks like. > even then it may be some design issue in valgrind that makes it misbehave when > compiled as a PIE, probably it's a question that upstream should be asked > about... strace -f did not reveal any failing mmap call. Therefore I enabled a few debug statement in coregrind/m_ume.c/mapelf(). It turned out, that VG_(am_mmap_file_fixed_clientAddr) fails, in turn probably because VG_(am_get_advisory) fails. This resulted in the misleading error message, that mmap would have failed, while actually a mmap related syscall was never made for the section in question. So this really looks like a valgrind issue.
(In reply to comment #15) > strace -f did not reveal any failing mmap call. Therefore I enabled a few debug > statement in coregrind/m_ume.c/mapelf(). It turned out, that > VG_(am_mmap_file_fixed_clientAddr) fails, in turn probably because > VG_(am_get_advisory) fails. This resulted in the misleading error message, that > mmap would have failed, while actually a mmap related syscall was never made > for the section in question. So this really looks like a valgrind issue. can you print out the parameters of the failing am_mmap_file_fixed_client call? my guess is that valgrind tries to mmap something (target exe or itself?) at an address (probably 0) which is already occupied by something else, hence the failure. in fact i'm almost sure that's what happens as i don't see any ET_EXEC/ET_DYN differentiation in mapelf(). to give this theory a quick test, you could prelink -r 0x04000000 <your target PIE> and see if valgrind can run it then. in any case, i think it's really an upstream issue and this bug can be closed now.
(In reply to comment #16) > can you print out the parameters of the failing am_mmap_file_fixed_client call? --18867:0:ume mmap_file_fixed_client #1 --18867:0:aspacem m_aspacemgr/aspacemgr-linux.c +2079: vgPlain_am_mmap_file_fixed_client():start=0x0,length=102400,prot=5,fd=3,offset=0 --18867:0:aspacem m_aspacemgr/aspacemgr-linux.c +2115: vgPlain_am_mmap_file_fixed_client(): returning VKI_EINVAL due to ok=0 || advised=0 != start=0 The trace stemming from source line +2115 proves that am_get_advisory flagged failure (ok=0) > my guess is that valgrind tries to mmap something (target exe or itself?) at an > address (probably 0) which is already occupied by something else, hence the > failure. in fact i'm almost sure that's what happens as i don't see any > ET_EXEC/ET_DYN differentiation in mapelf(). to give this theory a quick test, > you could > > prelink -r 0x04000000 <your target PIE> > > and see if valgrind can run it then. in any case, i think it's really an > upstream issue and this bug can be closed now. > Are you still taking processor handbooks for breakfast? Anyway, you are completely right. Prelinking the target to a different load address cleared the problem here. Previously it failed probably, because memcheck already was loaded at address 0, as one could see inspecting the logs I will attach for the sake of completeness here. Best
Created attachment 173971 [details] strace -f log of pie compiled valgrind, run with arg /bin/ls
Fixed in 2.6.25-r11 and 2.6.26-r7. Bug still in 2.6.27-r2, will fix in 2.6.27-r3 and resolve bug at that time.
hardened-sources-2.6.27-r3 is in the tree now.
