| Summary: | PaX: Max. per-task virtual memory too small for llvm asan and gcc-4.8 asan | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Klaus Kusche <klaus.kusche> |
| Component: | Hardened | Assignee: | The Gentoo Linux Hardened Team <hardened> |
| Status: | IN_PROGRESS --- | ||
| Severity: | normal | CC: | alexander, atoth, gentoo, gentoobugs, jaak, jj, pageexec, quentin, wbrana |
| Priority: | Normal | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| URL: | https://code.google.com/p/address-sanitizer/issues/detail?id=228&colspec=ID%20Type%20Status%20Priority%20OpSys%20Owner%20Summary | ||
| See Also: |
https://bugs.gentoo.org/show_bug.cgi?id=504200 https://bugs.gentoo.org/show_bug.cgi?id=526654 https://bugs.gentoo.org/show_bug.cgi?id=678956 |
||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Bug Depends on: | |||
| Bug Blocks: | 461954 | ||
| Attachments: | asan testsuite sum.log | ||
There is a program requiring more than 4 Terabytes of memory? ORLY? My fault it is just a bit more than 2TiB, but really, does somebody actually need that much? I think it's ASAN that should be fixed and not the kernel, the reason why the TASK_SIZE_MAX_SHIFT are what they are is mainly that this is needed in order for some of the hardening features (like the PER CPU PGD or RANDMMAP) to work properly, thus the 46 that works on your computer may not work on other setups. Pipacs, I'm CCing you since you probably can explain better why the values are what they currently are. (In reply to comment #1) > There is a program requiring more than 4 Terabytes of memory? ORLY? Yes, it does. Read their concept papers. There is a direct mapping for each byte in the "regular" address space of the process to some bytes in the asan shadow status tables. However, it actually uses only a tiny fraction of that: Most of it is never accessed and never backed by real memory, it's just page table entries (I'm happily working with asan on 2 GB RAM machines). And I heard that asan will be integrated in gcc 4.8, too... (In reply to comment #4) > (In reply to comment #1) > > There is a program requiring more than 4 Terabytes of memory? ORLY? > > Yes, it does. Read their concept papers. > There is a direct mapping for each byte in the "regular" address space > of the process to some bytes in the asan shadow status tables. > > However, it actually uses only a tiny fraction of that: > Most of it is never accessed and never backed by real memory, > it's just page table entries > (I'm happily working with asan on 2 GB RAM machines). > > And I heard that asan will be integrated in gcc 4.8, too... So this still means ASAN is broken, since anyway a program won't be able to access more than 2^42 bytes on a hardened system anyway (actually even in the best case it would be 2⁴48 given the current hardware limitations). (In reply to comment #5) > So this still means ASAN is broken, since anyway a program won't be able to > access more than 2^42 bytes on a hardened system anyway (actually even in > the best case it would be 2⁴48 given the current hardware limitations). Max. 2^42 address space size (valid bits in addresses), Max. 2^42 bytes virtually mapped, or max. 2^42 bytes mapped to real memory? As I said, setting TASK_SIZE_MAX_SHIFT to 46 makes ASAN work fine on my system (PaX, 8 GB RAM). Besides 2^42 is 4 TB, and ASAN needs only 2 TB, but it needs them at a very specific address (to mirror the addresses used by the programm in the lower 2 TB of the address space), so at least 8 TB address space is needed to fit all the mappings at their expected addresses. (In reply to comment #4) > (In reply to comment #1) > > There is a program requiring more than 4 Terabytes of memory? ORLY? > > Yes, it does. Read their concept papers. Please link this documenation. asan has hardcoded defaults assuming the usual 47 bit linux/amd64 userland address space size but iirc it can be changed at compile with -fsanitize-address-zero-base-shadow -mllvm asan-mapping-offset-log=38 (or something like that), or alternatively -mllvm asan-short-64bit-mapping-offset=1 (i think you'll need llvm/clang/compiler-rt trunk for these to be recognized). in 3.2 you can probably get away with chagnign llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp:kDefaultShadowOffset64 to a smaller value like 38 above but i haven't read the full asan docs to know if that is sufficient or there're dependencies that also have to be changed for a smaller address space. as for why PaX/PER_CPU_PGD has to reduce address space size, it's for performance reasons as any extra bit means the double amount of pgd entries that must be copied between page tables on each context switch. the 42 bit size means 8 such entries which is exactly one cache line on amd64 CPUs. it can be increased at the expense of more performance impact on context switching and certainly the value 47 should not have caused a problem, so please open a new bug about it and post any relevant info you were able to capture. (In reply to comment #7) > (In reply to comment #4) > > (In reply to comment #1) > > > There is a program requiring more than 4 Terabytes of memory? ORLY? > > > > Yes, it does. Read their concept papers. > > Please link this documenation. A quick introduction is here: http://code.google.com/p/address-sanitizer/wiki/AddressSanitizerAlgorithm You can also check http://llvm.org/devmtg/2011-11/Serebryany_FindingRacesMemoryErrors.pdf (In reply to comment #8) > asan has hardcoded defaults assuming the usual 47 bit linux/amd64 userland > address space size but iirc it can be changed at compile with > -fsanitize-address-zero-base-shadow -mllvm asan-mapping-offset-log=38 (or > something like that), or alternatively -mllvm > asan-short-64bit-mapping-offset=1 (i think you'll need > llvm/clang/compiler-rt trunk for these to be recognized). > > in 3.2 you can probably get away with chagnign > llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp: > kDefaultShadowOffset64 to a smaller value like 38 above but i haven't read > the full asan docs to know if that is sufficient or there're dependencies > that also have to be changed for a smaller address space. I'll look into it if I find some time (mid next week at best). > as for why PaX/PER_CPU_PGD has to reduce address space size, it's for > performance reasons as any extra bit means the double amount of pgd entries > that must be copied between page tables on each context switch. the 42 bit > size means 8 such entries which is exactly one cache line on amd64 CPUs. it > can be increased at the expense of more performance impact on context > switching and certainly the value 47 should not have caused a problem, so > please open a new bug about it and post any relevant info you were able to > capture. Nothing to capture: No panic, no BUG, just instant reboot. It was in the last quarter of startup scripts, so I assumed that X was the culprit. I would have to single-step through the startup scripts to find the step which caused that, but still, I would get no information about what happened or why. (In reply to comment #8) > asan has hardcoded defaults assuming the usual 47 bit linux/amd64 userland > address space size but iirc it can be changed at compile with > -fsanitize-address-zero-base-shadow -mllvm asan-mapping-offset-log=38 (or > something like that), or alternatively -mllvm > asan-short-64bit-mapping-offset=1 (i think you'll need > llvm/clang/compiler-rt trunk for these to be recognized). > > in 3.2 you can probably get away with chagnign > llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp: > kDefaultShadowOffset64 to a smaller value like 38 above but i haven't read > the full asan docs to know if that is sufficient or there're dependencies > that also have to be changed for a smaller address space. Unfortunately, it's not that simple. The shadow memory base and size are calculated by mapping the application memory start and end addresses to shadow addresses. All the constants for the addresses and the mapping are hardcoded at the beginning of projects/compiler-rt/lib/asan/asan_mapping.h : As ASAN_FLEXIBLE_MAPPING_AND_OFFSET == 1 is false, all the options you mentioned above and also kDefaultShadowOffset64 are completely ignored. After replacing the #if ASAN_FLEXIBLE_MAPPING_AND_OFFSET == 1 by #if 1, I got a clang which recognized e.g. -mllvm -asan-mapping-offset-log=39 -mllvm -asan-mapping-scale=3 (actually, it *requires* both options and fails without them). However, that does not solve the problem. I'm PaX, but use conventional a.out's, not PIE. This means that .so's and the stack are just below 0x040000000000. This in turn means that -asan-mapping-offset-log must be at least 42 to place the shadow segment above the memory used by the application. And with 42, shadow size is reduced by a factor of 4, but still get ==5958== ERROR: Failed to allocate 0x8000001000 (549755817984) bytes at address 0x03fffffff000 (12) (of course, because memory ends at 0x040000000000). The correct memory layout would be to put the shadow *between* low and high application memory, but this fails for any value of -asan-mapping-offset-log. The reason is that the high memory address end is hardcoded for 47 bit addresses, which always gives you a shadow segment which is way too large to fit between low and high application memory and collides with the .so's and stack. So the next attempt was to set kHighMemEnd = 0x000003ffffffffffUL in projects/compiler-rt/lib/asan/asan_mapping.h . Using a clang with that, when I compile an application with -mllvm -asan-mapping-offset-log=40 -mllvm -asan-mapping-scale=3 this results in * an executable which can be started without allocation errors * detects memory violations as it should * seems to have a reasonable memory and shadow layout (I'm no asan specialist) Could someone CC this bug to the llvm developers? I think the PaX kernel should stay at its default 42 bit user addresses, abd llvm should be patched as described in my previous comment (there is a pax_kernel USE flag which could be used to distinguish between PaX and non-PaX systems at compile time). Created attachment 341586 [details]
asan testsuite sum.log
make check-gcc RUNTESTFLAGS="asan.exp"
The sum log of that.
The gcc 4.8 have the same asan probblems as llvm have. FAIL: c-c++-common/asan/use-after-free-1.c -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects output pattern test, is ==18144== ERROR: AddressSanitizer failed to allocate 0x20000001000 (2199023259648) bytes at address 0x0ffffffff000 (12) ==18144== ReserveShadowMemoryRange failed while trying to map 0x20000001000 bytes. Perhaps you're using ulimit -v (In reply to comment #8) > and certainly the value 47 should not have caused a problem actually that's only half true ;), if you use UDEREF then the max value is 46 only since there must be space for the shadow pgds (with 47 the first context switch to userland will overwrite the kernel pgds). if you disable UDEREF then 47 will work with the already mentioned caveat. as for original problem, i'm afraid it's up to the clang/gcc guys to accomodate special address space layouts, in the meantime gentoo has to do the patching itself... (In reply to comment #15) > (In reply to comment #8) > > and certainly the value 47 should not have caused a problem > > actually that's only half true ;), if you use UDEREF then the max value is > 46 only since there must be space for the shadow pgds (with 47 the first > context switch to userland will overwrite the kernel pgds). if you disable > UDEREF then 47 will work with the already mentioned caveat. I've UDEREF=y, that explains why 46 worked and 47 failed for me. > as for original problem, i'm afraid it's up to the clang/gcc guys to > accomodate special address space layouts, in the meantime gentoo has to do > the patching itself... I think: * Short term: Gentoo patch at clang/gcc build time. * Next step: Merge that patch upstream. * Best solution: Find out the address space size dynamically (having two different executables for PaX and non-PaX is evil). Can this be done heuristically by the asan initialization? Does some interface already provide information about the address space size? Do we need some new kernel interface for that? For reasons not known to me, the hard ulimit for "address space" says "unlimited" instead of the actual size. I think this should be set by the kernel to the actual size, and used. Having written a library to catch the same kind of bugs that asan catches some 20 years ago, given the huge amount of solutions for catching the kind of bugs that asan catches that have been presented over the decades, given that asan is not good for catching any new class of bugs, but only a restricted and old class of bugs, in a way which brings only faster debug/checked code, having not been bitten by this kinds of bugs typical of old C code for many years, I would like to express my regret that things like this, a niche of interest, block the unmasking of gcc 4.8.1. I find it a pity that often the prevailing logic is that of regarding gcc as a means for people to use packages, and often it is forgotten that gcc itself has users, millions of them, many more than the other things it breaks. I'm sorry if this is out of place. Regret expressed successfully. (In reply to Miguel Ramos from comment #17) > ... I do not agree. * Pointer-Checking lib's are far inferior to compiler-based solutions like asan w.r.t. the kinds of bugs found. * Among all popular (and free) compiler-based solutions, asan is by far (roughly by an order of magnitude) the fastest, and among the best, and probably the best maintained one. * asan finds by far more bugs than mudflap (and will replace mudflap in gcc afaik). It finds different, but in practice more relevant bugs than valgrind (at least more relevant for teaching / learning purposes). * asan finds less bugs than bgcc and provides worse error messages than bgcc, but bgcc is dead code for almost 10 years (and painfully slow). However, I would still happily accept an improved bgcc as a replacement for asan, if you are willing to maintain that beast... However, I could live with a gcc 4.8.x without asan, as long as a current llvm version with asan for pax is provided. Besides, to fix, only three constants need to be changed (see my comment 11). Hence, a static solution (building different gcc's for non-pax and pax systems, controlled e.g. by USE flags) should be trivial. Building a single gcc which generates different a.out's for non-pax and pax (e.g. controlled by two different gcc command line switches) should be only slightly more difficult. Building a gcc which generates asan a.out's which adapt themselves dynamically to both pax and non-pax systems is probably hard, but do we really need that flexibility now? (In reply to Miguel Ramos from comment #17) > I would like to express my regret that things like this, a niche of > interest, block the unmasking of gcc 4.8.1. This bug is not blocking the unmasking of 4.8.1. (In reply to Ryan Hill from comment #19) > (In reply to Miguel Ramos from comment #17) > > > I would like to express my regret that things like this, a niche of > > interest, block the unmasking of gcc 4.8.1. > > This bug is not blocking the unmasking of 4.8.1. Mmm. I misread the dependency tree then. >=gcc-4.8.0 is masked due to bug #461954 and this bug blocks #461954. I thought that it meant that this bug was one of the reasons. I came to bugzilla to try to understand why >=gcc-4.8.0 is masked. (In reply to Klaus Kusche from comment #18) > (In reply to Miguel Ramos from comment #17) > > ... > > I do not agree. > > * Pointer-Checking lib's are far inferior to compiler-based solutions like > asan > w.r.t. the kinds of bugs found. Libs are not that bad, there's electricfence too. I feel than when one starts to use one such tool that catches maybe 50% of those overrun-type bugs, one becomes aware of the problem and the other 50% go away too. [...] > However, I could live with a gcc 4.8.x without asan, > as long as a current llvm version with asan for pax is provided. That's like saying you can't live without asan in some compiler... I understand your concern and understand that you find asan much better than the alternatives. My issue was one of relative importance, of asan versus gcc. gcc has lots of users, a portion of them is using C, maybe 30% of those which are actually writing new code, and of those, maybe 10% (3%) is aware of buffer overruns, and of those many will simply be more careful, others will use tools they have been using for years, etc, etc, and maybe 1% (0.3%) is just waiting for asan. I belong to the portion of gcc users which can't wait for gcc-4.8.1 because of C++11 support. What apparently is indispensable to a portion of gcc users, to another portion is something which they really couldn't care less. I think the importance of asan today is much less than it would have been 15-20 years ago. There are new languages, new libraries, even the C library has improved in that respect. Most newbie programmers are using Java and their buffer overruns are always caught. Others use Haskell and they don't have buffers, let alone overruns. The bugs which I write daily, unfortunately, are not caught that easily. It's mostly algorithms which didn't really consider all cases which they should have. Not many tools to help there. And also many floating-point bugs, for which there could be tools, but there aren't. How can we help to try to get us both satisfied? (In reply to Miguel Ramos from comment #21) > (In reply to Klaus Kusche from comment #18) > > (In reply to Miguel Ramos from comment #17) > > > ... > > > > I do not agree. > > > > * Pointer-Checking lib's are far inferior to compiler-based solutions like > > asan > > w.r.t. the kinds of bugs found. > > Libs are not that bad, there's electricfence too. I feel than when one > starts to use one such tool that catches maybe 50% of those overrun-type > bugs, one becomes aware of the problem and the other 50% go away too. Actually, I use such tools for educational purposes (in beginner's programming courses at school). So they should be simple to use and understand, and they should catch as much as possible without relying on the pupil's brains. Moreover, the PC's in the labs are quite ressource-restricted. > [...] > > However, I could live with a gcc 4.8.x without asan, > > as long as a current llvm version with asan for pax is provided. > > That's like saying you can't live without asan in some compiler... I > understand your concern and understand that you find asan much better than > the alternatives. I still have a working bgcc (4.0.something), and as long as speed doesn't matter, it is better than asan (I would be happy if someone maintained that or improved asan to that level). Moreover, there is mudflap (which is not that good). Hence, there is no urgent need for asan, but strategically, asan is the way to go, because all alternatives seem to be dead code. > My issue was one of relative importance, of asan versus gcc. gcc has lots of > users, a portion of them is using C, maybe 30% of those which are actually > writing new code, and of those, maybe 10% (3%) is aware of buffer overruns, > and of those many will simply be more careful, others will use tools they > have been using for years, etc, etc, and maybe 1% (0.3%) is just waiting for > asan. > > I belong to the portion of gcc users which can't wait for gcc-4.8.1 because > of C++11 support. What apparently is indispensable to a portion of gcc > users, to another portion is something which they really couldn't care less. I agree with you that gcc 4.8 is urgently needed and should not be blocked by asan (it isn't anyway). (In reply to Klaus Kusche from comment #11) > Using a clang with that, when I compile an application with > -mllvm -asan-mapping-offset-log=40 -mllvm -asan-mapping-scale=3 > this results in > * an executable which can be started without allocation errors > * detects memory violations as it should > * seems to have a reasonable memory and shadow layout (I'm no asan > specialist) Do the hack still work with llvm 3.3? Can't get it to work. I use the use-after-free code on the AddressSanitizer home page. On Gcc 4.8.1 the mapping-offset and mapping-scale can't be change. Can you upstream the bug to the project home for llvm and gcc use the same lib. (In reply to Magnus Granberg from comment #23) > (In reply to Klaus Kusche from comment #11) > > Using a clang with that, when I compile an application with > > -mllvm -asan-mapping-offset-log=40 -mllvm -asan-mapping-scale=3 > > this results in > > * an executable which can be started without allocation errors > > * detects memory violations as it should > > * seems to have a reasonable memory and shadow layout (I'm no asan > > specialist) > Do the hack still work with llvm 3.3? Can't get it to work. > I use the use-after-free code on the AddressSanitizer home page. > On Gcc 4.8.1 the mapping-offset and mapping-scale can't be change. > Can you upstream the bug to the project home for llvm and gcc use the same > lib. Well, they seem to have messed it up seriously... As far as the shadow memory address goes, the place to patch has moved, but my idea should still work: 1.) ASAN_FLEXIBLE_MAPPING_AND_OFFSET seems to be 1 by default now, so no need to patch anything for that. 2.) kHighMemEnd is no longer hardcoded in asan_mapping.h, but defined in asan_rtl.cc . Around line 324, changing kHighMemEnd = (1ULL << 47) - 1; to kHighMemEnd = (1ULL << 42) - 1; should do the trick. Then, asan should work as before with -mllvm -asan-mapping-offset-log=40 -mllvm -asan-mapping-scale=3 or something like that. But: ==== They defined some sort of memory allocation code related to asan (I've no ideas about that), and it has adresses and sizes hardcoded for 47 bits all over the code! This causes an assert as soon as any program compiled with asan is started. At least the constants kAllocatorSpace and kAllocatorSize must be adapted, and they are defined independently at least in tsan_rtl.h, msan_allocator.cc, asan_allocator2.cc, sanitizer_allocator_testlib.cc and sanitizer_allocator_test.cc . I've no idea how to fix that. About pushing the bug upstream: I'm no developer, and I never had any contact with upstream asan, llvm or gcc. But someone should tell them that not all address spaces are created equal... (In reply to Klaus Kusche from comment #22) > (In reply to Miguel Ramos from comment #21) > > I belong to the portion of gcc users which can't wait for gcc-4.8.1 because > > of C++11 support. What apparently is indispensable to a portion of gcc > > users, to another portion is something which they really couldn't care less. > > I agree with you that gcc 4.8 is urgently needed and should not be blocked > by asan (it isn't anyway). assuming that we are talking gcc-4.8.1 on Gentoo in the main portage tree, the gcc-4.8.1,ebuild has KEYWORDS="". I understand that this is fundamentally different from being in some package.mask. Speaking strickly for myself, I cannot tell if all this would make my x86_64 based machine unstable (with 6GB of RAM, running a 3.8.13 kernel, llvm-3.1-r2). On the other hand, I am working on a program that can use advanced features supported in Fortran-08 and C++-11. asan behavior is not blocking the process of adding ~arch to gcc-4.8 > About pushing the bug upstream: > I'm no developer, and I never had any contact with upstream asan, llvm or > gcc. > But someone should tell them that not all address spaces are created equal... We know. asan works on a variety of platforms, not all of which are even x86. Fixing asan for PaX is a matter of a couple of #ifdefs in the code. The patches are welcome. I would encourage you to continue the discussion in https://code.google.com/p/address-sanitizer/issues/detail?id=228 >> I still have a working bgcc (4.0.something), >> and as long as speed doesn't matter, it is better than asan I am intrigued. Can someone please give me an example of a bug detectable by bgcc and not detectable by asan+msan+tsan? Best if you could submit a feature request here: https://code.google.com/p/address-sanitizer/issues/list As you correctly say, asan is "probably the best maintained" tool like this. And we'd like to make it even better. (In reply to Kostya Serebryany from comment #27) > >> I still have a working bgcc (4.0.something), > >> and as long as speed doesn't matter, it is better than asan > I am intrigued. Can someone please give me an example of a bug detectable > by bgcc and not detectable by asan+msan+tsan? > Best if you could submit a feature request here: > https://code.google.com/p/address-sanitizer/issues/list I believe there were some obscure bugs detected by bgcc and not by asan (bugs about non-array pointers I think? use-after-return? mid-object free?), but I don't remember for sure. But that's not the point. As I said before: Actually, I use such tools for educational purposes (in beginner's programming courses at school). The point is that asan's messages are intended for experts (and remain mysterious for non-experts), while bgcc's (not miro's) messages are readily readable and interpretable by beginners: They contain the *name* of the violated array and/or the bad pointer (from the debug info), they give sizes and offsets in elements (not bytes), they tell you to which object the pointer *should* point (which isn't always the nearest object), they read much more natural. Moreover, bgcc detects problems earlier. For example, mis-calculate some pointer (without immediately dereferencing it), and store the bad pointer in some data structure or pass it to some function. bgcc will point you right to the origin of the trouble. With asan, you have to work your way back yourself (from the bad dereference to the pointer calculation). And as a bonus, bgcc gives a nicely readable leak list, too. But as bgcc will not be reanimated, it's better to have a working asan than a rotten bgcc. (In reply to Klaus Kusche from comment #28) i'm not sure i'd knock asan's output that much. it's pretty good. i'll grant you it's not as good as the boundschecking gcc patch, but it's waaaaay better than mudflap :). you do mean boundschecking when you say bgcc right ? http://williambader.com/bounds/ i'm a big fan of that project. it's why we still carry it in gcc-3.3.x and 3.4.x. i also see there's a 4.0.x patch ... if someone wanted to submit the changes needed for adding that to Gentoo, i'd be happy to merge it. (In reply to SpanKY from comment #29) > you do mean boundschecking when you say bgcc right ? > http://williambader.com/bounds/ Yes, exactly. I was the one who asked for including it in Gentoo gcc years ago. http://endl.ch/clang-sanitizers-with-pax Fix and patches but is not something that should run on production. Were there any updates on this one? |
PaX hard limits the maximum virtual memory size of a task in security/Kconfig: config TASK_SIZE_MAX_SHIFT int depends on X86_64 default 47 if !PAX_PER_CPU_PGD default 42 if PAX_PER_CPU_PGD With these settings, llvm asan fails to allocate the memory it needs: ==13244== ERROR: Failed to allocate 0x20000001000 (2199023259648) bytes at address 0x0ffffffff000 (12) ==13244== ReserveShadowMemoryRange failed while trying to map 0x20000001000 bytes. Perhaps you're using ulimit -v (I'm not using ulimit -v) (This is llvm & clang 3.2) I experimented: 43 is still too small 45 results in a different asan error about overlapping memory 46 makes asan happy 47 fails to boot (spontaneous reboot while X is starting) Could we please have a default which allows asan to be used?