There are two problems when trying to emerge dev-java/icedtea-7.2.0-r1 on a PaX-enabled x86 kernel: 1) early in the build process javac from the current system JDK is run with memory pool size of 1GB. This fail when SEGMEXEC is enabled with a message of "Could not reserve enough space for object heap". The solution is to disable both SEGMEXEC and PAGEEXEC on javac. PAGEEXEC has to be disabled too, because when it is enabled the compiler (from icedtea-6.1.10 on my system) deadlocks for unknown reasons, eating 100% CPU. This is really an existing JDK problem, which is triggered by IcedTea build process. 2) The "launcher" binary built during IcedTea build process gets killed when run. The solution is to disable MPROTECT, SEGMEXEC and PAGEEXEC on it just after linking. I added "paxctl -spm $@" to "$(LAUNCHER)" target in openjdk/hotspot/make/linux/makefiles/launcher.make to work it around and let the build process to complete successfully. However, this should be done more cleanly, preferably patched by ebuild when emerging icedtea on a PaX kernel. Reproducible: Always Steps to Reproduce: 1. Have PaX-enabled x86 kernel 2. emerge dev-java/icedtea
I can confirm part 2 of this bug on gentoo ~amd64 no-multilib hardened with a pax-enabled kernel 3.0.8-hardened. Suggested workaround works too. For part 1, I probably was lucky my javac already had appropiate pax flags.
Part 1 should not be an issue on amd64 as it has a native NX bit support without tricks such as SEGMEXEC or PAGEEXEC. MPROTECT still does apply, but javac has this disabled by ebuild when installed.
The icedtea-7 missing the no-test_gamma.patch
Created attachment 292281 [details, diff] Patch to skip the test_gamma script that makes the build fail After looking at the icedtea-6 ebuilds we found that this issue is a old one and a patch for it had been submitted before, see bug 244901, and after looking in the patches dir for icedtea-7 the patch wasn't found there.
http://icedtea.classpath.org/hg/icedtea6/rev/cb463b94b82d is the commit which fixed this issue for IcedTea6. Committed a fix replicating the above for IcedTea7. Works for me on a PaX enabled kernel on amd64. Thanks to Daniel Kuehn and Magnus Granberg for their findings.
The patch only fixes non bootstrap builds. I changed the ebuild to use the non bootstrap path whenever possible, which means no PaX if using gcj-jdk for now. As for issue 1) is building icedtea-7 the only time this is triggered and other java applications run just fine? If using icedtea-bin-7 do you see the same happen?
(In reply to comment #6) > The patch only fixes non bootstrap builds. I changed the ebuild to use the non > bootstrap path whenever possible, which means no PaX if using gcj-jdk for now. > > As for issue 1) is building icedtea-7 the only time this is triggered and other > java applications run just fine? If using icedtea-bin-7 do you see the same > happen? Well if I try to build hello world using "-J-Xmx1024m" it exits with the same error. Compiling it (hello world) without this switch or with object heap size of 700m or less works. For reference malloc(1024*1024*1024) from C program fails too when SEGMEXEC is enabled. I can try to install icedtea-bin, but I am pretty sure the effect will be the same if it does not have SEGMEXEC disabled.
(In reply to comment #7) > Well if I try to build hello world using "-J-Xmx1024m" it exits with the same > error. > Compiling it (hello world) without this switch or with object heap size of 700m > or less works. > For reference malloc(1024*1024*1024) from C program fails too when SEGMEXEC is > enabled. Thanks for the explanation, just a bit surprising that this only surfaced now. All 1.6/1.7 JDKs/JREs in tree have now the required PaX markings. You might have to reinstall the latest unstable version of each slot.
Builds fine now, thanks.
@Maciej Szmigiero no, malloc(1024*1024*1024) did not fail. Code: #include <stdlib.h> #include <stdio.h> int main() { void * p = malloc(1024 * 1024 * 1024); if ( p ) { free(p); return 0; } else { printf("malloc failed: %m\n"); return 1; } } # gcc test.c # pax -S a.out # a.out a.out return 0 It seems, what java do overlapping mmaps calls. Here strace linsting: [pid 9533] munmap(0x7f34644f4000, 4096) = 0 [pid 9533] mmap(NULL, 1073741824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f3422196000 [pid 9533] munmap(0x7f3422196000, 1073741824) = 0 [pid 9533] mmap(NULL, 1075838976, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f3421f96000 [pid 9533] munmap(0x7f3421f96000, 1075838976) = 0 [pid 9533] mmap(0x7f3422000000, 1073741824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f3422000000 [pid 9533] mmap(0x7f3422000000, 4194304, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS|0x40000, -1, 0) = 0x7f3422000000 [pid 9533] mmap(NULL, 16777216, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f3421000000 [pid 9533] mmap(0x7f3421000000, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS|0x40000, -1, 0) = 0x7f3421000000 [pid 9533] mmap(0xca800000, 897581056, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0xca800000 [pid 9533] mmap(NULL, 1757184, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f3420e53000 [pid 9533] mmap(0x7f3420fff000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f3420fff000 [pid 9533] mmap(0xf3800000, 209715200, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS|0x40000, -1, 0) = 0xf3800000 [pid 9533] mmap(0x7f3420f9b000, 409600, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f3420f9b000 [pid 9533] mmap(0xda800000, 419430400, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS|0x40000, -1, 0) = -1 ENOMEM (Cannot allocate memory) [pid 9533] write(1, "Error occurred during initializa"..., 43Error occurred during initialization of VM ) = 43 [pid 9533] write(1, "Could not reserve enough space f"..., 46Could not reserve enough space for object heap) = 46 [pid 9533] write(1, "\n", 1 ) = 1 Suspicious are: [pid 9533] mmap(0xca800000, 897581056, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0xca800000 and pid 9533] mmap(0xda800000, 419430400, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS|0x40000, -1, 0) = -1 ENOMEM (Cannot allocate memory) The first mmap would alloc memory until 0x000100000000, and 0xda800000 is before that. Then it fail with MAP_FIXED set.
Created attachment 294193 [details] strace of java call with -Xms600m -Xmx600m flags
Are you sure you are testing this on 32-bit system? # cat test.c #include <malloc.h> #include <stdio.h> int main(void) { void *p = malloc(1024*1024*1024); printf("%p\n", p); return 0; } # gcc -o test test.c # paxctl -v test PaX control v0.5 Copyright 2004,2005,2006,2007 PaX Team <pageexec@freemail.hu> - PaX flags: -------x-e-- [test] RANDEXEC is disabled EMUTRAMP is disabled # strace ./test execve("./test", ["./test"], [/* 35 vars */]) = 0 brk(0) = 0x13988370 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x51281000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=58282, ...}) = 0 mmap2(NULL, 58282, PROT_READ, MAP_PRIVATE, 3, 0) = 0x51272000 close(3) = 0 open("/lib/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0@o\1\0004\0\0\0\374"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=1417548, ...}) = 0 mmap2(NULL, 1428672, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x51115000 mprotect(0x5126b000, 4096, PROT_NONE) = 0 mmap2(0x5126c000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x156) = 0x5126c000 mmap2(0x5126f000, 11456, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x5126f000 close(3) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x51114000 set_thread_area({entry_number:-1 -> 6, base_addr:0x511146c0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0 mprotect(0x5126c000, 8192, PROT_READ) = 0 mprotect(0x13985000, 4096, PROT_READ) = 0 mprotect(0x5129f000, 4096, PROT_READ) = 0 munmap(0x51272000, 58282) = 0 mmap2(NULL, 1073745920, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory) brk(0) = 0x13988370 brk(0x539a9370) = 0x13988370 mmap2(NULL, 1073876992, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory) mmap2(NULL, 2097152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x50f14000 munmap(0x50f14000, 966656) = 0 munmap(0x51100000, 81920) = 0 mprotect(0x51000000, 135168, PROT_READ|PROT_WRITE) = 0 mmap2(NULL, 1073745920, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory) fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 6), ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x51280000 write(1, "(nil)\n"..., 6(nil) ) = 6 exit_group(0) = ?
Created attachment 294195 [details] analyse of mmap calls This file has form: <start address> <end address> <size> [*] With * marked lines mean, what mmap call failed.
*** Bug 344135 has been marked as a duplicate of this bug. ***
it's 64-bit system, but i get same error $ /usr/bin/java -Xms600m -Xmx600m Error occurred during initialization of VM Could not reserve enough space for object heap
(In reply to comment #15) > it's 64-bit system, but i get same error > $ /usr/bin/java -Xms600m -Xmx600m > Error occurred during initialization of VM > Could not reserve enough space for object heap Andrej, I think you are having an different issue here, as SEGMEXEC is only used on 32-bit x86 architecture. 64-bit x86 has native NX bit support and does not require hacks like split code/data vmas (SEGMEXEC). Unfortunately, I do not have an 64-bit PaX-enabled box to check what is wrong there.
Fixed bootstrap builds in icedtea-6.1.10.4-r3 and icedtea-7.2.0-r3 by building and pax marking the intermediate jdk. Thanks for the report and the support from various sides in this matter. @Andrej Gelenberg: That looks like a sufficiently different issue, please open a separate bug.
Hello, icedtea-7.2.2.1 has this again. --- # /var/tmp/portage/dev-java/icedtea-7.2.2.1/work/icedtea-2.2.1/openjdk.build-boot/bin/java -XX:-PrintVMOptions -XX:+UnlockDiagnosticVMOptions -XX:-LogVMOutput -Xmx512m -Xms512m -XX:PermSize=32m -XX:MaxPermSize=160m -cp /var/tmp/portage/dev-java/icedtea-7.2.2.1/work/icedtea-2.2.1/openjdk.build-boot/classes sun.rmi.rmic.Main -classpath "/var/tmp/portage/dev-java/icedtea-7.2.2.1/work/icedtea-2.2.1/openjdk.build-boot/classes" -d /var/tmp/portage/dev-java/icedtea-7.2.2.1/work/icedtea-2.2.1/openjdk.build-boot/classes -v1.2 -keepgenerated javax.management.remote.rmi.RMIConnectionImpl Error occurred during initialization of VM Could not reserve enough space for code cache --- --- # paxctl -v /var/tmp/portage/dev-java/icedtea-7.2.2.1/work/icedtea-2.2.1/openjdk.build-boot/bin/java PaX control v0.7 Copyright 2004,2005,2006,2007,2009,2010,2011,2012 PaX Team <pageexec@freemail.hu> - PaX flags: -------x-e-- [/var/tmp/portage/dev-java/icedtea-7.2.2.1/work/icedtea-2.2.1/openjdk.build-boot/bin/java] RANDEXEC is disabled EMUTRAMP is disabled --- Adding MPROTECT solves the issue. Do you want to reopen or should I open a new bug?
(In reply to comment #18) > Hello, > icedtea-7.2.2.1 has this again. Guys, Please reopen, or response...
I haven't tested, but somehow I don't doubt it. I'm opening unconfirmed.
Thanks to Andrew John Hughes the PaX patches from back then were added to upstream icedtea 7. This time, the reason is a security patch which changed where a freshly built jdk is used first. Either way, it's fixed now. 24 Jun 2012; Ralph Sennhauser <sera@gentoo.org> icedtea-6.1.11.3.ebuild, files/icedtea-6_pax_kernel_support.patch, icedtea-7.2.1.1.ebuild, +files/icedtea-7.2.1.1-pax_mark_rmic_java.patch, icedtea-7.2.2.1.ebuild, +files/icedtea-7.2.2.1-pax_mark_rmic_java.patch: Fix building with PaX enabled kernels. #422525