Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 389751 - =dev-java/icedtea-7.2.2.1 does not build on a PaX-enabled kernel
Summary: =dev-java/icedtea-7.2.2.1 does not build on a PaX-enabled kernel
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Java (show other bugs)
Hardware: x86 Linux
: Normal normal (vote)
Assignee: Java team
URL:
Whiteboard:
Keywords:
: 344135 (view as bug list)
Depends on:
Blocks:
 
Reported: 2011-11-06 19:32 UTC by Maciej S. Szmigiero
Modified: 2012-06-24 08:35 UTC (History)
5 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
Patch to skip the test_gamma script that makes the build fail (no-test_gamma.patch,1.97 KB, patch)
2011-11-12 13:52 UTC, Daniel Kuehn (RETIRED)
Details | Diff
strace of java call with -Xms600m -Xmx600m flags (stracelog,264.81 KB, text/plain)
2011-11-29 16:19 UTC, Andrej Gelenberg
Details
analyse of mmap calls (mmapana,6.32 KB, text/plain)
2011-11-29 16:21 UTC, Andrej Gelenberg
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Maciej S. Szmigiero 2011-11-06 19:32:02 UTC
There are two problems when trying to emerge dev-java/icedtea-7.2.0-r1 on a PaX-enabled x86 kernel:

1) early in the build process javac from the current system JDK is run with memory pool size of 1GB. This fail when SEGMEXEC is enabled with a message of "Could not reserve enough space for object heap".
The solution is to disable both SEGMEXEC and PAGEEXEC on javac. PAGEEXEC has to be disabled too, because when it is enabled the compiler (from icedtea-6.1.10 on my system) deadlocks for unknown reasons, eating 100% CPU.
This is really an existing JDK problem, which is triggered by IcedTea build process.

2) The "launcher" binary built during IcedTea build process gets killed when run. The solution is to disable MPROTECT, SEGMEXEC and PAGEEXEC on it just after linking.
I added "paxctl -spm $@" to "$(LAUNCHER)" target in openjdk/hotspot/make/linux/makefiles/launcher.make to work it around and let the build process to complete successfully.

However, this should be done more cleanly, preferably patched by ebuild when emerging icedtea on a PaX kernel.

Reproducible: Always

Steps to Reproduce:
1. Have PaX-enabled x86 kernel
2. emerge dev-java/icedtea
Comment 1 satmd 2011-11-07 03:19:57 UTC
I can confirm part 2 of this bug on gentoo ~amd64 no-multilib hardened with a pax-enabled kernel 3.0.8-hardened.

Suggested workaround works too.

For part 1, I probably was lucky my javac already had appropiate pax flags.
Comment 2 Maciej S. Szmigiero 2011-11-07 12:01:58 UTC
Part 1 should not be an issue on amd64 as it has a native NX bit support without tricks such as SEGMEXEC or PAGEEXEC.
MPROTECT still does apply, but javac has this disabled by ebuild when installed.
Comment 3 Magnus Granberg gentoo-dev 2011-11-12 13:52:23 UTC
The icedtea-7 missing the no-test_gamma.patch
Comment 4 Daniel Kuehn (RETIRED) gentoo-dev 2011-11-12 13:52:32 UTC
Created attachment 292281 [details, diff]
Patch to skip the test_gamma script that makes the build fail

After looking at the icedtea-6 ebuilds we found that this issue is a old one and a patch for it had been submitted before, see bug 244901, and after looking in the patches dir for icedtea-7 the patch wasn't found there.
Comment 5 Ralph Sennhauser (RETIRED) gentoo-dev 2011-11-13 22:55:29 UTC
http://icedtea.classpath.org/hg/icedtea6/rev/cb463b94b82d is the commit which fixed this issue for IcedTea6.

Committed a fix replicating the above for IcedTea7. Works for me on a PaX enabled kernel on amd64.

Thanks to Daniel Kuehn and Magnus Granberg for their findings.
Comment 6 Ralph Sennhauser (RETIRED) gentoo-dev 2011-11-18 11:22:29 UTC
The patch only fixes non bootstrap builds. I changed the ebuild to use the non bootstrap path whenever possible, which means no PaX if using gcj-jdk for now.

As for issue 1) is building icedtea-7 the only time this is triggered and other java applications run just fine? If using icedtea-bin-7 do you see the same happen?
Comment 7 Maciej S. Szmigiero 2011-11-18 15:21:14 UTC
(In reply to comment #6)
> The patch only fixes non bootstrap builds. I changed the ebuild to use the non
> bootstrap path whenever possible, which means no PaX if using gcj-jdk for now.
> 
> As for issue 1) is building icedtea-7 the only time this is triggered and other
> java applications run just fine? If using icedtea-bin-7 do you see the same
> happen?

Well if I try to build hello world using "-J-Xmx1024m" it exits with the same error.
Compiling it (hello world) without this switch or with object heap size of 700m or less works.
For reference malloc(1024*1024*1024) from C program fails too when SEGMEXEC is enabled.

I can try to install icedtea-bin, but I am pretty sure the effect will be the same if it does not have SEGMEXEC disabled.
Comment 8 Ralph Sennhauser (RETIRED) gentoo-dev 2011-11-23 19:08:11 UTC
(In reply to comment #7)
> Well if I try to build hello world using "-J-Xmx1024m" it exits with the same
> error.
> Compiling it (hello world) without this switch or with object heap size of 700m
> or less works.
> For reference malloc(1024*1024*1024) from C program fails too when SEGMEXEC is
> enabled.

Thanks for the explanation, just a bit surprising that this only surfaced now.

All 1.6/1.7 JDKs/JREs in tree have now the required PaX markings. You might have to reinstall the latest unstable version of each slot.
Comment 9 Maciej S. Szmigiero 2011-11-29 13:29:12 UTC
Builds fine now, thanks.
Comment 10 Andrej Gelenberg 2011-11-29 16:16:59 UTC
@Maciej Szmigiero  no, malloc(1024*1024*1024) did not fail. Code:

#include <stdlib.h>
#include <stdio.h>

int main() {
  void * p = malloc(1024 * 1024 * 1024);
  if ( p ) {
    free(p);
    return 0;
  } else {
    printf("malloc failed: %m\n");
    return 1;
  }
}

# gcc test.c
# pax -S a.out
# a.out
a.out return 0

It seems, what java do overlapping mmaps calls. Here strace linsting:

[pid  9533] munmap(0x7f34644f4000, 4096) = 0
[pid  9533] mmap(NULL, 1073741824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f3422196000
[pid  9533] munmap(0x7f3422196000, 1073741824) = 0
[pid  9533] mmap(NULL, 1075838976, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f3421f96000
[pid  9533] munmap(0x7f3421f96000, 1075838976) = 0
[pid  9533] mmap(0x7f3422000000, 1073741824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f3422000000
[pid  9533] mmap(0x7f3422000000, 4194304, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS|0x40000, -1, 0) = 0x7f3422000000
[pid  9533] mmap(NULL, 16777216, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f3421000000
[pid  9533] mmap(0x7f3421000000, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS|0x40000, -1, 0) = 0x7f3421000000
[pid  9533] mmap(0xca800000, 897581056, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0xca800000
[pid  9533] mmap(NULL, 1757184, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f3420e53000
[pid  9533] mmap(0x7f3420fff000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f3420fff000
[pid  9533] mmap(0xf3800000, 209715200, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS|0x40000, -1, 0) = 0xf3800000
[pid  9533] mmap(0x7f3420f9b000, 409600, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f3420f9b000
[pid  9533] mmap(0xda800000, 419430400, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS|0x40000, -1, 0) = -1 ENOMEM (Cannot allocate memory)
[pid  9533] write(1, "Error occurred during initializa"..., 43Error occurred during initialization of VM
) = 43
[pid  9533] write(1, "Could not reserve enough space f"..., 46Could not reserve enough space for object heap) = 46
[pid  9533] write(1, "\n", 1
)           = 1

Suspicious are:
[pid  9533] mmap(0xca800000, 897581056, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0xca800000
and
pid  9533] mmap(0xda800000, 419430400, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS|0x40000, -1, 0) = -1 ENOMEM (Cannot allocate memory)

The first mmap would alloc memory until 0x000100000000, and 0xda800000 is before that. Then it fail with MAP_FIXED set.
Comment 11 Andrej Gelenberg 2011-11-29 16:19:25 UTC
Created attachment 294193 [details]
strace of java call with -Xms600m -Xmx600m flags
Comment 12 Maciej S. Szmigiero 2011-11-29 16:21:19 UTC
Are you sure you are testing this on 32-bit system?

# cat test.c

#include <malloc.h>
#include <stdio.h>

int main(void)
{
        void *p = malloc(1024*1024*1024);
        printf("%p\n", p);
        return 0;
}

# gcc -o test test.c
# paxctl -v test
PaX control v0.5
Copyright 2004,2005,2006,2007 PaX Team <pageexec@freemail.hu>

- PaX flags: -------x-e-- [test]
        RANDEXEC is disabled
        EMUTRAMP is disabled


# strace ./test
execve("./test", ["./test"], [/* 35 vars */]) = 0
brk(0)                                  = 0x13988370
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x51281000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=58282, ...}) = 0
mmap2(NULL, 58282, PROT_READ, MAP_PRIVATE, 3, 0) = 0x51272000
close(3)                                = 0
open("/lib/libc.so.6", O_RDONLY)        = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0@o\1\0004\0\0\0\374"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1417548, ...}) = 0
mmap2(NULL, 1428672, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x51115000
mprotect(0x5126b000, 4096, PROT_NONE)   = 0
mmap2(0x5126c000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x156) = 0x5126c000
mmap2(0x5126f000, 11456, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x5126f000
close(3)                                = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x51114000
set_thread_area({entry_number:-1 -> 6, base_addr:0x511146c0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
mprotect(0x5126c000, 8192, PROT_READ)   = 0
mprotect(0x13985000, 4096, PROT_READ)   = 0
mprotect(0x5129f000, 4096, PROT_READ)   = 0
munmap(0x51272000, 58282)               = 0
mmap2(NULL, 1073745920, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
brk(0)                                  = 0x13988370
brk(0x539a9370)                         = 0x13988370
mmap2(NULL, 1073876992, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap2(NULL, 2097152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x50f14000
munmap(0x50f14000, 966656)              = 0
munmap(0x51100000, 81920)               = 0
mprotect(0x51000000, 135168, PROT_READ|PROT_WRITE) = 0
mmap2(NULL, 1073745920, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 6), ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x51280000
write(1, "(nil)\n"..., 6(nil)
)               = 6
exit_group(0)                           = ?
Comment 13 Andrej Gelenberg 2011-11-29 16:21:47 UTC
Created attachment 294195 [details]
analyse of mmap calls

This file has form:
<start address> <end address> <size> [*]

With * marked lines mean, what mmap call failed.
Comment 14 Ralph Sennhauser (RETIRED) gentoo-dev 2011-11-29 16:32:45 UTC
*** Bug 344135 has been marked as a duplicate of this bug. ***
Comment 15 Andrej Gelenberg 2011-11-29 16:37:52 UTC
it's 64-bit system, but i get same error
$ /usr/bin/java -Xms600m -Xmx600m
Error occurred during initialization of VM
Could not reserve enough space for object heap
Comment 16 Maciej S. Szmigiero 2011-11-29 17:31:29 UTC
(In reply to comment #15)
> it's 64-bit system, but i get same error
> $ /usr/bin/java -Xms600m -Xmx600m
> Error occurred during initialization of VM
> Could not reserve enough space for object heap

Andrej,

I think you are having an different issue here, as
SEGMEXEC is only used on 32-bit x86 architecture.

64-bit x86 has native NX bit support and does not
require hacks like split code/data vmas (SEGMEXEC).

Unfortunately, I do not have an 64-bit PaX-enabled box
to check what is wrong there.
Comment 17 Ralph Sennhauser (RETIRED) gentoo-dev 2011-12-02 12:45:52 UTC
Fixed bootstrap builds in icedtea-6.1.10.4-r3 and icedtea-7.2.0-r3 by building and pax marking the intermediate jdk.

Thanks for the report and the support from various sides in this matter.

@Andrej Gelenberg:
That looks like a sufficiently different issue, please open a separate bug.
Comment 18 Alon Bar-Lev 2012-06-20 20:06:16 UTC
Hello,
icedtea-7.2.2.1 has this again.

---
# /var/tmp/portage/dev-java/icedtea-7.2.2.1/work/icedtea-2.2.1/openjdk.build-boot/bin/java -XX:-PrintVMOptions -XX:+UnlockDiagnosticVMOptions -XX:-LogVMOutput -Xmx512m -Xms512m -XX:PermSize=32m -XX:MaxPermSize=160m -cp /var/tmp/portage/dev-java/icedtea-7.2.2.1/work/icedtea-2.2.1/openjdk.build-boot/classes sun.rmi.rmic.Main -classpath "/var/tmp/portage/dev-java/icedtea-7.2.2.1/work/icedtea-2.2.1/openjdk.build-boot/classes"                    -d /var/tmp/portage/dev-java/icedtea-7.2.2.1/work/icedtea-2.2.1/openjdk.build-boot/classes                              -v1.2                                           -keepgenerated                                  javax.management.remote.rmi.RMIConnectionImpl
Error occurred during initialization of VM
Could not reserve enough space for code cache
---

---
# paxctl -v /var/tmp/portage/dev-java/icedtea-7.2.2.1/work/icedtea-2.2.1/openjdk.build-boot/bin/java
PaX control v0.7
Copyright 2004,2005,2006,2007,2009,2010,2011,2012 PaX Team <pageexec@freemail.hu>

- PaX flags: -------x-e-- [/var/tmp/portage/dev-java/icedtea-7.2.2.1/work/icedtea-2.2.1/openjdk.build-boot/bin/java]
        RANDEXEC is disabled
        EMUTRAMP is disabled
---

Adding MPROTECT solves the issue.

Do you want to reopen or should I open a new bug?
Comment 19 Alon Bar-Lev 2012-06-22 20:13:33 UTC
(In reply to comment #18)
> Hello,
> icedtea-7.2.2.1 has this again.

Guys,
Please reopen, or response...
Comment 20 Anthony Basile gentoo-dev 2012-06-22 20:21:55 UTC
I haven't tested, but somehow I don't doubt it.  I'm opening unconfirmed.
Comment 21 Ralph Sennhauser (RETIRED) gentoo-dev 2012-06-24 08:35:25 UTC
Thanks to Andrew John Hughes the PaX patches from back then were added to upstream icedtea 7. This time, the reason is a security patch which changed where a freshly built jdk is used first. Either way, it's fixed now.

  24 Jun 2012; Ralph Sennhauser <sera@gentoo.org> icedtea-6.1.11.3.ebuild,
  files/icedtea-6_pax_kernel_support.patch, icedtea-7.2.1.1.ebuild,
  +files/icedtea-7.2.1.1-pax_mark_rmic_java.patch, icedtea-7.2.2.1.ebuild,
  +files/icedtea-7.2.2.1-pax_mark_rmic_java.patch:
  Fix building with PaX enabled kernels. #422525