Here is a summary, as far as I can tell after some testing:
1. The problem is with using a hardened compiler, not with hardened kernel. JamVM segfaults when executed under regular kernel, too.
2. JamVM worked fine when compiled with hardened gcc-4.4
3. JamVM, when compiled with *any* profile of hardened gcc-4.5.3-r1 (including vanilla) segfaults on any class file (e.g., empty main(String args)).
4. Compiling jamvm on a non-hardened x86 host (same gcc version), and then running the resulting /usr/bin/jamvm in the hardened environment (regardless of the kernel), does not segfault.
5. Compiling jamvm with a hardened gcc, and running the resulting /usr/bin/jamvm on a regular x86 host (no hardened libraries or kernel) produces a segfault. So the issue appears to be limited to code produced by gcc for /usr/bin/jamvm (i.e., not any of the other hardened libraries, or gnu-classpath, etc.).
5. The problem persists with any CFLAGS, LDFLAGS and USE flags used for jamvm, e.g.: CFLAGS=-O0 LDFLAGS= emerge -q1 jamvm - still segfaults.
6. With -Xnoinlining, jamvm runs normally, even with complex long-running applications.
I did the testing on x86 only, as I have no hardened amd64. With regular amd64 gcc, jamvm does not segfault.
gcc USE-flags: hardened nls nptl openmp
jamvm USE-flags: (none)
gnu-classpath USE-flags: alsa gconf gmp gstreamer gtk xml
The problem persists with hardened gcc-4.5.3-r2 (piepatchset 0.4.7).
Hi, is anyone else able to confirm? This seems like a really strange issue: caused by any profile of hardened gcc, easy to reproduce.
Is it still there with gcc 4.8.x or newer