Bug 647954 – dev-java/icedtea-bin-3.6.0 on x86 with sys-devel/gcc-7.3.0 sys-kernel/gentoo-sources-4.15.0 - Internal Error (os_linux_x86.cpp:291), pid=21702, tid=0x8f689b40 // fatal error: An irrecoverable SI_KERNEL SIGSEGV has occurred due to unstable signal handling

• Status: RESOLVED FIXED
• Product: Gentoo Linux

[Description lebkoungcity 2018-02-17 19:21:04 UTC]

Created attachment 519896 [details]
emerge --info dev-java/icedtea-bin

Hello,

after upgrading gcc to sys-devel/gcc-7.3.0 and the kernel to sys-kernel/gentoo-sources-4.15.0 java doesn't run anymore. But only my 32bit-laptop is affected, on the 64bit-desktop java runs fine. I wanted to switch from dev-java/icedtea-bin-3.6.0 to dev-java/icedtea-3.6.0 but I found out I cannot compile it without working java... I also tried dev-java/oracle-jre-bin-1.8.0.162 and later took the ebuild (from funtoo put in my local overlay) for dev-java/icedtea-bin-7.2.6.10 (the latter in hope I could use it to compile dev-java/icedtea-3.6.0) but they don't work neither.


So, the current configuration is:
System is 32bit --> x86
sys-kernel/gentoo-sources-4.15.0
sys-devel/gcc-7.3.0
dev-java/icedtea-bin-3.6.0


When I want to start an java application - even if it was just 'java -version' the message is:

#                                                                                                                                              
# A fatal error has been detected by the Java Runtime Environment:                                                                             
#                                                                                                                                              
#  Internal Error (os_linux_x86.cpp:291), pid=21702, tid=0x8f689b40                                                                            
#  fatal error: An irrecoverable SI_KERNEL SIGSEGV has occurred due to unstable signal handling in this distribution.                          
#                                                                                                                                              
# JRE version:  (8.0_151-b12) (build )                                                                                                         
# Java VM: OpenJDK Server VM (25.151-b12 mixed mode linux-x86 )                                                                                
# Derivative: IcedTea 3.6.0                                                                                                                    
# Distribution: NAME=Gentoo, package Gentoo icedtea-3.6.0
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/test/hs_err_pid21702.log
#
# If you would like to submit a bug report, please include
# instructions on how to reproduce the bug and visit:
#   http://icedtea.classpath.org/bugzilla
#


I add as attachments:
emerge --info dev-java-/icedtea-bin
hs_err_pid21702.log
build.log for dev-java/icedtea-3.6.0


If there's more information needed, please tell me. I'd be glad if I could provide it.

Greetings,
Andy

[Comment 1 lebkoungcity 2018-02-17 19:25:23 UTC]

Created attachment 519898 [details]
hs_err_pid21702.log

[Comment 2 lebkoungcity 2018-02-17 19:28:11 UTC]

Created attachment 519900 [details]
build.log of dev-java/icedtea-3.6.0

[Comment 3 James Le Cuirot 2018-02-18 10:43:57 UTC]

Thanks for trying Oracle as well, that certainly makes things interesting. Just to clarify, you cannot even run say a simple Hello World with Oracle's java? At this point, I have no idea what might be causing this. :(

[Comment 4 lebkoungcity 2018-02-18 12:41:03 UTC]

(In reply to James Le Cuirot from comment #3)
> Thanks for trying Oracle as well, that certainly makes things interesting.
> Just to clarify, you cannot even run say a simple Hello World with Oracle's
> java? At this point, I have no idea what might be causing this. :(

I just tried it:
nano HelloWorld.java (added as attachment)

- trying to compile it with icedtea-bin ('javac HelloWorld.java') --> gives the same error as before:
'Internal Error (os_linux_x86.cpp:291), pid=21702, tid=0x8f689b40 // fatal error: An irrecoverable SI_KERNEL SIGSEGV has occurred due to unstable signal handling')

- trying to compile it with oracle-jre-bin ('javac HelloWorld.java') --> gives:
* javac is not available for oracle-jre-bin on i686
* IMPORTANT: some Java tools are not available on some VMs on some architectures

- so I compiled it on the 64bit-system (with icedtea-bin) and copied onto the 32bit-system

- trying to run it on the 32bit-system with icedtea-bin gives:
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (os_linux_x86.cpp:291), pid=13290, tid=0x8f696b40
#  fatal error: An irrecoverable SI_KERNEL SIGSEGV has occurred due to unstable signal handling in this distribution.
#
# JRE version:  (8.0_151-b12) (build )
# Java VM: OpenJDK Server VM (25.151-b12 mixed mode linux-x86 )
# Derivative: IcedTea 3.6.0
# Distribution: NAME=Gentoo, package Gentoo icedtea-3.6.0
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/andy/tmp/java-test/hs_err_pid13290.log
#
# If you would like to submit a bug report, please include
# instructions on how to reproduce the bug and visit:
#   http://icedtea.classpath.org/bugzilla
#
Abgebrochen

- trying to run it on the 32bit-system with oracle-jre-bin gives:
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (os_linux_x86.cpp:291), pid=13171, tid=0xa4958b40
#  fatal error: An irrecoverable SI_KERNEL SIGSEGV has occurred due to unstable signal handling in this distribution.
#
# JRE version:  (8.0_162-b12) (build )
# Java VM: Java HotSpot(TM) Client VM (25.162-b12 mixed mode linux-x86 )
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/andy/tmp/java-test/hs_err_pid13171.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
Abgebrochen



At the end of the resulting hs_err_pidXXXXX.log-file I noticed some lines:
- icedtea-bin:

Memory: 4k page, physical 1550696k(34260k free), swap 2097148k(1978620k free)

vm_info: OpenJDK Server VM (25.151-b12) for linux-x86 JRE (1.8.0_151-b12), built on Nov  3 2017 11:10:10 by "portage" with gcc 5.4.0

- oracle-jre-bin:

Memory: 4k page, physical 1550696k(34772k free), swap 2097148k(1817084k free)

vm_info: Java HotSpot(TM) Client VM (25.162-b12) for linux-x86 JRE (1.8.0_162-b12), built on Dec 19 2017 20:38:55 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)


--> Since it worked until updating to gcc-7.3.0 could it be that there are incompatibilities to gcc 5.4.0 / gcc 4.3.0? (OK, that would not explain why it works on the 64bit-system...)

--> Would it be possible that I have misconfigured the kernel in some way while doing the upgrade to gentoo-sources-4.15.0?

[Comment 5 lebkoungcity 2018-02-18 12:42:02 UTC]

Created attachment 520016 [details]
HelloWorld.java

[Comment 6 James Le Cuirot 2018-02-18 13:21:25 UTC]

I don't know the answers to those questions yet but I missed that you were trying to use Oracle JRE. Please use oracle-jdk-bin, which will give you the javac compiler.

[Comment 7 lebkoungcity 2018-02-18 15:13:14 UTC]

(In reply to James Le Cuirot from comment #6)
> I don't know the answers to those questions yet but I missed that you were
> trying to use Oracle JRE. Please use oracle-jdk-bin, which will give you the
> javac compiler.

OK, did that, unmerged oracle-jre-bin just to be shure, emerged oracle-jdk-bin and switched to it:

eselect java-vm list
Available Java Virtual Machines:                                                                                                               
  [1]   icedtea-bin-8                                                                                                                          
  [2]   oracle-jdk-bin-1.8  system-vm user-vm


Then to use the compiler:

javac HelloWorld.java                                                                             
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (os_linux_x86.cpp:291), pid=22880, tid=0xa4958b40
#  fatal error: An irrecoverable SI_KERNEL SIGSEGV has occurred due to unstable signal handling in this distribution.
#
# JRE version:  (8.0_162-b12) (build )
# Java VM: Java HotSpot(TM) Client VM (25.162-b12 mixed mode linux-x86 )
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/andy/tmp/java-test/hs_err_pid22880.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
Abgebrochen


Right now I'm compiling a new kernel: sys-kernel/gentoo-sources-4.15.4 but this will take some time on this machine.

[Comment 8 lebkoungcity 2018-02-18 19:10:07 UTC]

(In reply to lebkoungcity from comment #7)

> Right now I'm compiling a new kernel: sys-kernel/gentoo-sources-4.15.4 but
> this will take some time on this machine.


There's no difference with the new kernel sys-kernel/gentoo-sources-4.15.4.

[Comment 9 James Le Cuirot 2018-02-18 20:03:51 UTC]

Okay. To be honest, I'm thinking this must be something wrong with your kernel config or something else unique to your system. If there was a wider problem with Oracle's Java on 32-bit Linux, I think I would have heard about it by now. You can upload your kernel config but spotting the issue would be tricky. It would be good to try a kernel from another distribution like Fedora. An easy way to do this would be to boot Fedora from a USB stick and chroot into your system.

[Comment 10 R030t1 2018-02-24 07:44:06 UTC]

I highly recommend seeking guidance from the OpenJDK community.

[Comment 11 lebkoungcity 2018-02-24 21:26:27 UTC]

(In reply to James Le Cuirot from comment #9)
> Okay. To be honest, I'm thinking this must be something wrong with your
> kernel config or something else unique to your system. If there was a wider
> problem with Oracle's Java on 32-bit Linux, I think I would have heard about
> it by now. You can upload your kernel config but spotting the issue would be
> tricky. It would be good to try a kernel from another distribution like
> Fedora. An easy way to do this would be to boot Fedora from a USB stick and
> chroot into your system.


Now I had the time to get Fedora, boot it and chroot from this into the system on the hdd. But java spits out the same error...

see attachments:
'hs_err_pid1772.log' --> icedtea-bin
'hs_err_pid1962.log' --> oracle-jre-bin


So it seems that my kernel config might not be the cause of the error...

[Comment 12 lebkoungcity 2018-02-24 21:27:31 UTC]

Created attachment 520952 [details]
hs_err_pid1772.log

[Comment 13 lebkoungcity 2018-02-24 21:28:13 UTC]

Created attachment 520954 [details]
hs_err_pid1962.log

[Comment 14 James Le Cuirot 2018-02-24 22:43:38 UTC]

(In reply to lebkoungcity from comment #11)
> Now I had the time to get Fedora, boot it and chroot from this into the
> system on the hdd. But java spits out the same error...
Many thanks for trying that. If we assume it's not merely the kernel version then the remaining likely culprits are glibc and gcc. You said this happened after upgrading gcc. Obviously you didn't compile Oracle's Java but gcc also provides libstdc++.so.6. If you still have gcc 6 installed, maybe you could try:

LD_LIBRARY_PATH=/usr/lib/gcc/i686-pc-linux-gnu/6.4.0 java ...

[Comment 15 lebkoungcity 2018-02-24 23:00:08 UTC]

(In reply to James Le Cuirot from comment #14)
> (In reply to lebkoungcity from comment #11)
> > Now I had the time to get Fedora, boot it and chroot from this into the
> > system on the hdd. But java spits out the same error...
> Many thanks for trying that. If we assume it's not merely the kernel version
> then the remaining likely culprits are glibc and gcc. You said this happened
> after upgrading gcc. Obviously you didn't compile Oracle's Java but gcc also
> provides libstdc++.so.6. If you still have gcc 6 installed, maybe you could
> try:
> 
> LD_LIBRARY_PATH=/usr/lib/gcc/i686-pc-linux-gnu/6.4.0 java ...



I have to thank you for showing interest!

Some minutes before you posted this I started recompiling of gcc (7.3.0) and glibc (2.25-r10) just for the sake of it. This will last about 9 - 10 hours.

Meanwhile I tried your suggestion but unfortunately this gives the same error.
I tried:
LD_LIBRARY_PATH=/usr/lib/gcc/i686-pc-linux-gnu/6.4.0 java -version
LD_LIBRARY_PATH=/usr/lib/gcc/i686-pc-linux-gnu/6.4.0 /opt/icedtea-bin-3.6.0/bin/java -version
LD_LIBRARY_PATH=/usr/lib/gcc/i686-pc-linux-gnu/6.4.0 /opt/oracle-jdk-bin-1.8.0.162/bin/java -version
And no significant difference in the output...

[Comment 16 Francis Devereux 2018-02-26 15:33:12 UTC]

I'm having the same problem and I think it *may* have started happening when I upgraded from the 13.0 to the 17.0 profile.

I'm currently running gcc-6.4.0-r1. I tried downgrading to =sys-devel/gcc-5.4.0-r4 (just installed it and used gcc-config to select it, didn't rebuild everything with it) and that did not fix the problem.

[Comment 17 James Le Cuirot 2018-02-26 16:00:53 UTC]

(In reply to Francis Devereux from comment #16)
> I'm currently running gcc-6.4.0-r1. I tried downgrading to
> =sys-devel/gcc-5.4.0-r4 (just installed it and used gcc-config to select it,
> didn't rebuild everything with it) and that did not fix the problem.

I don't think selecting it is enough, you need to use LD_PRELOAD as above. I checked /etc/ld.so.conf.d/05gcc-i686-pc-linux-gnu.conf and found that it always orders the GCC versions the same with the newest first, regardless of which one is selected.

[Comment 18 Wendy 2018-02-26 23:44:29 UTC]

I had something similar with building (with bootstrap flag) of icedtea on a Intel Core duo 32-bits.
Temporarily removing -march=native solved the build for me

[Comment 19 Francis Devereux 2018-02-27 12:32:29 UTC]

(In reply to James Le Cuirot from comment #17)
> (In reply to Francis Devereux from comment #16)
> > I'm currently running gcc-6.4.0-r1. I tried downgrading to
> > =sys-devel/gcc-5.4.0-r4 (just installed it and used gcc-config to select it,
> > didn't rebuild everything with it) and that did not fix the problem.
> 
> I don't think selecting it is enough, you need to use LD_PRELOAD as above.

Thanks for the suggestion, LD_PRELOAD and LD_LIBRARY_PATH don't make a difference though:

francis@gecko ~/src/java/hellow $ LD_PRELOAD=/usr/lib/gcc/i686-pc-linux-gnu/5.4.0/libstdc++.so LD_LIBRARY_PATH=/usr/lib/gcc/i686-pc-linux-gnu/5.4.0 javac HelloWorld.java 
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (os_linux_x86.cpp:291), pid=17849, tid=0xb6c06b40
#  fatal error: An irrecoverable SI_KERNEL SIGSEGV has occurred due to unstable signal handling in this distribution.
#
# JRE version: OpenJDK Runtime Environment (8.0_151-b12) (build 1.8.0_151-b12)
# Java VM: OpenJDK Server VM (25.151-b12 mixed mode linux-x86 )
# Derivative: IcedTea 3.6.0
# Distribution: NAME=Gentoo, package Gentoo icedtea-3.6.0
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/francis/src/java/hellow/hs_err_pid17849.log
#
# If you would like to submit a bug report, please include
# instructions on how to reproduce the bug and visit:
#   http://icedtea.classpath.org/bugzilla
#
Aborted

[Comment 20 James Le Cuirot 2018-03-04 11:29:11 UTC]

(In reply to Wendy from comment #18)
> I had something similar with building (with bootstrap flag) of icedtea on a
> Intel Core duo 32-bits.
> Temporarily removing -march=native solved the build for me

I'm not sure that's really related as these users cannot run Java at all and you need it to build icedtea in the first place. I build icedtea-bin with -march=i686 anyway.

(In reply to Francis Devereux from comment #19)
> (In reply to James Le Cuirot from comment #17)
> > (In reply to Francis Devereux from comment #16)
> > > I'm currently running gcc-6.4.0-r1. I tried downgrading to
> > > =sys-devel/gcc-5.4.0-r4 (just installed it and used gcc-config to select it,
> > > didn't rebuild everything with it) and that did not fix the problem.
> > 
> > I don't think selecting it is enough, you need to use LD_PRELOAD as above.
> 
> Thanks for the suggestion, LD_PRELOAD and LD_LIBRARY_PATH don't make a
> difference though:

Sorry for mixing up LD_PRELOAD and LD_LIBRARY_PATH but you got the idea. :)

The ebuild isn't published yet but I've just built the new icedtea-bin for 3.7.0 and this will be the first build from a 17.0 profile. 17.0 is primarily about PIE and I doubt that would be the cause as that tends to result in build time issues rather than runtime. The switch to gcc 6 may well make a difference though. Grab it from this URL and you should be able to just extract it and run it in place without an ebuild.

https://dev.gentoo.org/~chewi/distfiles/icedtea-bin-core-3.7.0-x86.tar.xz

[Comment 21 James Le Cuirot 2018-03-04 11:37:48 UTC]

Wow okay, just as I posted that, I thought I better try it myself and although it worked fine in my 32-bit chroot, I got the segfault on the host 64-bit system. This is true for 3.6.0 and 3.7.0. I then tried Oracle's 32-bit JRE and got the same. I'll see what I can find out now.

[Comment 22 James Le Cuirot 2018-03-04 21:47:41 UTC]

So in my chroot, I tried updating to glibc 2.26 (from 2.25) and updating to gcc 7 (from 6) but neither of those broke anything. I then rebuilt icedtea in that environment and it still segfaults on the host.

I also tried running the java binary from the host but using libraries from the chroot. This does work but probably isn't that much different to actually chrooting.

This is a tricky one...

[Comment 23 James Le Cuirot 2018-03-04 23:20:25 UTC]

Hmmm, maybe Wendy was right though I may have misunderstood her initially. I rebuilt glibc with the same CFLAGS as the host and although HelloWorld works, just calling javac triggers the segfault, as does using jar on an archive. The initial "emerge --info" posted here has -march=pentium-m, which is quite old, and suggests this may have something to do with SSE. I'll keep digging.

[Comment 24 Maciej S. Szmigiero 2018-03-05 17:55:21 UTC]

I have also hit this issue on 32-bit x86, with gcc 7.3.0,
CFLAGS="-O2 -march=pentium-m -pipe".

I think there are two issues at play there:
1) icedtea-bin and icedtea are unstable with glibc compiled with
gcc 7.3.0,
I don't know the reason for this but it is easily worked around by
just recompiling glibc with gcc 6.4.0,
These crashes happen from time to time, especially during large Java
compiles (what is important that simple invocation of java or javac
usually does not crash - as long as the issue number 2 is taken care
of),

The rest of this comment will not be about this issue.

2) icedtea does not build - the bootstrapped compiler immediately crashes.
This happens all the time, even when just calling java or javac without
any parameters.

This issue is caused by HotSpot dynamic compiler generating a machine code
that does not keep the 16-byte stack alignement required by i386 psABI.
Then, when such generated code calls back a gcc-compiled C/C++ code (in JVM,
glibc or some other library) this misaligned stack travels along a call chain
until it hits a SSE operation.
Which then segfaults due to a misaligned address.

Why this didn't occur earlier I don't know - perhaps it worked by pure luck.

I have tried to fix the code generator but gave up - there are simply too
many places there needing a fix (pretty much every generated stack push in
functions that call C/C++ code has to be accounted for).

However, gcc has a nice "force_align_arg_pointer" function attribute that we
can attach to C/C++ code entry points so the compiler will realign the stack
pointer for us.
I've created a "icedtea-align-fix.patch" patch that does this.
With this patch applied I was able to compile and run icedtea successfully
with gcc 7.3.0 on 32-bit x86.

I have also prepared "icedtea-align-check.patch" patch which will immediately
crash the VM with an access to 0xbad574cc ("badstacc") if it detects unaligned
stack on an entry from generated code.
This helps debugging immensely, since the VM no longer crashes somewhere deeply
in the call chain, in an unrelated function that just happen to use SSE.

How to build a working icedtea:
1) Switch to gcc 6.4.0 temporarily,

2) Rebuild glibc (tried glibc-2.26-r6),
This is needed since with glibc built with gcc 7.3.0 icedtea-bin crashes
inside glibc and also built icedtea is unstable due to the issue number 1,

3) Emerge icedtea-bin,

4) Switch back to gcc 7.3.0,

5) Start emerging icedtea-3.6.0, stop it (Ctrl-Z) just after it unpacks
HotSpot sources, but before it starts to compile it (the best moment is when
the build process prints something about touching
"stamps/extract-hotspot.stamp"),

6) Go to the package build source tree and patch it with
icedtea-align-fix.patch (and, optionally, also icedtea-align-check.patch),

7) Resume the build process (fg),

8) You can now remove icedtea-bin.

Will open an icedtea bug for this issue since it is clearly a HotSpot bug
(checked the current HotSpot hg repository, the relevant code there is the
same, just files are moved around).

[Comment 25 Maciej S. Szmigiero 2018-03-05 17:57:08 UTC]

Created attachment 522456 [details, diff]
add force_align_arg_pointer to C/C++ code entry points

[Comment 26 Maciej S. Szmigiero 2018-03-05 17:57:47 UTC]

Created attachment 522458 [details, diff]
better unaligned stack checking

[Comment 27 Maciej S. Szmigiero 2018-03-05 19:47:05 UTC]

(In reply to Maciej S. Szmigiero from comment #24)
> Will open an icedtea bug for this issue since it is clearly a HotSpot bug

Opened a icedtea bug 3533 there:
https://icedtea.classpath.org/bugzilla/show_bug.cgi?id=3533

[Comment 28 Andrew John Hughes 2018-03-07 04:48:40 UTC]

If this is something that can be reproduced on Oracle's binaries, I suggest filing a bug at:

https://bugs.java.com/

But, for anyone to work on this, it needs to be reproducible and I don't see a clear process for achieving that on this bug.

Moreover, it seems that there is something odd going on with the system in question if glibc is also broken. Before releasing 3.7.0, I did a full bootstrap on multiple architectures on both RHEL and Fedora without issue.

https://koji.fedoraproject.org/koji/taskinfo?taskID=25355865

Fedora 27 is using glibc 2.26, gcc 7.2 and Linux 4.13.9. That suggests to me this may be something to do with the newer 4.15. I don't see any mention here of rolling back the kernel to a version where the JDK worked.

IIRC, 4.15 is the first version to introduce the security preventions for Meltdown & SPECTRE, which may be related.

A debug build might be useful to diagnose this further. I'll look at providing instructions for how to do that via the Gentoo build.

[Comment 29 Andrew John Hughes 2018-03-07 06:38:13 UTC]

I think it may be this:

https://bugs.openjdk.java.net/browse/JDK-8197429
http://mail.openjdk.java.net/pipermail/hotspot-dev/2018-February/030076.html

which is a recurrence of

https://bugs.openjdk.java.net/browse/JDK-8023956

due to kernel changes, as I suggested in comment #28.

I'll get it into 3.8 so you can build and test.

[Comment 30 Maciej S. Szmigiero 2018-03-07 11:42:02 UTC]

(In reply to Andrew John Hughes from comment #29)
> I think it may be this:
> 
> https://bugs.openjdk.java.net/browse/JDK-8197429
> http://mail.openjdk.java.net/pipermail/hotspot-dev/2018-February/030076.html
> 
> which is a recurrence of
> 
> https://bugs.openjdk.java.net/browse/JDK-8023956
> 
> due to kernel changes, as I suggested in comment #28.

No i'ts not this issue.

The JVM very specifically crashes on a misaligned SSE operation.
The stack address for this operation is correct, just not
16-byte aligned as required by x86 ISA.

I gave an example in icedtea bugzilla about a code path in HotSpot
that generates misaligned stack.

[Comment 31 Maciej S. Szmigiero 2018-03-07 11:50:44 UTC]

(In reply to Andrew John Hughes from comment #28)
> If this is something that can be reproduced on Oracle's binaries, I suggest
> filing a bug at:
> 
> https://bugs.java.com/

Will try Oracle binaries in a moment.

> But, for anyone to work on this, it needs to be reproducible and I don't see
> a clear process for achieving that on this bug.

This problem is simple to reproduce: just build icedtea with "icedtea-align-check.patch" patch which will immediately crash the VM with an access to 0xbad574cc ("badstacc") if it detects not 16-byte aligned stack on (at least some) C/C++ code entries from generated code.
And see how fast the build JVM crashes.

Note that 16-byte stack alignement is required by i386 psABI so every such crash
signifies a violation of this ABI (these violations now started to bite since newer GCCs do make use of this requirement to generate code with SSE instructions in things like glibc).

[Comment 32 Maciej S. Szmigiero 2018-03-07 12:19:46 UTC]

(In reply to Maciej S. Szmigiero from comment #24)
> I have also hit this issue on 32-bit x86, with gcc 7.3.0,
> CFLAGS="-O2 -march=pentium-m -pipe".
> 
> I think there are two issues at play there:
> 1) icedtea-bin and icedtea are unstable with glibc compiled with
> gcc 7.3.0,
> I don't know the reason for this but it is easily worked around by
> just recompiling glibc with gcc 6.4.0,
> These crashes happen from time to time, especially during large Java
> compiles (what is important that simple invocation of java or javac
> usually does not crash - as long as the issue number 2 is taken care
> of),

Okay, it seems that the cause for issue number 1 is... the same as for
issue 2.
It turns out not every C/C++ code entry point in icedtea was wrapped in
*_ENTRY_* or similar macros from interfaceSupport.hpp.

In fact, there are a lot of other calls generated that enter functions
not wrapped so, for example:
SafepointSynchronize::handle_polling_page_exception
JavaThread::check_special_condition_for_native_trans
JavaThread::check_special_condition_for_native_trans_and_transition
SharedRuntime::get_utf
Deoptimization::uncommon_trap
MacroAssembler::debug32
warning
MacroAssembler::print_state32
os::breakpoint
_print_CPU_state
_verify_FPU
StubRoutines::verify_oop_subroutine_entry_address
StubRoutines::x86::verify_mxcsr_entry
StubRoutines::x86::verify_fpu_cntrl_wrd_entry
StubRoutines::updateBytesCRC32
Interpreter::trace_code
BarrierSet::static_write_ref_array_post
BarrierSet::static_write_ref_array_pre
handle_unsafe_access
OptoRuntime::handle_exception_C

While it is certainly possible to find and add appropriate attributes
to every C/C++ code entry point it would be a very fragile solution,
since every new HotSpot version would need checking whether there were
new possible entry points introduced.
And missing an entry point would not generate a crash 99% of time so it
would be a really subtle bug.

That's why an other solution is necessary in form of
"-mincoming-stack-boundary=2" {C,CXX}FLAG for icedtea (not
"-mstackrealign" since it does not work in our case as this flag seems
to be for leaf - and not intermediate - functions).
It is a big hammer but it allows building of a working icedtea (with
glibc compiled by gcc-7.3.0) without any patching of the HotSpot code.

How to build a working icedtea (updated instructions):
1) Switch to gcc 6.4.0 temporarily,

2) Rebuild glibc (tried glibc-2.26-r6),
This is needed since with glibc built with gcc 7.3.0 icedtea-bin crashes
inside glibc,

3) Emerge icedtea-bin,

4) Switch back to gcc 7.3.0,

5) Patch icedtea-3.6.0 ebuild with icedtea-3.6.0.ebuild.patch,

6) Emerge icedtea-3.6.0,

7) You can now remove icedtea-bin and reemerge glibc with gcc 7.3.0.

Once such 'fixed' icedtea is built further icedtea emerges do not require
glibc to be temporarily build with gcc 6.4.0.

[Comment 33 Maciej S. Szmigiero 2018-03-07 12:20:36 UTC]

Created attachment 522650 [details, diff]
ebuild patch with -mincoming-stack-boundary=2

[Comment 34 Maciej S. Szmigiero 2018-03-07 12:55:44 UTC]

(In reply to Maciej S. Szmigiero from comment #31)
> (In reply to Andrew John Hughes from comment #28)
> > If this is something that can be reproduced on Oracle's binaries, I suggest
> > filing a bug at:
> > 
> > https://bugs.java.com/
> 
> Will try Oracle binaries in a moment.

Oracle binaries (jdk-8u162-linux-i586.tar.gz) also crash the same way.

> Moreover, it seems that there is something odd going on with the system in 
> question if glibc is also broken. 

glibc is not broken, it just happen to be the victim of a HotSpot misaligned stack in this case.
It could easily been zlib or even GTK+ if they happen to be called from JVM
and they were compiled with a compiler that generates SSE operations
(like gcc 7.3.0 with "-O2 -march=pentium-m" does).

[Comment 35 Maciej S. Szmigiero 2018-03-07 12:59:14 UTC]

(In reply to Maciej S. Szmigiero from comment #34)
> It could easily been zlib or even GTK+ if they happen to be called from JVM
> and they were compiled with a compiler that generates SSE operations
> (like gcc 7.3.0 with "-O2 -march=pentium-m" does).

And before gcc starts getting blamed for this: remember that its generated code
is correct and conforms to the i386 psABI, it is the HotSpot-generated code
non-conformance to ABI that makes the gcc-generated code crash.

[Comment 36 James Le Cuirot 2018-03-07 13:10:22 UTC]

I'm with Maciej on this one, I didn't see the issue in my chroot until I rebuilt glibc with SSE.

[Comment 37 James Le Cuirot 2018-03-08 22:02:01 UTC]

I have recreated icedtea-bin-3.7.0 with the -mincoming-stack-boundary=2 flag and confirmed that it deals with the issue. I am uploading it now so that I can finally do the 3.7.0 release. Ideally I'd wait for this discussion to conclude but I don't want to hold up the release any longer.

[Comment 38 Francis Devereux 2018-03-11 15:56:37 UTC]

Thank you, dev-java/icedtea-bin 3.7.0 fixes this for me, with that version I can compile & run HelloWorld.java successfully.

Note that I have never to my knowledge had gcc 7.x installed on this system, the latest version I've installed is 6.4.0-r1.

[Comment 39 lebkoungcity 2018-03-11 20:54:47 UTC]

Thanks a lot for all your work!

dev-java/icedtea-bin-3.7.0 fixes this issue for me too. All of my java applications run again.

[Comment 40 James Le Cuirot 2018-06-10 10:46:18 UTC]

This should be fixed by upstream in 3.8.0, which is now pushed.

[Comment 41 Maciej S. Szmigiero 2018-06-10 16:35:22 UTC]

(In reply to James Le Cuirot from comment #40)
> This should be fixed by upstream in 3.8.0, which is now pushed.

Unfortunately, it still crashes since upstream added only the
"-mstackrealign" flag, which is not enough - 
"-mincoming-stack-boundary=2" seems to be required, too.

I have posted details and traces of this issue in 3.8.0 in the
IcedTea bug.

[Comment 42 James Le Cuirot 2018-06-10 17:01:04 UTC]

(In reply to Maciej S. Szmigiero from comment #41)
> (In reply to James Le Cuirot from comment #40)
> > This should be fixed by upstream in 3.8.0, which is now pushed.
> 
> Unfortunately, it still crashes since upstream added only the
> "-mstackrealign" flag, which is not enough - 
> "-mincoming-stack-boundary=2" seems to be required, too.
> 
> I have posted details and traces of this issue in 3.8.0 in the
> IcedTea bug.

Thanks, aware of that. I had hoped that Andrew was somehow right despite your deep understanding. Let's resolve this quickly, I've already built icedtea-bin for x86 and requested stabilisation. :(

[Comment 43 Larry the Git Cow 2018-10-21 19:35:57 UTC]

The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=72896f7156538716e2d4176ea8925be73a5608f2

commit 72896f7156538716e2d4176ea8925be73a5608f2
Author:     James Le Cuirot <[e-mail]>
AuthorDate: 2018-10-21 19:34:49 +0000
Commit:     James Le Cuirot <[e-mail]>
CommitDate: 2018-10-21 19:35:44 +0000

    dev-java/icedtea-bin: Rebuild for x86 with stack alignment
    
    The other arches are unchanged.
    
    Bug: https://bugs.gentoo.org/647954
    Signed-off-by: James Le Cuirot <[e-mail]>
    Package-Manager: Portage-2.3.51, Repoman-2.3.11

 dev-java/icedtea-bin/Manifest                                     | 4 ++--
 .../{icedtea-bin-3.9.0.ebuild => icedtea-bin-3.9.0-r1.ebuild}     | 8 ++++----
 2 files changed, 6 insertions(+), 6 deletions(-)

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=3d16aaaa8d4a7b0de9316374a0c222a7d94000ea

commit 3d16aaaa8d4a7b0de9316374a0c222a7d94000ea
Author:     James Le Cuirot <[e-mail]>
AuthorDate: 2018-10-21 19:33:05 +0000
Commit:     James Le Cuirot <[e-mail]>
CommitDate: 2018-10-21 19:35:43 +0000

    dev-java/icedtea: Reintroduce stack alignment fix for x86
    
    This is still not addressed upstream.
    
    Bug: https://bugs.gentoo.org/647954
    Signed-off-by: James Le Cuirot <[e-mail]>
    Package-Manager: Portage-2.3.51, Repoman-2.3.11

 dev-java/icedtea/icedtea-3.9.0.ebuild | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

[Comment 44 James Le Cuirot 2018-10-21 19:37:13 UTC]

*** Bug 668762 has been marked as a duplicate of this bug. ***

[Comment 45 Petr F. Kartsev 2019-11-19 20:56:19 UTC]

Hi, can I comment here the same error for icedtea-bin-3.13.0 and icedtea-3.13.0
 on amd64 / gcc 9.2.0 / gentoo-sources-4.19.82 ? or should open new bug for this version? (Sorry, not sure what is approppriate)

[Comment 46 Petr F. Kartsev 2019-12-21 08:30:06 UTC]

(In reply to Petr F. Kartsev from comment #45)
> Hi, can I comment here the same error for icedtea-bin-3.13.0 and
> icedtea-3.13.0
>  on amd64 / gcc 9.2.0 / gentoo-sources-4.19.82 ? or should open new bug for
> this version? (Sorry, not sure what is approppriate)

Hi, my problem is solved. Please ignore previous message. 
It was broken JRE bundled with OmegaT starting instead of icedtea version.