Bug 589864 – dev-java/icedtea{,-bin}:{7,8} with FEATURES="userpriv usersandbox": A fatal error has been detected by the Java Runtime Environment: SIGSEGV Problematic frame: C [libsandbox.so+0x3161]# [ timer expired, abort... ]

• Status: RESOLVED UPSTREAM
• Product: Gentoo Linux

[Description Andrew Savchenko 2016-07-27 18:47:43 UTC]

Created attachment 441746 [details]
config.log

Hello,

javac from icedtea:{7,8} _and_ icedtea-bin:{7,8} segfault when usersandbox and userpriv are enabled.

please note that this bug affects any package using javac with FEATURES="userpriv u7.2.6.6-r1sersandbox" during their build, not just icedtea{,-web} themselves. Error messages below are just examples.

With icedtea-bin-7.2.6.6-r1 as a system java-vm during build of dev-java/icedtea-web-1.6.2 I have the following:

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0xa76c4161, pid=8941, tid=2792061760
#
# JRE version: OpenJDK Runtime Environment (7.0_101) (build 1.7.0_101-b00)
# Java VM: OpenJDK Client VM (24.95-b01 mixed mode linux-x86 )
# Derivative: IcedTea 2.6.6
# Distribution: NAME=Gentoo, package Gentoo icedtea-7.2.6.6
# Problematic frame:
# C  [libsandbox.so+0x3161]# [ timer expired, abort... ]

On all other dev-java/icedtea{,-bin}:{7,8} versions I have the same problem in libsandbox.so.

With FEATURES="-usersandbox" the problem can be workarounded.

icedtea-6 (already not in the tree, but still on my host) works fine.

Possibly related issue is described here:
https://stackoverflow.com/questions/20427361/jvm-crashes-with-no-frame-specified-only-timer-expired-abort

[Comment 1 Andrew Savchenko 2016-07-27 18:48:19 UTC]

Created attachment 441748 [details]
build.log

[Comment 2 Andrew Savchenko 2016-07-27 18:48:50 UTC]

Created attachment 441750 [details]
environment

[Comment 3 Andrew Savchenko 2016-07-27 18:49:24 UTC]

Created attachment 441752 [details]
emerge --info

[Comment 4 James Le Cuirot 2016-07-27 20:05:34 UTC]

Something must be up with your system because I've had userpriv and usersandbox enabled for ages, plus those are enabled by default, so surely almost every Gentoo Java user would have complained about this by now.

[Comment 5 Andrew Savchenko 2016-07-27 21:10:44 UTC]

(In reply to James Le Cuirot from comment #4)
> Something must be up with your system because I've had userpriv and
> usersandbox enabled for ages, plus those are enabled by default, so surely
> almost every Gentoo Java user would have complained about this by now.

1. If my system is broken, why icedtea-6 works fine then? Also I tried binary packages for a reason: to be sure, that this isn't my icedtea builds guilty.

2. This issue seems to be x86-related, I can't reproduce it on amd64 hosts. And it is likely that number of x86 users is much less then amd64 ones.

3. Maybe this has something to do with hard-coded 2-minute timeout in java (see URL in the first post), CPU on this host is quite old.

[Comment 6 James Le Cuirot 2016-07-27 21:54:07 UTC]

(In reply to Andrew Savchenko from comment #5)
> (In reply to James Le Cuirot from comment #4)
> > Something must be up with your system because I've had userpriv and
> > usersandbox enabled for ages, plus those are enabled by default, so surely
> > almost every Gentoo Java user would have complained about this by now.
> 
> 1. If my system is broken, why icedtea-6 works fine then? Also I tried
> binary packages for a reason: to be sure, that this isn't my icedtea builds
> guilty.

I appreciate that but there are more than a few differences between the major versions. I had major problems with icedtea:7 on ppc64 that were never resolved. On the one hand, icedtea:8 worked fine, but on the other hand, upstream never managed to reproduce the issue on the same architecture.

> 2. This issue seems to be x86-related, I can't reproduce it on amd64 hosts.
> And it is likely that number of x86 users is much less then amd64 ones.

I keep a generic x86 chroot for preparing the icedtea-bin builds and I've not been able to reproduce it there. I did notice that you're on an Atom so I tried to fire up that chroot via proot+qemu but it's not playing ball. I've done this with ARM in the past but I recall x86 userspace emulation not working so well. I do have a real Atom N270 here but it's running Fedora. I'd have to NFS mount or copy chroot and I don't have time to do that right now.

> 3. Maybe this has something to do with hard-coded 2-minute timeout in java
> (see URL in the first post), CPU on this host is quite old.

If that is happening, it's only masking a different underlying issue. The timeout is for generating the crash report.

This is a shot in the dark but is /proc/sys/kernel/randomize_va_space set to anything other than 2?

[Comment 7 Andrew Savchenko 2016-07-27 22:31:40 UTC]

(In reply to James Le Cuirot from comment #6)
> This is a shot in the dark but is /proc/sys/kernel/randomize_va_space set to
> anything other than 2?

No, it is 2:

$ cat /proc/sys/kernel/randomize_va_space
2

Speaking of uncommon features, I have Yama security framework enabled:
$ cat /proc/sys/kernel/yama/ptrace_scope 
1

But disabling it (set to 0) doesn't help.

[Comment 8 Andrew Savchenko 2016-07-27 22:33:06 UTC]

Created attachment 441770 [details]
config-4.6.4-hitomi.xz

My kernel config just in case you will need to check more options.

[Comment 9 Andrew Savchenko 2016-07-27 23:24:52 UTC]

Created attachment 441782 [details]
hello_world.java

I can reproduce this issue with simple hello world java app.

Without sandbox:

$ javac hello_world.java
$ java HelloWorld
Hello World!

With sandbox:

$ sandbox
============================= Gentoo path sandbox ==============================
Detection of the support files.
Verification of the required files.
Setting up the required environment variables.
The protected environment has been started.
--------------------------------------------------------------------------------
Process being started in forked instance.
$ javac hello_world.java
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (os_linux_x86.cpp:291), pid=1177, tid=2788539200
#  fatal error: An irrecoverable SI_KERNEL SIGSEGV has occurred due to unstable signal handling in this distribution.
#
# JRE version: OpenJDK Runtime Environment (8.0_91-b14) (build 1.8.0_91-b14)
# Java VM: OpenJDK Server VM (25.91-b14 mixed mode linux-x86 )
# Derivative: IcedTea 3.0.1
# Distribution: NAME=Gentoo, package Gentoo icedtea-3.0.1
# Core dump written. Default location: /home/andrew/prog/java/core or core.1177

[Comment 10 Andrew Savchenko 2016-07-27 23:26:58 UTC]

Here is backtrace from core dump (active vm: icedtea-bin-3.0.1):

Core was generated by `/opt/icedtea-bin-3.0.1/bin/javac hello_world.java'.
Program terminated with signal SIGABRT, Aborted.
#0  0xa778bbd1 in __kernel_vsyscall ()
[Current thread is 1 (Thread 0xa752e980 (LWP 1177))]
(gdb) bt
#0  0xa778bbd1 in __kernel_vsyscall ()
#1  0xa75960c9 in raise () from /lib/libc.so.6
#2  0xa75977e4 in abort () from /lib/libc.so.6
#3  0xa6cfe199 in ?? () from /opt/icedtea-bin-3.0.1/jre/lib/i386/server/libjvm.so
#4  <signal handler called>
#5  0xa778bbcf in __kernel_vsyscall ()
#6  0xa753665b in pthread_join () from /lib/libpthread.so.0
#7  0xa7714e72 in ?? () from /opt/icedtea-bin-3.0.1/bin/../lib/i386/jli/libjli.so
#8  0xa7710fb9 in ?? () from /opt/icedtea-bin-3.0.1/bin/../lib/i386/jli/libjli.so
#9  0xa7714f7b in ?? () from /opt/icedtea-bin-3.0.1/bin/../lib/i386/jli/libjli.so
#10 0xa771233d in JLI_Launch () from /opt/icedtea-bin-3.0.1/bin/../lib/i386/jli/libjli.so
#11 0x080484c6 in ?? ()
#12 0xa758157e in __libc_start_main () from /lib/libc.so.6
#13 0x080484ec in ?? ()

[Comment 11 Andrew Savchenko 2016-07-27 23:36:50 UTC]

icedtea-bin-7.2.6.6-r1 gives somewhat different error message and backtrace (for the same test file and commands):

$ javac hello_world.java
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0xa76bc161, pid=15986, tid=2792061760
#
# JRE version: OpenJDK Runtime Environment (7.0_101) (build 1.7.0_101-b00)
# Java VM: OpenJDK Client VM (24.95-b01 mixed mode linux-x86 )
# Derivative: IcedTea 2.6.6
# Distribution: NAME=Gentoo, package Gentoo icedtea-7.2.6.6
# Problematic frame:
# C  [libsandbox.so+0x3161]# [ timer expired, abort... ]
Aborted (core dumped)


Core was generated by `/opt/icedtea-bin-7.2.6.6/bin/javac hello_world.java'.
Program terminated with signal SIGABRT, Aborted.
#0  0xa7727bd1 in __kernel_vsyscall ()
[Current thread is 1 (Thread 0x7f82fb40 (LWP 15997))]
(gdb) bt
#0  0xa7727bd1 in __kernel_vsyscall ()
#1  0xa75110c9 in raise () from /lib/libc.so.6
#2  0xa75127e4 in abort () from /lib/libc.so.6
#3  0xa6d24704 in ?? () from /opt/icedtea-bin-7.2.6.6/jre/lib/i386/client/libjvm.so
#4  0xa6df1ada in ?? () from /opt/icedtea-bin-7.2.6.6/jre/lib/i386/client/libjvm.so
#5  0xa6d23687 in ?? () from /opt/icedtea-bin-7.2.6.6/jre/lib/i386/client/libjvm.so
#6  0xa76a33eb in start_thread () from /lib/libpthread.so.0
#7  0xa75d299e in clone () from /lib/libc.so.6

[Comment 12 Andrew Savchenko 2016-07-27 23:45:33 UTC]

Test files and core dumps are here:
ftp://brunestud.info/java/

[Comment 13 James Le Cuirot 2016-07-28 07:33:25 UTC]

A good test to rule some things out would be to try oracle-jdk-bin. Sorry if that goes against your principles. ;) It is largely built from the same sources though.

[Comment 14 Andrew Savchenko 2016-07-28 16:31:05 UTC]

(In reply to James Le Cuirot from comment #13)
> A good test to rule some things out would be to try oracle-jdk-bin. Sorry if
> that goes against your principles. ;) It is largely built from the same
> sources though.

Ah, there is no need for that.

The reason is in old gcc bug which is alive again: sse + 32-bit = broken code.
See bug 270120 and upstream bugs:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=40838
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=41156
The last time I hit this bug was gcc-4.5, but now it is back again in gcc-5.4 :(

I recompiled sys-apps/sandbox-2.10-r2 with following:
  CFLAGS="-march=native -O3"
Bug is here, javac fails in the sandbox.

With
  CFLAGS="-march=native -O2"
it works.

Both march and -O2 are complex flags, so I dissected this problem further, minimal set of flags to reproduce the bug:
  CFLAGS="-msse2 -m32 -O2 -ftree-loop-vectorize"

As can be seen from gcc bug 40838, -ftree-loop-vectorize is not a bug source here, but it triggers the bug by mixing 16-byte SSE aligned and 4-byte code.

This issue can be workarounded by adding -mstackrealign to CFLAGS.

While it is possible to fix the bug for this exact package by removing -ftree-loop-vectorize, any SSE code is unsafe on 32-bit without -mstackrealign :/

[Comment 15 James Le Cuirot 2016-07-28 16:43:46 UTC]

Thanks for the thorough investigation. Is there anything for Java team to do here then? When you say "this exact package", I'm not sure if you mean sandbox or icedtea. Not that I'd want to do anything as I'd ideally like this addressed in gcc.

Your many flags did raise an eyebrow but I figured, being a dev, that you knew what you were doing. Indeed these are not to blame if -O3 is enough to trigger it.

[Comment 16 Andrew Savchenko 2016-07-28 17:13:37 UTC]

(In reply to James Le Cuirot from comment #15)
> Thanks for the thorough investigation. Is there anything for Java team to do
> here then?

No, java was only a trigger to the problem, not a cause.

> When you say "this exact package", I'm not sure if you mean
> sandbox or icedtea.

I mean sandbox.

> Not that I'd want to do anything as I'd ideally like
> this addressed in gcc.

You are right. I'm adding toolchain to this, maybe they will consider adding -mstackrealign to x86 profile if SSE is enabled (though this flag should be filtered out for at least gcc itself and unwind library).

> Your many flags did raise an eyebrow but I figured, being a dev, that you
> knew what you were doing. Indeed these are not to blame if -O3 is enough to
> trigger it.

Most of these flags are actually -O3 expansion. I just removed several from the list where code size grow is too much for this little CPU's cache. Though the list is scary, I agree :)

[Comment 17 James Le Cuirot 2016-08-01 13:25:30 UTC]

Sorry, I guess you want to keep this open. Reassigning to toolchain.

[Comment 18 Sergei Trofimovich (RETIRED) 2019-08-19 23:28:15 UTC]

toolchain@ will follow upstream in stack realignment decision.