Booting a hardened kernel from the 4.2.3 causes in several programs a segfault due to a bad frame in rt_sigreturn in libc-2.22.so (using sys-devel/glibc-2.22-r1). A typical error log (seen when starting /etc/init.d/net.enp3s0): is as follows: Oct 22 12:45:13 localhost kernel: openrc-run.sh[1853] bad frame in rt_sigreturn frame:000003b1e2ee5678 ip:3910eb819c0 sp:3b1e2ee5c18 orax:ffffffffffffffff in libc-2.22.so[3910ea9a000+1a1000] Oct 22 12:45:13 localhost kernel: grsec: Segmentation fault occurred at (nil) in /lib64/rc/sh/openrc-run.sh[openrc-run.sh:1853] uid/euid:0/0 gid/egid:0/0, parent /lib64/rc/sh/openrc-run.sh[openrc-run.sh:1794] uid/euid:0/0 gid/egid:0/0 Oct 22 12:45:13 localhost kernel: grsec: bruteforce prevention initiated for the next 30 minutes or until service restarted, stalling each fork 30 seconds. Please investigate the crash report for /lib64/rc/sh/openrc-run.sh[openrc-run.sh:1853] uid/euid:0/0 gid/egid:0/0, parent /lib64/rc/sh/openrc-run.sh[openrc-run.sh:1794] uid/euid:0/0 gid/egid:0/0 Oct 22 12:45:13 localhost kernel: grsec: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /lib64/rc/sh/openrc-run.sh[openrc-run.sh:1853] uid/euid:0/0 gid/egid:0/0, parent /lib64/rc/sh/openrc-run.sh[openrc-run.sh:1794] uid/euid:0/0 gid/egid:0/0 The affected process hangs afterwards and has to be killed manually. The issue has been seen so far in: /etc/init.d/net.enp3s0 (a symlink to /etc/init.d/net.lo from net-misc/netifrc-0.3.1; other scripts in /etc/init.d are unaffected) /usr/sbin/lightdm (from x11-misc/lightdm-1.15.3; but X server can be started normally from a virtual console) /usr/bin/gcc-config (from sys-devel/gcc-config-1.8) The issue does not happen when booting an old kernel (hardened-sources-4.1.7-r1) Reproducible: Always
Created attachment 415182 [details] emerge --info glibc
Created attachment 415184 [details] kernel config
if you remove this line GLIBC_PATCH_EXCLUDE+=" 00_all_0002-workaround-crash-when-handling-signals-in-static-PIE.patch" from the ebuild and rebuild glibc do it work then?
(In reply to Magnus Granberg from comment #3) > if you remove this line > GLIBC_PATCH_EXCLUDE+=" > 00_all_0002-workaround-crash-when-handling-signals-in-static-PIE.patch" > from the ebuild and rebuild glibc do it work then? I tried it, but the behaviour is still the same.
(In reply to Christian Apeltauer from comment #4) > (In reply to Magnus Granberg from comment #3) > > if you remove this line > > GLIBC_PATCH_EXCLUDE+=" > > 00_all_0002-workaround-crash-when-handling-signals-in-static-PIE.patch" > > from the ebuild and rebuild glibc do it work then? > > I tried it, but the behaviour is still the same. Bad frame sounds like virtual memory problem in the kernel most likely due to a bug in pax. If you can, proceed as follows before I assign to upstream: 1) i just added hardened-sources-4.2.4 to the tree. try it and see if you get the same bad frame error 2) if you do, then try vanilla-4.2.4 and see if you get the same frame error. this will isolate the problem to hardened-sources. also if it is hardened-sources, upload your kernel config file.
The same problem. I tried the gentoo-sources-4.2.4 and it works fine. The hardned-sources-4.2.4 fails even if CONFIG_GRKERNSEC is not set. A crash log looks like that: [ 41.612752] uptime[6002] bad frame in rt_sigreturn frame:00007ffd080887f8 ip:7ff7abb464f0 sp:7ffd08088d98 orax:ffffffffffffffff in libc-2.21.so[7ff7aba62000+19d000] [ 104.815966] chrome[7235] bad frame in rt_sigreturn frame:00007f31deb0e6b8 ip:7f31ea4d3db7 sp:7f31deb0ec78 orax:ffffffffffffffff in libc-2.21.so[7f31ea3fa000+19d000] [ 104.816071] chrome[7239] bad frame in rt_sigreturn frame:00007f31deb0e6b8 ip:7f31ea4d3db7 sp:7f31deb0ec78 orax:ffffffffffffffff in libc-2.21.so[7f31ea3fa000+19d000] [ 104.817183] traps: chrome[7239] general protection ip:7f31ea42f8c6 sp:7f31deb0c1e0 error:0 in libc-2.21.so[7f31ea3fa000+19d000] [ 104.817515] chrome[7241] bad frame in rt_sigreturn frame:00007f31de30d6b8 ip:7f31ea4d3db7 sp:7f31de30dc78 orax:ffffffffffffffff in libc-2.21.so[7f31ea3fa000+19d000] [ 104.817738] chrome[7255] bad frame in rt_sigreturn frame:00007f31de30d6b8 ip:7f31ea4d3db7 sp:7f31de30dc78 orax:ffffffffffffffff in libc-2.21.so[7f31ea3fa000+19d000] [ 104.819097] chrome[7259] bad frame in rt_sigreturn frame:00007f31ddb0c6b8 ip:7f31ea4d3db7 sp:7f31ddb0cc78 orax:ffffffffffffffff in libc-2.21.so[7f31ea3fa000+19d000] [ 104.819158] chrome[7258] bad frame in rt_sigreturn frame:00007f31ddb0c6b8 ip:7f31ea4d3db7 sp:7f31ddb0cc78 orax:ffffffffffffffff in libc-2.21.so[7f31ea3fa000+19d000] [ 104.820012] chrome[7260] bad frame in rt_sigreturn frame:00007f31ddb0c6b8 ip:7f31ea4d3db7 sp:7f31ddb0cc78 orax:ffffffffffffffff in libc-2.21.so[7f31ea3fa000+19d000] [ 104.821325] traps: chrome[7258] general protection ip:7f31ea42f8c6 sp:7f31ddb0a1e0 error:0 in libc-2.21.so[7f31ea3fa000+19d000] [ 115.312756] chrome[7300] bad frame in rt_sigreturn frame:00007f31dd30b6b8 ip:7f31ea4d3db7 sp:7f31dd30bc78 orax:ffffffffffffffff in libc-2.21.so[7f31ea3fa000+19d000] The problem reproduced for all 4.2 releases.
Created attachment 415444 [details] emerge --info glibc
Created attachment 415446 [details] working config (gentoo-sources-4.2.4)
Created attachment 415458 [details] config with crashes (hardened-sources-4.2.4)
Created attachment 415460 [details, diff] difference between working and wrong configs
(In reply to Ilya Mochalov from comment #8) > Created attachment 415446 [details] > working config (gentoo-sources-4.2.4) okay this is not hardened only. @mpagano, have you seen this elsewhere? this looks like we should bring it upstream
(In reply to Ilya Mochalov from comment #10) > Created attachment 415460 [details, diff] [details, diff] > difference between working and wrong configs this looks suspect CONFIG_PROC_PAGE_MONITOR. this is probably a vm error in the vanilla kernel, which makes it serious.
(In reply to Ilya Mochalov from comment #8) > Created attachment 415446 [details] > working config (gentoo-sources-4.2.4) did you try vanilla-sources?
(In reply to Anthony Basile from comment #13) > (In reply to Ilya Mochalov from comment #8) > > Created attachment 415446 [details] > > working config (gentoo-sources-4.2.4) > > did you try vanilla-sources? Yep, I tried vanilla-sources-4.2.4 with the same config (except for CONFIG_GENTOO_* and CONFIG_FB_CON_DECOR which are missing). It works as fine as gentoo-sources-4.2.4.
(In reply to Ilya Mochalov from comment #6) > The same problem. I tried the gentoo-sources-4.2.4 and it works fine. The > hardned-sources-4.2.4 fails even if CONFIG_GRKERNSEC is not set. A crash log > looks like that: > > The problem reproduced for all 4.2 releases. (In reply to Ilya Mochalov from comment #14) > (In reply to Anthony Basile from comment #13) > > (In reply to Ilya Mochalov from comment #8) > > > Created attachment 415446 [details] > > > working config (gentoo-sources-4.2.4) > > > > did you try vanilla-sources? > > Yep, I tried vanilla-sources-4.2.4 with the same config (except for > CONFIG_GENTOO_* and CONFIG_FB_CON_DECOR which are missing). It works as fine > as gentoo-sources-4.2.4. I'm sorry, I misunderstood. So it fails only for all *hardened* 4.2 releases. But it works for all gentoo-sources and vanilla-sources. Correct?
(In reply to Anthony Basile from comment #15) > (In reply to Ilya Mochalov from comment #6) > > The same problem. I tried the gentoo-sources-4.2.4 and it works fine. The > > hardned-sources-4.2.4 fails even if CONFIG_GRKERNSEC is not set. A crash log > > looks like that: > > > > The problem reproduced for all 4.2 releases. > > (In reply to Ilya Mochalov from comment #14) > > (In reply to Anthony Basile from comment #13) > > > (In reply to Ilya Mochalov from comment #8) > > > > Created attachment 415446 [details] > > > > working config (gentoo-sources-4.2.4) > > > > > > did you try vanilla-sources? > > > > Yep, I tried vanilla-sources-4.2.4 with the same config (except for > > CONFIG_GENTOO_* and CONFIG_FB_CON_DECOR which are missing). It works as fine > > as gentoo-sources-4.2.4. > > I'm sorry, I misunderstood. So it fails only for all *hardened* 4.2 > releases. But it works for all gentoo-sources and vanilla-sources. Correct? Yes, only hardened 4.2 releases are broken in my case. I apologize if I misled you, my English is not too good yet.
(In reply to Ilya Mochalov from comment #16) > Yes, only hardened 4.2 releases are broken in my case. > I apologize if I misled you, my English is not too good yet. you're english is fine, i just missed where you said hardened and later saw all 4.2. @mpagano, sorry for the noise @pipacs, its all yours!
I just uploaded a new test patch that includes my fix for this issue. -Brad
(In reply to Brad Spengler from comment #18) > I just uploaded a new test patch that includes my fix for this issue. > > -Brad hardened-sources-4.2.4-r1 = grsecurity-3.1-4.2.4-201510240907 its on the tree now. can people please test.
(In reply to Anthony Basile from comment #19) > (In reply to Brad Spengler from comment #18) > > I just uploaded a new test patch that includes my fix for this issue. > > > > -Brad > > hardened-sources-4.2.4-r1 = grsecurity-3.1-4.2.4-201510240907 > > its on the tree now. can people please test. I try to use 4.2.4-r1 and it looks like the problem is gone. Thank you very much! =)
(In reply to Anthony Basile from comment #19) > (In reply to Brad Spengler from comment #18) > > I just uploaded a new test patch that includes my fix for this issue. > > > > -Brad > > hardened-sources-4.2.4-r1 = grsecurity-3.1-4.2.4-201510240907 > > its on the tree now. can people please test. Unfortunately, the bug is still here but it happens much less frequently. Also a message about the error in libpthread.so appeared. [ 1921.142394] chrome[11682] bad frame in rt_sigreturn frame:00007fd640ed46b8 ip:7fd64d09adb7 sp:7fd640ed4c78 orax:ffffffffffffffff in libc-2.21.so[7fd64cfc1000+19d000] [ 2291.584869] HTMLParserThrea[12500] bad frame in rt_sigreturn frame:00007fd63e3aee38 ip:7fd64d379fa4 sp:7fd63e3af3d8 orax:ffffffffffffffff in libpthread-2.21.so[7fd64d369000+17000] [ 2293.296022] chrome[12519] bad frame in rt_sigreturn frame:00007fd6416d56b8 ip:7fd64d09adb7 sp:7fd6416d5c78 orax:ffffffffffffffff in libc-2.21.so[7fd64cfc1000+19d000] [ 2364.032521] cpu[12847] bad frame in rt_sigreturn frame:00007fff5c3b7df8 ip:7f83692ed524 sp:7fff5c3b83b0 orax:ffffffffffffffff in libc-2.21.so[7f83692b9000+19d000] [ 2364.034049] cpu[12852] bad frame in rt_sigreturn frame:00007fff5c3b7f38 ip:7f836939d4f0 sp:7fff5c3b84e8 orax:ffffffffffffffff in libc-2.21.so[7f83692b9000+19d000] [ 2364.034918] cpu[12854] bad frame in rt_sigreturn frame:00007fff5c3b7e38 ip:7f836939db90 sp:7fff5c3b83e8 orax:ffffffffffffffff in libc-2.21.so[7f83692b9000+19d000] [ 2364.035607] cpu[12856] bad frame in rt_sigreturn frame:00007fff5c3b7e38 ip:7f83692ed524 sp:7fff5c3b83e0 orax:ffffffffffffffff in libc-2.21.so[7f83692b9000+19d000] [ 2364.036317] cpu[12858] bad frame in rt_sigreturn frame:00007fff5c3b7e38 ip:7f836939db90 sp:7fff5c3b83e8 orax:ffffffffffffffff in libc-2.21.so[7f83692b9000+19d000] [ 2364.036987] cpu[12860] bad frame in rt_sigreturn frame:00007fff5c3b7e38 ip:7f83692ed524 sp:7fff5c3b83e0 orax:ffffffffffffffff in libc-2.21.so[7f83692b9000+19d000] [ 2662.242681] netstat[13323] bad frame in rt_sigreturn frame:00007fff21aa57f8 ip:7fbf22b6db90 sp:7fff21aa5da8 orax:ffffffffffffffff in libc-2.21.so[7fbf22a89000+19d000] [ 2960.903924] if_enp2s0[13832] bad frame in rt_sigreturn frame:00007ffe3f4383f8 ip:7fa7c57f5524 sp:7ffe3f4389a0 orax:ffffffffffffffff in libc-2.21.so[7fa7c57c1000+19d000]
(In reply to Ilya Mochalov from comment #21) > (In reply to Anthony Basile from comment #19) > > (In reply to Brad Spengler from comment #18) > > > I just uploaded a new test patch that includes my fix for this issue. > > > > > > -Brad > > > > hardened-sources-4.2.4-r1 = grsecurity-3.1-4.2.4-201510240907 > > > > its on the tree now. can people please test. > > Unfortunately, the bug is still here but it happens much less frequently. > Also a message about the error in libpthread.so appeared. > I removed all hardened-sources-4.2.x except 4.2.4-r1 and was about to close this bug. I guess its not fixed yet.
What specific patch did you test with? If it was with the hardened gentoo one, the date listed is from yesterday and so wouldn't contain my fix. -Brad
(In reply to Brad Spengler from comment #23) > What specific patch did you test with? If it was with the hardened gentoo > one, the date listed is from yesterday and so wouldn't contain my fix. > > -Brad I have automated downloading/applying of grsec patches, but its only once a day. So today's run was for 2015-10-24. If you put up another patch, automagic will push it out tomorrow. Watch out for hardened-sources-4.2.4-r2.
If they want they can also test the split-out fix here: https://grsecurity.net/~spender/fpu_fix.diff -Brad
Okay I think we're done. hardened-sources-4.2.4-r2 has the fix. Please test and reopen this bug if we hit the bad frames again.
Nope, the issue is not solved yet. # uname -a Linux nh 4.2.4-hardened-r2-cfg-nh-nogrsec-r1 #1 SMP PREEMPT Tue Oct 27 12:03:52 YEKT 2015 x86_64 AMD FX(tm)-8350 Eight-Core Processor AuthenticAMD GNU/Linux # dmesg | grep rt_sigreturn [ 18.372549] openrc-run.sh[4801] bad frame in rt_sigreturn frame:00007ffd85692038 ip:7fd358704b90 sp:7ffd856925e8 orax:ffffffffffffffff in libc-2.21.so[7fd358620000+19d000] [ 133.799193] chrome[7234] bad frame in rt_sigreturn frame:00007fd5ad84c6b8 ip:7fd5b9211db7 sp:7fd5ad84cc78 orax:ffffffffffffffff in libc-2.21.so[7fd5b9138000+19d000] [ 133.801475] chrome[7261] bad frame in rt_sigreturn frame:00007fd5ad04b6b8 ip:7fd5b9211db7 sp:7fd5ad04bc78 orax:ffffffffffffffff in libc-2.21.so[7fd5b9138000+19d000] [ 203.293488] chrome[7390] bad frame in rt_sigreturn frame:00007fd5ac84a6b8 ip:7fd5b9211db7 sp:7fd5ac84ac78 orax:ffffffffffffffff in libc-2.21.so[7fd5b9138000+19d000] Is there any useful info that I can add to this discussion?
What's the exact name of the patch that kernel is using? -Brad
Can you also give me your /proc/cpuinfo output?
As far as I understood, from an ebuild, the patches were used: 1003_linux-4.2.4.patch 4420_grsecurity-3.1-4.2.4-201510251836.patch 4425_grsec_remove_EI_PAX.patch 4430_grsec-remove-localversion-grsec.patch 4435_grsec-mute-warnings.patch 4440_grsec-remove-protected-paths.patch 4450_grsec-kconfig-default-gids.patch 4465_selinux-avc_audit-log-curr_ip.patch 4470_disable-compat_vdso.patch 4475_emutramp_default_on.patch
Created attachment 415588 [details] /proc/cpuinfo
Can you also show me a full dmesg with CONFIG_X86_DEBUG_FPU=y ? Thanks, -Brad
Something else to try is booting with eagerfpu=on added to the kernel command-line. -Brad
(In reply to Brad Spengler from comment #32) > Can you also show me a full dmesg with CONFIG_X86_DEBUG_FPU=y ? > > Thanks, > -Brad Yes, of course. Excuse me for delay.
Created attachment 415648 [details] dmesg output (CONFIG_X86_DEBUG_FPU=y)
Created attachment 415650 [details] dmesg output (CONFIG_X86_DEBUG_FPU=y, eagerfpu=on)
No one error during two days of testing. I think hardened-sources-4.2.5 resolves the issue. Thank you very much! =)
(In reply to Ilya Mochalov from comment #37) > No one error during two days of testing. I think hardened-sources-4.2.5 > resolves the issue. Thank you very much! =) Please keep testing 4.2.5 because it is the next stabization canditate.
(In reply to Anthony Basile from comment #38) > (In reply to Ilya Mochalov from comment #37) > > No one error during two days of testing. I think hardened-sources-4.2.5 > > resolves the issue. Thank you very much! =) > > Please keep testing 4.2.5 because it is the next stabization canditate. Okay I will write if I see any anomalies.