Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 201329 - hardened-sources-2.6.23-any revision kernel SYSSEGV's /sbin/{rc,runscript} during boot....
Summary: hardened-sources-2.6.23-any revision kernel SYSSEGV's /sbin/{rc,runscript} du...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Core system (show other bugs)
Hardware: AMD64 Linux
: High normal
Assignee: The Gentoo Linux Hardened Team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2007-12-05 06:58 UTC by brankob
Modified: 2008-02-22 21:48 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
emerge --info output (emerge_info.txt,4.21 KB, text/plain)
2007-12-05 07:00 UTC, brankob
Details
ouptut of "strace -o logfile.txt /sbin/rc default" (stracelog.txt,10.44 KB, text/plain)
2007-12-05 07:03 UTC, brankob
Details
config of hardened-sources-2.6.23-r5 (.config,52.75 KB, text/plain)
2007-12-27 16:26 UTC, brankob
Details

Note You need to log in before you can comment on or make changes to this bug.
Description brankob 2007-12-05 06:58:50 UTC
System : 
CPU:	Dual Opteron 240 
Board: 	TYAN S2895 with BIOS 1.05
RAM:	6GB ( 4x 1GiB + 4x 512 MiB)

cards:
	GF7600 GSin PCIe slot 
2x	NIC with Broadcom BCM5704 in PCI-X slots
1x	NIC with Realtek 8169 in PCI slot
1x	Areca HW RAID ARC-1160 in PCI-X slot with 16 disks connected to the card
1x	SATA disk, connected directly to the board, from which I boot and where gentoo system is
1x	DVD-RW

Gentoo: /etc/make.profile« -> »/usr/portage/profiles/selinux/2007.0/amd64/hardened
kernel: 2.6.23-hardened-r3 ( error happens with any other rev also )

System is updated regularly.

Error description:

Booting with 2.6.20* kernel works fine.

Booting with 2.6.23* works up to a point when init should start /sbin/rc and then /sbin/rc fails.

System is stuck at a point of a prompt and I can't login, since nothing is done in the background ( fs in /etc/fstab are not mounted, no service is started etc).

I tried to start the system with kernel parameter "init=/bin/bash" and to check things manually and it worked.

Bash started normally. I could mount filesestems manually etc, but when i tried to start /sbin/rc, it crashed.
Since rc is simply a script that is interpreted by runscript, I have tried recompiling everything related ( baselayout, glibc, policycoreutils etc ) but result was exactly the same.

Then I straced the /sbin/rc and it seems that everything works fine up to a point when runscript repeatedly mmaps and unmaps a small block, always of the same size ( so it always gets te same address). On one of the attempts it gets SIGSEGV when trying to unmap the block it just successfully mmapped.

It seems as some kind of race condition in multisocket NUMA system.

Practically same SW configuration works fine on unisocket dual-core X2 6000+ system ( hardened-sources-2.6.23-r1)

I will append in subequent post the output of "strace -o stracelog.txt /sbin/rc default"  and "emerge --info"...

Reproducible: Always
Comment 1 brankob 2007-12-05 07:00:47 UTC
Created attachment 137775 [details]
emerge --info output
Comment 2 brankob 2007-12-05 07:03:55 UTC
Created attachment 137777 [details]
ouptut of "strace -o logfile.txt /sbin/rc default"
Comment 3 brankob 2007-12-06 04:26:52 UTC
I have just tried gentoo-sources-2.6.23-r3. It works. 
Comment 4 Christian Heim (RETIRED) gentoo-dev 2007-12-27 15:23:43 UTC
I need your /proc/config.gz (or .config) of hardened-sources.
Comment 5 brankob 2007-12-27 16:26:20 UTC
Created attachment 139434 [details]
config of hardened-sources-2.6.23-r5

here is the config of the latest hardened-sources-2.6.23 i have tried ( -r5), with same results as with any previous attempt...
Comment 6 Christian Heim (RETIRED) gentoo-dev 2007-12-28 18:02:11 UTC
Maybe pipacs has an idea. I certainly don't.
Comment 7 Noah Jaehnert 2007-12-28 22:06:20 UTC
I am having the same issues as the previous posters. 

In order to resolve the issue, I switched to gcc-vanilla (using gcc-config), ran source /etc/profile and then recompiled the latest kernel (2.6.23-hardened-r4).

Lo and behold, upon booting, no issues whatsoever.

-Noah Jaehnert
Comment 8 PaX Team 2007-12-30 14:13:11 UTC
(In reply to comment #0)
> Then I straced the /sbin/rc and it seems that everything works fine up to a
> point when runscript repeatedly mmaps and unmaps a small block, always of the
> same size ( so it always gets te same address). On one of the attempts it gets
> SIGSEGV when trying to unmap the block it just successfully mmapped.

you could gather more information by running rc inside gdb:

1. in the shell: gdb /sbin/rc
2. in gdb: run default

this will run rc under the control of gdb and will cause gdb to break in at the time the segfault happens. then you can issue further commands (for best results you should have debug info stored, not stripped):

1. x/8i $pc
2. x/8x $sp
3. info reg
4. bt

you can also suspend gdb and then get the /proc/<pid of rc>/maps file which will be helpful to see the memory layout of rc.
Comment 9 brankob 2008-02-22 21:48:06 UTC
(In reply to comment #8)
> (In reply to comment #0)

Sorry, I didn't have the time to check that.

In any case, after several rounds worth of system updates it works now.

I have no idea what was the cause ( kernel, baselayout, sme lib or something else )...