With new kernel > 2.6.25, my macbook doesn't boot at all. I just have a black screen without any error message just after grub. Reproducible: Always
Created attachment 165630 [details] config
Which kernel version, specifically, are you trying to boot? What was the last working kernel? Post your emerge --info, please. May want to try a netconsole or serial console and see if you can get any output from that.
I try sys-kernel/hardened-sources-2.6.25-r4 and sys-kernel/hardened-sources-2.6.25-r5 Last version that works, is sys-kernel/hardened-sources-2.6.24-r3. emerge --info: Portage 2.1.4.4 (hardened/x86/2.6, gcc-3.4.6, glibc-2.6.1-r0, 2.6.24-hardened-r3-mactel i686) ================================================================= System uname: 2.6.24-hardened-r3-mactel i686 Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz Timestamp of tree: Tue, 16 Sep 2008 17:45:01 +0000 ccache version 2.4 [enabled] app-shells/bash: 3.2_p33 dev-java/java-config: 1.3.7, 2.1.6 dev-lang/python: 2.5.2-r7 dev-util/ccache: 2.4-r7 sys-apps/baselayout: 1.12.11.1 sys-apps/sandbox: 1.2.18.1-r2 sys-devel/autoconf: 2.13, 2.61-r2 sys-devel/automake: 1.5, 1.7.9-r1, 1.9.6-r2, 1.10.1 sys-devel/binutils: 2.18-r3 sys-devel/gcc-config: 1.4.0-r4 sys-devel/libtool: 1.5.26 virtual/os-headers: 2.6.23-r3 ACCEPT_KEYWORDS="x86" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -march=nocona -fomit-frame-pointer -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/revdep-rebuild /etc/terminfo /etc/udev/rules.d" CXXFLAGS="-O2 -march=nocona -fomit-frame-pointer -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="ccache collision-protect distcc distlocks metadata-transfer parallel-fetch sandbox sfperms strict unmerge-orphans userfetch userpriv usersandbox" GENTOO_MIRRORS="http://gentoo.tiscali.nl/ http://mirror.bytemark.co.uk/gentoo/ http://ftp.heanet.ie/pub/gentoo/" LANG="en_US.UTF-8" LC_ALL="en_US.UTF-8" LDFLAGS="-Wl,--as-needed" MAKEOPTS="-j2" PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/portage/local/layman/pythonhead /usr/portage/local/layman/sunrise /usr/portage/local/layman/armagetron /usr/portage/local/layman/tryton /usr/local/portage" SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage" USE="X acl alsa avahi bash-completion bzip2 cdr cracklib crypt cups dbus dri dvd dvdread gtk hardened ipv6 jpeg lirc logrotate midi mmx motif mpeg nls nptl nptlonly opengl pam pic png readline sdl sse sse2 ssl svg threads truetype unicode urandom v4l2 vim-syntax win32codecs x86 xattr xcb xinerama xorg xscreensaver xulrunner zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="keyboard mouse synaptics evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIRC_DEVICES="devinput" USERLAND="GNU" VIDEO_CARDS="i810" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LINGUAS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Could you please try your config (sans GRSEC/PAX options) with sys-kernel/gentoo-sources-2.6.25-r8?
I have the same issue. Always reproducible (PAX enabled or disabled). I have run oldconfig on my config from 2.6.25-gentoo-r7 and then enabled the PAX options (no grsec) and my kernel enters grub, then immediately reboots. I first thought this may be a regression, and compiled the kernel as monolithic (no initrd, no modules), and the error still persists (immediate reboot after grub loads). I repeated these steps with PAX disabled on the same kernel. I doubt this helps much except to confirm the behaviour. If there are further steps you'd like me to take to provide more info, just let me know. This is on a Macbook2,1 C2D
(In reply to comment #5) > I have the same issue. Always reproducible (PAX enabled or disabled). > > I have run oldconfig on my config from 2.6.25-gentoo-r7 and then enabled the > PAX options (no grsec) and my kernel enters grub, then immediately reboots. > I first thought this may be a regression, and compiled the kernel as monolithic > (no initrd, no modules), and the error still persists (immediate reboot after > grub loads). I repeated these steps with PAX disabled on the same kernel. > > I doubt this helps much except to confirm the behaviour. If there are further > steps you'd like me to take to provide more info, just let me know. > > This is on a Macbook2,1 C2D > Yes, there is. Which hardened kernel version did you try? What were the results with PaX disabled? Please post your emerge --info as well.
The kernel fails to boot with PAX enabled or disabled. In addition, I am running the amd64 port of gentoo. If this checks out, seems that the severity of this bug should be bumped. emerge --info: Portage 2.1.4.4 (default/linux/amd64/2008.0, gcc-4.1.2, glibc-2.6.1-r0, 2.6.27-rc8 x86_64) ================================================================= System uname: 2.6.27-rc8 x86_64 Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz Timestamp of tree: Thu, 02 Oct 2008 17:45:04 +0000 app-shells/bash: 3.2_p33 dev-java/java-config: 1.3.7, 2.1.6 dev-lang/python: 2.4.4-r13, 2.5.2-r7 dev-python/pycrypto: 2.0.1-r6 sys-apps/baselayout: 1.12.11.1 sys-apps/sandbox: 1.2.18.1-r2 sys-devel/autoconf: 2.13, 2.61-r2 sys-devel/automake: 1.5, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10.1-r1 sys-devel/binutils: 2.18-r3 sys-devel/gcc-config: 1.4.0-r4 sys-devel/libtool: 1.5.26 virtual/os-headers: 2.6.23-r3 ACCEPT_KEYWORDS="amd64" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -march=nocona -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/terminfo /etc/texmf/web2c /etc/udev/rules.d" CXXFLAGS="-O2 -march=nocona -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="distlocks metadata-transfer sandbox sfperms strict unmerge-orphans userfetch" GENTOO_MIRRORS="http://gentoo.osuosl.org/ http://www.gtlib.gatech.edu/pub/gentoo http://mirror.fslutd.org/linux/distributions/gentoo/ " LANG="C" LDFLAGS="-Wl,-O1" MAKEOPTS="-j3" PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage" USE="X a52 aac aalib acl acpi alsa amd64 apache2 bash-completion berkdb bluetooth bzip2 cdr cli cracklib crypt cups dbus device-mapper divx dri dts dv dvd dvdr dvdread dynamicplugin emacs encode enscript erandom fastcgi fbcon ffmpeg firefox flac fortran fuse gcrypt gd gdbm gif gnome gnome-keyring gnutls gocr gpg gpgme gphoto2 gpm grub gs gstreamer gtk gzip hal hddtemp iceweasel iconv imagemagick imlib innodb ipv6 isdnlog isight java6 jpeg jpeg2k keyscrub kvm laptop libgcrypt libssh2 logrotate lvm midi mime mjpeg mmap mmx mp2 mp3 mp4 mpeg mplayer mudflap multilib mysql nautilus ncurses nls nptl nptlonly ogg opengl openmp openssl opensslcrypt pam pcre perl php png pppd python quicktime rar readline reflection rtc samba sdl sensord session skins socks5 spl sqlite sqlite3 sse sse2 sse3 ssl ssse3 subversion svg svgz sysfs tcpd theora threads truetype type1 unicode usb v4l v4l2 vorbis xml xorg xv xvid zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias proxy" APACHE2_MPMS="worker" ELIBC="glibc" INPUT_DEVICES="keyboard mouse synaptics evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" USERLAND="GNU" VIDEO_CARDS="i810 vesa" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PORTDIR_OVERLAY
(In reply to comment #7) > The kernel fails to boot with PAX enabled or disabled. Which kernel version, specifically? > > In addition, I am running the amd64 port of gentoo. > > If this checks out, seems that the severity of this bug should be bumped. > From your comment #5, I want to confirm gentoo-sources-2.6.25-r7 works ok for you?
> Which kernel version, specifically? hardened-sources-2.6.25-r7 hardened-sources-2.6.26-r1 hardened-sources-2.6.25-r8 hardened-sources-2.6.26-r2 Yes, 2.6.25-gentoo-r7 works fine without any problems, but using its config to preconfigure the kernel fails to produce a bootable kernel. It also fails if I manually configure the kernel for my hardware from scratch.
it could be related to EFI support that i rewrote long ago in PaX but could never test it myself. can you tell me what the last version was that worked with EFI? all of .24 but none ever since? could you guys also test the latest .27 PaX test patch please?
Ok. So the last 'working' kernel was in the .24 series, but it doesn't support the entirety of my hardware, thus isn't a viable option for me. I am currently running 2.6.27-gentoo-r2 with pax patch 'pax-linux-2.6.27.4-test10.patch' applied and running OK so far for 48 hours no crashes. Let me know if you need more info.
That's great news. If you have the time, I'd love to figure out what fixed it and do any necessary backporting. So if you have the desire to give one of the hardened-sources-2.6.26 releases a go please let us know how it goes.
(In reply to comment #11) > Let me know if you need more info. do you by any chance enable EFI in your .config? if yes then the early crash problems should be fixed indeed, otherwise i don't really know what caused it for you in earlier kernels except that it got fixed.
Just noting sys-kernel/hardened-sources-2.6.27 is in the tree. But it's not going to be marked stable for quite awhile, so I'm still interested in fixing up 2.6.25 and 2.6.26.
After upgrading from 2.6.26-hardened-r3 to the 27-r1 version, I've encountered a problem that may be related. The system reboots right after the kernel decompression output. I suspected that my initramfs might be the problem, so I spent half an hour recompiling/updating its busybox. When that didn't help, I tried it without an initramfs, which booted without issue (other than the fact that I need the initramfs to get at the root filesystem). I then grabbed the equivalent vanilla-sources version (2.6.27.5), did make oldconfig with the 27-r1's .config, and presto, the old initramfs works fine again. Not sure if this is related since submitter does not mention initramfs, if it's not I'll open a new bug, but in any case hardened-sources-2.6.27-r1 breaks initramfs for me.
(In reply to comment #15) > The system reboots right after the kernel decompression output. what's your binutils version? do you use EFI? > I suspected > that my initramfs might be the problem, so I spent half an hour > recompiling/updating its busybox. When that didn't help, I tried it without an > initramfs, which booted without issue (other than the fact that I need the > initramfs to get at the root filesystem). can you post the readelf -e output of both kernels (i.e., with and without initramfs)?
Following your binutils suggestion, I rebuilt the kernel with several binutils versions: kern binutils initramfs result hardened 26-r3 2.18-r4 grub initrd OK hardened 27-r1 2.19 grub initrd reboot hardened 27-r1 2.19 none reboot hardened 27-r1 2.18-r4 grub initrd reboot hardened 27-r1 2.18-r4 none reboot hardened 27-r1 2.18-r4 baked in reboot vanilla 2.6.27.5 2.19 grub initrd OK It looks like I was wrong about this being related with initramfs, since I can't reproduce 27-r1 booting even without without initramfs. Also, reverting binutils to the version that worked with the previous kernel makes no difference. Looks like a bug in the hardened patches... I'll attach readelf -e for the 2.19 built 27-r1 kernel.
Created attachment 171427 [details] # readelf -e /usr/src/linux-2.6.27-hardened-r1/vmlinux
(In reply to comment #18) > Created an attachment (id=171427) [edit] > # readelf -e /usr/src/linux-2.6.27-hardened-r1/vmlinux the size of the init.ramfs section doesn't quite look normal (0x85 bytes only), what is it on a working kernel? also can you try the latest PaX test patch alone (just to confirm it's due to PaX and not something else)?
From "readelf -e /usr/src/linux-2.6.26-hardened-r3/vmlinux": [33] .init.ramfs PROGBITS ffffffff80885000 00a85000 0000000000000086 0000000000000000 A 0 0 1 On these kernels, the initramfs isn't built into them but is loaded by grub with the initrd option. I've tried patching vanilla 2.6.27.5 with pax-linux-2.6.27.5-test13.patch, but I get a build error. After some missing initializer warnings in scripts/genksyms/keywords.gperf and scripts/mod/modpost.c, there's: CC arch/x86/kernel/process_64.o arch/x86/kernel/process_64.c: In function 'get_wchan': arch/x86/kernel/process_64.c:733: error: expected expression before 'u64' arch/x86/kernel/process_64.c:737: error: expected expression before 'u64' make[1]: *** [arch/x86/kernel/process_64.o] Error 1 make: *** [arch/x86/kernel] Error 2 grsecurity-2.1.12-2.6.27.5-200811071900.patch applies and builds OK ("make oldconfig" on the linux-2.6.27-hardened-r1's .config). Booting it gives exactly the same result as with the 2.6.27-hardened-r1 kernel. "Decompressing linux..." and then a reboot. (Vanilla 2.6.27.5 without the grsec patch boots fine.) I've gone though another "make oldconfig" run 2.6.26-hardened-r3 -> 2.6.27-hardened-r1 but nothing stands out as a possible cause to me. I've also reproduced the problem on a completely different amd64 machine (different cpu, memory, hardware..)
(In reply to comment #20) > From "readelf -e /usr/src/linux-2.6.26-hardened-r3/vmlinux": > [33] .init.ramfs PROGBITS ffffffff80885000 00a85000 > 0000000000000086 0000000000000000 A 0 0 1 > On these kernels, the initramfs isn't built into them but is loaded by grub > with the initrd option. ah, i see. for an experiment, could you build in the initramfs? > I've tried patching vanilla 2.6.27.5 with pax-linux-2.6.27.5-test13.patch, but > I get a build error. oops, sorry, try test15. > I've also reproduced the problem on a completely different amd64 machine > (different cpu, memory, hardware..) ok, it's probably some PaX changes but i don't yet see what would cause such an early crash (it works fine with a built-in initramfs here).
(In reply to comment #21) > ah, i see. for an experiment, could you build in the initramfs? I did, as per comment #17, it made no difference. > oops, sorry, try test15. It's showing up as an empty file on: http://www.grsecurity.net/~paxguy1/?M=D > ok, it's probably some PaX changes but i don't yet see what would cause such an > early crash (it works fine with a built-in initramfs here). Is there some kernel option I can set to get some debug output? On *BSD I'd be seeing a nice little ddb> prompt.. Thanks for your help.
(In reply to comment #22) > > oops, sorry, try test15. > It's showing up as an empty file on: fixed now, we ran out of diskspace ;). > Is there some kernel option I can set to get some debug output? On *BSD I'd be > seeing a nice little ddb> prompt.. Thanks for your help. you should see an oops or early exception report (try earlyprintk=vga maybe) not a reboot, it indicates some very fundamental problem that makes the cpu triple fault or something similar. if you're up to some debugging, you can try to insert a 1: jmp 1b line into arch/x86/kernel/head_64.S and by moving it around you can do a binary search to determine at which point it stops rebooting - the next line of code is the culprit (or at least where the problem manifests). if head_64.S turns out to be clean (you can do a quick test by placing the above infinite loop at just before the lretq line) then you can insert a while(1); line into .c code and continue with this approach, but i reckon, it's quite tedious... another approach could be if your kernel produced the same problem in qemu, if you can try that and it does crash there as well, then please send it to me (i'll need both bzImage/vmlinux) and i'll debug it myself.
(In reply to comment #23) > fixed now, we ran out of diskspace ;). Okay, I've just tried -test15 and it does not exhibit the problem. I guess that means it's due to the grsec part of the patch? > search to determine at which point it stops rebooting - the next line of code > is the culprit (or at least where the problem manifests). if head_64.S turns > out to be clean (you can do a quick test by placing the above infinite loop at > just before the lretq line) Inserting the infinite loop before lretq leads to livelock, no reboot, so I guess the cause is in C code somewhere. As for looking for the cause in C code.. Well let's just say that "the source" is stronger with you than it is with me ;) > another approach could be if your kernel produced the same problem in qemu, if > you can try that and it does crash there as well, then please send it to me > (i'll need both bzImage/vmlinux) and i'll debug it myself. I can reproduce it with qemu as well, it fails with "qemu: fatal: triple fault" and then dumps the registers. (Command "qemu-system-x86_64 -kernel /tmp/bzImage-2.6.27.5-grsec /tmp/diskimg-1M-zero"). I've got a tbz2 with the kernel binaries and the .config. Of the 2.6.27.5 kernel with the grsecurity-2.1.12-2.6.27.5-200811071900.patch, that is. Looks like gentoo's bugzilla hasn't yet clued in to the fact that 1M of storage = 0.1 cents. I'll send it to you by mail. Thanks.
OK, bug was caused by changes in early init ordering made to .27. It's triggered only when PREEMPT is enabled. Code fix is in pax-linux-2.6.27.5-test16.patch; # interdiff -U0 pax-linux-2.6.27.5-test1[56]* only in patch2: --- linux-2.6.27.5/arch/x86/kernel/head64.c 2008-10-10 09:18:47.000000000 +0200 +++ linux-2.6.27.5-pax/arch/x86/kernel/head64.c 2008-11-12 02:19:11.000000000 +0100 @@ -95,0 +96,2 @@ + x86_64_init_pda(); + @@ -113,2 +114,0 @@ - x86_64_init_pda(); - Hardened could you please add this patch to the hardened-sources patchset? Or bump the grsec patch when a new version with the fix is released. Thanks pax guy and sorry for hijacking this bug ;)
(In reply to comment #25) > > Hardened could you please add this patch to the hardened-sources patchset? Or > bump the grsec patch when a new version with the fix is released. > Thanks pax guy and sorry for hijacking this bug ;) > It will go in with 2.6.27-r2, but it'll be a bit before that release is added to the tree.
2.6.27-r2 is in the tree fixing bbee's issue and also appears to fix the original problem described in this bug. The last planned 2.6.25 and 2.6.26 releases are in the tree, older 2.6.27's removed. Doesn't look like much more is going to happen here. Closing bug, re-open if its not fixed in 2.6.27*.
