Bug 565732 – sys-kernel/hardened-sources-4.2.x kernel BUG (invalid opcode 0000) during early boot if CONFIG_PAX_KERNEXEC is enabled on a real physical hardware

• Status: RESOLVED FIXED
• Product: Gentoo Linux

[Description Attila Tóth 2015-11-14 00:20:15 UTC]

I haven't been able to successfully boot any hardened-sources-4.2.x kernels so far. However hardened-sources-4.1.7-r1 runs fine. After correcting various problems in earlier hardened-sources-4.2.x kernels, this BUG still remains. It seems right after the kernel reaches init, it just BUGs out.

Please communicate any idea on how to proceed with this problem. Thx: Dw.

Reproducible: Always

[Comment 1 Attila Tóth 2015-11-14 00:26:41 UTC]

Created attachment 416886 [details]
BUG image part1

[Comment 2 Attila Tóth 2015-11-14 00:26:59 UTC]

Created attachment 416888 [details]
BUG image part2

[Comment 3 Anthony Basile 2015-11-14 08:24:45 UTC]

Can you test the 4.2.6-r2 which is the latest in the tree.  Also, can you please attach your kernel config file.  Also, did you test vanilla with the same settings (minus the grsec/pax stuff)?

[Comment 4 Attila Tóth 2015-11-16 00:06:28 UTC]

(In reply to Anthony Basile from comment #3)
> Can you test the 4.2.6-r2 which is the latest in the tree.  Also, can you
> please attach your kernel config file.  Also, did you test vanilla with the
> same settings (minus the grsec/pax stuff)?

Synced yesterday afternoon, the latest was 4.2.6-hardened-r1. Couldn't find -r2.
This kernel shows the same symptoms.

[Comment 5 Attila Tóth 2015-11-16 00:07:20 UTC]

Created attachment 417052 [details]
hardened-sources-4.2.6-r1 configuration

As requested

[Comment 6 Attila Tóth 2015-11-16 00:27:06 UTC]

4.2.6-gentoo passes boot stages. It has some problems after completion because of hardened user space.

[Comment 7 Attila Tóth 2015-11-18 19:46:28 UTC]

I've just tried hardened-sources-4.2.6-r3 and it BUGs exactly the same way.
Still couldn't boot any 4.2.x-hardened kernel so far...

[Comment 8 Alexander Tsoy 2015-11-28 23:50:56 UTC]

(In reply to Anthony Basile from comment #3)
> Also, did you test vanilla with the same settings (minus the grsec/pax stuff)?

This BUG_ON is from grsec/pax patch.

[Comment 9 Alexander Tsoy 2015-11-28 23:55:00 UTC]

Looks like you can workaround this bug by disabling CONFIG_PAX_KERNEXEC

[Comment 10 Attila Tóth 2015-11-28 23:56:29 UTC]

(In reply to Alexander Tsoy from comment #8)
> (In reply to Anthony Basile from comment #3)
> > Also, did you test vanilla with the same settings (minus the grsec/pax stuff)?
> 
> This BUG_ON is from grsec/pax patch.

hardened-sources-4.2.6-r6 BUGs exactly the same way as prior hardened-4.2.6-* versions. I tried a modified config, but I run into some crypto-related PANICs.
I still have to hang on using hardened-4.1.7-r1. That is the last one that boots flawlessly.

[Comment 11 Attila Tóth 2015-11-29 00:02:52 UTC]

(In reply to Alexander Tsoy from comment #9)
> Looks like you can workaround this bug by disabling CONFIG_PAX_KERNEXEC

The boxes are real physical computers, I'm not booting virtual machines. I would rather keep KERNEXEC as it isn't feature told be useless or deprecated. Hardened-4.1.7-r1 kernel boots fine. Something happened while bumping to 4.2.6*.

[Comment 12 Attila Tóth 2015-12-10 00:25:34 UTC]

hardened-sources-4.2.6-r8 still BUGs (invalid opcode 0000) while kernel poking init during early boot, if CONFIG_PAX_KERNEXEC is enabled on a real physical hardware (both with bts or or).
The issue appeared by the advent of 4.2.6* kernels. 4.1.7-hardened-r1 boots fine with KERNEXEC enabled.

[Comment 13 Attila Tóth 2015-12-10 00:28:39 UTC]

It would be disappointing if I have to neglect this hardening feature (CONFIG_PAX_KERNEXEC)...

[Comment 14 PaX Team 2015-12-16 03:00:58 UTC]

some time ago i added some self-checking code to detect any leftover rwx kernel memory that KERNEXEC should have taken care of, my guess is that it's the one that triggers here. one thing i noticed is that you enabled DEBUG_PAGEALLOC, is there a reason for it?

[Comment 15 Attila Tóth 2015-12-17 06:27:05 UTC]

(In reply to PaX Team from comment #14)
> some time ago i added some self-checking code to detect any leftover rwx
> kernel memory that KERNEXEC should have taken care of, my guess is that it's
> the one that triggers here. one thing i noticed is that you enabled
> DEBUG_PAGEALLOC, is there a reason for it?

You are right, that was the problem. I don't remember when it was enabled, but disabling DEBUG_PAGEALLOC let me select boot the kernel with KERNEXEC enabled. These two configuration option therefore incompatible. It would be good to make them mutually exclusive.

Many thanks: Dw.

[Comment 16 PaX Team 2015-12-17 09:58:51 UTC]

(In reply to Attila Tóth from comment #15)
> You are right, that was the problem. I don't remember when it was enabled,
> but disabling DEBUG_PAGEALLOC let me select boot the kernel with KERNEXEC
> enabled. These two configuration option therefore incompatible. It would be
> good to make them mutually exclusive.

they're not necessarily incompatible, just a combination that noone used in production so far (DEBUG_PAGEALLOC is a debugging option with a non-trivial performance impact, i'm surprised you didn't notice ;). i'll try to reproduce this in qemu and see what the best way to resolve it is.

[Comment 17 Attila Tóth 2015-12-17 20:22:04 UTC]

(In reply to PaX Team from comment #16)
> (In reply to Attila Tóth from comment #15)
> > You are right, that was the problem. I don't remember when it was enabled,
> > but disabling DEBUG_PAGEALLOC let me select boot the kernel with KERNEXEC
> > enabled. These two configuration option therefore incompatible. It would be
> > good to make them mutually exclusive.
> 
> they're not necessarily incompatible, just a combination that noone used in
> production so far (DEBUG_PAGEALLOC is a debugging option with a non-trivial
> performance impact, i'm surprised you didn't notice ;). i'll try to
> reproduce this in qemu and see what the best way to resolve it is.

Thanks for you effort, Pipacs: Békés Karácsonyt!