Xen DomU hardened kernel 3.1.8 and later crashes on boot. This happens because after kernel 3.0.x CONFIG_GRKERNSEC_HARDENED_VIRTUALIZATION kernel configuration option auto selects CONFIG_PAX_MEMORY_STACKLEAK. DomU kernel crashes on boot with CONFIG_PAX_MEMORY_STACKLEAK enabled. Reproducible: Always Steps to Reproduce: 1. Compile hardened DomU pv kernel 3.1.x+ with CONFIG_GRKERNSEC_HARDENED_VIRTUALIZATION selected. 2. Try to but this kernel as DomU 3. Kernel crashes on boot. Actual Results: Kernel crashed with the following back trace: rip: ffffffff8100dfbc pax_check_alloca+0xc flags: 00000296 i s nz a p rsp: ffffffff8192fe70 rax: 0000000000000080 rcx: 0000000000000000 rdx: ffffffff81b54000 rbx: 0000000000000008 rsi: 0000000000000008 rdi: 0000000000000008 rbp: ffffffff8192fee8 r8: 00000000ffffffff r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 r12: 0000000000000020 r13: ffffffff8192ff48 r14: 0000000000000080 r15: 0000000000000000 cs: e033 ss: e02b ds: 0000 es: 0000 fs: 0000 @ 0000000000000000 gs: 0000 @ 0000000000000000/0000000000000000 Code (instr addr ffffffff8100dfbc) ff 0f 1f 84 00 00 00 00 00 55 48 89 e5 53 48 89 fb 48 83 ec 28 <65> 48 8b 04 25 40 b4 00 00 48 8b Stack: 0000000000000000 0000000000000000 0000000000000000 ffffffff8100dfbc 000000010000e030 0000000000010096 ffffffff8192feb8 000000000000e02b 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ffffffff8170f000 ffffffff8192ff38 Call Trace: [<ffffffff8100dfbc>] pax_check_alloca+0xc <-- [<ffffffff8100dfbc>] pax_check_alloca+0xc [<ffffffff819db746>] xen_load_gdt_boot+0x45 [<ffffffff8101a5dc>] switch_to_new_gdt+0x2c [<ffffffff819dbbca>] xen_start_kernel+0x35d Expected Results: Kernel boots normally. Work around for this issue is to select CONFIG_GRKERNSEC_CUSTOM instead of CONFIG_GRKERNSEC_HARDENED_VIRTUALIZATION and disable CONFIG_PAX_MEMORY_STACKLEAK manually. If fixing a BUG() itself is not possible, at least it would be nice to drop CONFIG_PAX_MEMORY_STACKLEAK from CONFIG_GRKERNSEC_HARDENED_VIRTUALIZATION auto-selection.
I can confirm the PAX_MEMORY_STACKLEAK crash and also to save some time for those who find this bug: The kernel also panics when CONFIG_TCG_TPM is set, when booting in PV mode. (Our kernels mimic production settings except Xen / physical drivers, that't why TPM was set, it's not needed). Tested on the following kernel versions: hardened-sources-3.2.2-r1 hardened-sources-3.2.14-r1 hardened-sources-3.3.1-r1 On these hypervisors: XenServer 6.0 build 50762p (uses Xen hypervisor 4.1.1) [1] XenServer 5.6.0 build 31188p (uses Xen hypervisor 3.4.2-5.6.0.597.20014) [1] [1] http://support.citrix.com/article/CTX122443 Console output: Linux version 3.2.14-hardened-r1 (root@vshop-humc-1) (gcc version 4.5.3 (Gentoo Hardened 4.5.3-r2 p1.1, pie-0.4.7) ) #13 SMP Fri Apr 13 13:27:41 CEST 2012 Command line: root=/dev/xvda3 Released 0 pages of unused memory Set 0 page(s) to 1-1 mapping BIOS-provided physical RAM map: Xen: 0000000000000000 - 00000000000a0000 (usable) Xen: 00000000000a0000 - 0000000000100000 (reserved) Xen: 0000000000100000 - 0000000060000000 (usable) NX (Execute Disable) protection: active DMI not present or invalid. last_pfn = 0x60000 max_arch_pfn = 0x400000000 init_memory_mapping: 0000000000000000-0000000060000000 Zone PFN ranges: DMA 0x00000010 -> 0x00001000 DMA32 0x00001000 -> 0x00100000 Normal empty Movable zone start PFN for each node early_node_map[2] active PFN ranges 0: 0x00000010 -> 0x000000a0 0: 0x00000100 -> 0x00060000 SMP: Allowing 1 CPUs, 0 hotplug CPUs No local APIC present APIC: disable apic facility APIC: switched to apic NOOP Allocating PCI resources starting at 60000000 (gap: 60000000:a0000000) Booting paravirtualized kernel on Xen Xen version: 4.1.1 (preserve-AD) setup_percpu: NR_CPUS:8 nr_cpumask_bits:8 nr_cpu_ids:1 nr_node_ids:1 PERCPU: Embedded 21 pages/cpu @ffff88005ffd7000 s62016 r0 d24000 u86016 Built 1 zonelists in Zone order, mobility grouping on. Total pages: 386209 Kernel command line: root=/dev/xvda3 PID hash table entries: 4096 (order: 3, 32768 bytes) Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) Memory: 1530572k/1572864k available (3408k kernel code, 448k absent, 41844k reserved, 1238k data, 436k init) SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 Hierarchical RCU implementation. NR_IRQS:4352 nr_irqs:24 16 Console: colour dummy device 80x25 console [tty0] enabled console [hvc0] enabled installing Xen timer for CPU 0 Detected 2327.570 MHz processor. Calibrating delay loop (skipped), value calculated using timer frequency.. 4655.14 BogoMIPS (lpj=23275700) pid_max: default: 32768 minimum: 501 Mount-cache hash table entries: 256 CPU: Physical Processor ID: 0 CPU: Processor Core ID: 0 SMP alternatives: switching to UP code Freeing SMP alternatives: 24k freed cpu 0 spinlock event irq 17 Performance Events: unsupported p6 CPU model 15 no PMU driver, software events only. Brought up 1 CPUs Grant table initialized NET: Registered protocol family 16 bio: create slab <bio-0> at 0 xen/balloon: Initialising balloon driver. xen-balloon: Initialising balloon driver. Switching to clocksource xen NET: Registered protocol family 2 IP route cache hash table entries: 65536 (order: 7, 524288 bytes) TCP established hash table entries: 262144 (order: 10, 4194304 bytes) TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) TCP: Hash tables configured (established 262144 bind 65536) TCP reno registered UDP hash table entries: 1024 (order: 3, 32768 bytes) UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) NET: Registered protocol family 1 platform rtc_cmos: registered platform RTC device (no PNP device found) microcode: CPU0 sig=0x6fb, pf=0x40, revision=0xbc microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba Intel AES-NI instructions are not detected. HugeTLB registered 2 MB page size, pre-allocated 0 pages VFS: Disk quotas dquot_6.5.2 Dquot-cache hash table entries: 512 (order 0, 4096 bytes) msgmni has been set to 2989 Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253) io scheduler noop registered io scheduler deadline registered (default) io scheduler cfq registered Event-channel device installed. Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled BUG: unable to handle kernel paging request at ffff88005d827140 IP: [<ffffffff810057d3>] xen_set_pte_at+0x23/0x40 PGD 1365067 PUD ee9067 PMD fd6067 PTE 801000005d827065 Oops: 0003 [#1] SMP CPU 0 Pid: 1, comm: swapper/0 Not tainted 3.2.14-hardened-r1 #13 RIP: e030:[<ffffffff810057d3>] [<ffffffff810057d3>] xen_set_pte_at+0x23/0x40 RSP: e02b:ffff88005d843d20 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88005d827140 RCX: 80000000fed40463 RDX: 0000000000000000 RSI: 80000000fed40463 RDI: ffff88005d827140 RBP: 80000000fed40463 R08: 00000000000000d0 R09: ffff88005da637c0 R10: 0000000000000000 R11: 0000000000000000 R12: 00000000000fed40 R13: 8000000000000573 R14: ffffc9000002d000 R15: 0000000000000001 FS: 0000000000000000(0000) GS:ffff88005ffd7000(0000) knlGS:0000000000000000 CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88005d827140 CR3: 0000000001364000 CR4: 0000000000002660 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swapper/0 (pid: 1, threadinfo ffff88005d8443d8, task ffff88005d844000) Stack: 00003ffffffff000 ffffc90000028000 ffff88005d827140 ffffffff811bda71 00000000000fed45 0000000000000001 0000000000008000 ffff88005da637c0 ffffc9000002d000 ffff88005d826000 ffffc9000002cfff 00003700fed18000 Call Trace: [<ffffffff811bda71>] ? ioremap_page_range+0x271/0x350 [<ffffffff8102ca5d>] ? __ioremap_caller+0x2bd/0x3a0 [<ffffffff8134e19f>] ? _raw_spin_lock+0xf/0x20 [<ffffffff814ba9f9>] ? init_tis+0xc6/0x62e [<ffffffff810076ef>] ? xen_restore_fl_direct_reloc+0x4/0x4 [<ffffffff8134e299>] ? _raw_spin_unlock_irqrestore+0x29/0x30 [<ffffffff814ba933>] ? misc_init+0xb7/0xb7 [<ffffffff8149cbab>] ? do_one_initcall+0x75/0x13a [<ffffffff8149cd05>] ? kernel_init+0x95/0x118 [<ffffffff81350184>] ? kernel_thread_helper+0x4/0x10 [<ffffffff8134ef73>] ? int_ret_from_sys_call+0x7/0x18 [<ffffffff8134e6f6>] ? retint_restore_args+0x5/0x6 [<ffffffff81350180>] ? gs_change+0x13/0x13 Code: 0b 0f 0b 0f 0b 0f 1f 00 48 83 ec 18 48 89 ce 48 89 5c 24 08 48 89 6c 24 10 48 89 d3 48 89 cd 48 89 d7 e8 c1 fe ff ff 84 c0 75 03 <48> 89 2b 48 8b 5c 24 08 48 8b 6c 24 10 48 83 c4 18 c3 66 66 2e RIP [<ffffffff810057d3>] xen_set_pte_at+0x23/0x40 RSP <ffff88005d843d20> CR2: ffff88005d827140 ---[ end trace 936528fd05cf662e ]--- Kernel panic - not syncing: Attempted to kill init! Lost connection to the server.
(In reply to comment #0) > [<ffffffff819db746>] xen_load_gdt_boot+0x45 what happens here is that this function has a local variable-length-array variable that at runtime causes a call to alloca and the PaX specific check function. however it's apparently so early during boot that the per-cpu variables are not available, hence the page fault. i've just fixed this by allocating a fixed sized array (there's no point in saving a few bytes of stack space over this) so the next patches will fix this crash but i'm sure xen has other surprises ;).
(In reply to comment #1) > BUG: unable to handle kernel paging request at ffff88005d827140 > IP: [<ffffffff810057d3>] xen_set_pte_at+0x23/0x40 > PGD 1365067 PUD ee9067 PMD fd6067 PTE 801000005d827065 > Oops: 0003 [#1] SMP > CPU 0 > Pid: 1, comm: swapper/0 Not tainted 3.2.14-hardened-r1 #13 > RIP: e030:[<ffffffff810057d3>] [<ffffffff810057d3>] xen_set_pte_at+0x23/0x40 this looks like an attempt to update a read-only (even in the guest) page table. it would be nice to know if you can still reproduce it with our latest 3.2 or 3.7 patches. also open a new bug for this please ;).
I will do these, just give me some time to catch up with work-work.
Sorry for the delay in reporting back. I don't have access anymore to the mentioned environments, ergo cannot move this issue further. Hopefully it was already sorted out.
Sounds like this might have been resolved by the PaX folks.
