Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 457176 - sys/zfs-kmod-9999 with sys-kernel/hardened-sources-3.7.5 "unable to handle kernel NULL pointer dereference"
Summary: sys/zfs-kmod-9999 with sys-kernel/hardened-sources-3.7.5 "unable to handle ke...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Hardened (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Richard Yao (RETIRED)
URL:
Whiteboard:
Keywords: Bug, REGRESSION
Depends on:
Blocks:
 
Reported: 2013-02-14 00:22 UTC by Marcin Mirosław
Modified: 2013-02-16 22:14 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
.config (.config,70.20 KB, text/plain)
2013-02-14 00:26 UTC, Marcin Mirosław
Details
emerge --info (ingo,3.00 KB, text/plain)
2013-02-14 00:30 UTC, Marcin Mirosław
Details
zfs-kmod-9999 config.log (config.log,131.95 KB, text/plain)
2013-02-14 23:08 UTC, Marcin Mirosław
Details
spl-9999 config.log (config.log,96.14 KB, text/plain)
2013-02-14 23:18 UTC, Marcin Mirosław
Details
Proposed patch (zfs-kmod-0.6.0_rc14-fix-hardened-support-regression.patch,594 bytes, patch)
2013-02-14 23:52 UTC, Richard Yao (RETIRED)
Details | Diff
Proposed patch (zfs-kmod-0.6.0_rc14-fix-hardened-support-regression.patch,1.59 KB, patch)
2013-02-15 01:06 UTC, Richard Yao (RETIRED)
Details | Diff
Proposed patch (zfs-kmod-0.6.0_rc14-fix-hardened-support-regression.patch,17.16 KB, patch)
2013-02-15 06:50 UTC, Richard Yao (RETIRED)
Details | Diff
spl-9999 -3.7.5 config.log (config.log,98.57 KB, text/plain)
2013-02-15 08:09 UTC, Marcin Mirosław
Details
zfs-kmod-9999 -3.7.5 config.log (config.log,135.12 KB, text/plain)
2013-02-15 08:10 UTC, Marcin Mirosław
Details
zfs-kmod-9999 -3.7.5 with patch config.log (config.log,109.60 KB, text/plain)
2013-02-15 13:48 UTC, Marcin Mirosław
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Marcin Mirosław 2013-02-14 00:22:17 UTC
After all zfs related modules are loaded I'm getting:
2013-02-14T00:52:04.112488+01:00  kernel: PAX: please report this to pageexec@freemail.hu
2013-02-14T00:52:04.112488+01:00  kernel: BUG: unable to handle kernel NULL pointer dereference at 000000000000003d
2013-02-14T00:52:04.112488+01:00  kernel: IP: [<ffffffffa07ca10c>] zrl_is_locked+0xa5c/0x1970 [zfs]
2013-02-14T00:52:04.112488+01:00  kernel: PGD 412af4066
2013-02-14T00:52:04.112488+01:00  kernel: Thread overran stack, or stack corrupted
2013-02-14T00:52:04.112489+01:00  kernel: Oops: 0000 [#1] SMP
2013-02-14T00:52:04.112489+01:00  kernel: Modules linked in: hid_generic usbhid dm_mod zfs(PO+) zunicode(PO) zavl(PO) zcommon(PO) znvpair(PO) spl(O) sr_mod coretemp cdrom kvm_intel zlib_inflate zlib_deflate kvm ehci_hcd crc32c_intel aesni_intel xts usbcore aes_x86_64 lrw e1000e gf128mul ablk_helper usb_common iTCO_wdt ahci libahci cryptd i2c_i801 lpc_ich mfd_core i2c_core microcode thermal processor fan thermal_sys evdev hwmon button unix
2013-02-14T00:52:04.112491+01:00  kernel: CPU 0
2013-02-14T00:52:04.112491+01:00  kernel: Pid: 356, comm: modprobe Tainted: P           O 3.7.5-hardened #3 Supermicro X9SCL/X9SCM/X9SCL/X9SCM
2013-02-14T00:52:04.112491+01:00  kernel: RIP: 0010:[<ffffffffa07ca10c>]  [<ffffffffa07ca10c>] zrl_is_locked+0xa5c/0x1970 [zfs]
2013-02-14T00:52:04.112491+01:00  kernel: RSP: 0018:ffff880412adba78  EFLAGS: 00010246
2013-02-14T00:52:04.112491+01:00  kernel: RAX: ffffffffa07ca0e0 RBX: ffff88041ec28300 RCX: 000000000000000f
2013-02-14T00:52:04.112491+01:00  kernel: RDX: 8000000000000000 RSI: 0000000000000001 RDI: ffff88041ec28300
2013-02-14T00:52:04.112492+01:00  kernel: RBP: ffff880412adbad8 R08: 0000000000000000 R09: 041ec284700c0000
2013-02-14T00:52:04.112492+01:00  kernel: R10: fbc33e8340951c03 R11: 0000000000000001 R12: 0000000000000001
2013-02-14T00:52:04.112492+01:00  kernel: R13: ffffffff81162570 R14: 0000000000000001 R15: 0000000000000000
2013-02-14T00:52:04.112492+01:00  kernel: FS:  00000275d8913700(0000) GS:ffff88042fc00000(0000) knlGS:0000000000000000
2013-02-14T00:52:04.112492+01:00  kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2013-02-14T00:52:04.112492+01:00  kernel: CR2: 000000000000003d CR3: 0000000001447000 CR4: 00000000001407f0
2013-02-14T00:52:04.112492+01:00  kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
2013-02-14T00:52:04.112492+01:00  kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
2013-02-14T00:52:04.112492+01:00  kernel: Process modprobe (pid: 356, threadinfo ffff8804181cce98, task ffff8804181cca60)
2013-02-14T00:52:04.112492+01:00  kernel: Stack:
2013-02-14T00:52:04.112492+01:00  kernel:  ffff880412adbb14 ffff880418012a00 0000000000000001 ffff880412077870
2013-02-14T00:52:04.112493+01:00  kernel:  ffff880412adbab8 ffffffff812cab0a ffff880418012a00 ffff880412077800
2013-02-14T00:52:04.112493+01:00  kernel:  ffff88041ec28318 ffffffffa0713940 0000000000000001 ffff88041ec28300
2013-02-14T00:52:04.112493+01:00  kernel: Call Trace:
2013-02-14T00:52:04.112493+01:00  kernel:  [<ffffffff812cab0a>] ? get_device+0x1a/0x40
2013-02-14T00:52:04.112493+01:00  kernel:  [<ffffffff8116361d>] __blkdev_get+0xed/0x4f0
2013-02-14T00:52:04.112493+01:00  kernel:  [<ffffffff81163ba5>] blkdev_get+0x185/0x2d0
2013-02-14T00:52:04.112493+01:00  kernel:  [<ffffffff811433d2>] ? unlock_new_inode+0x42/0x70
2013-02-14T00:52:04.112493+01:00  kernel:  [<ffffffff812cab7a>] ? put_device+0x1a/0x30
2013-02-14T00:52:04.112494+01:00  kernel:  [<ffffffff8121eda4>] add_disk+0x394/0x4b0
2013-02-14T00:52:04.112494+01:00  kernel:  [<ffffffff81114508>] ? kfree+0xd8/0x100
2013-02-14T00:52:04.112494+01:00  kernel:  [<ffffffffa07c9f5f>] zrl_is_locked+0x8af/0x1970 [zfs]
2013-02-14T00:52:04.112494+01:00  kernel:  [<ffffffffa07c9fe0>] ? zrl_is_locked+0x930/0x1970 [zfs]
2013-02-14T00:52:04.112494+01:00  kernel:  [<ffffffffa07ca006>] zrl_is_locked+0x956/0x1970 [zfs]
2013-02-14T00:52:04.112494+01:00  kernel:  [<ffffffffa0735a67>] dmu_objset_find_spa+0x347/0x480 [zfs]
2013-02-14T00:52:04.112494+01:00  kernel:  [<ffffffffa07c9fe0>] ? zrl_is_locked+0x930/0x1970 [zfs]
2013-02-14T00:52:04.112494+01:00  kernel:  [<ffffffffa07358df>] dmu_objset_find_spa+0x1bf/0x480 [zfs]
2013-02-14T00:52:04.112494+01:00  kernel:  [<ffffffffa07c9fe0>] ? zrl_is_locked+0x930/0x1970 [zfs]
2013-02-14T00:52:04.112494+01:00  kernel:  [<ffffffffa07cb841>] zvol_create_minors+0xb1/0xe0 [zfs]
2013-02-14T00:52:04.112494+01:00  kernel:  [<ffffffffa07a4480>] ? _fini+0xe0/0xe0 [zfs]
2013-02-14T00:52:04.112494+01:00  kernel:  [<ffffffffa07a4480>] ? _fini+0xe0/0xe0 [zfs]
2013-02-14T00:52:04.112494+01:00  kernel:  [<ffffffffa07cba9e>] zvol_init+0xce/0x110 [zfs]
2013-02-14T00:52:04.112495+01:00  kernel:  [<ffffffffa07a428d>] _init+0x1d/0x130 [zfs]
2013-02-14T00:52:04.112495+01:00  kernel:  [<ffffffffa07a448e>] init_module+0xe/0x20 [zfs]
2013-02-14T00:52:04.112495+01:00  kernel:  [<ffffffff81000327>] do_one_initcall+0x147/0x170
2013-02-14T00:52:04.112495+01:00  kernel:  [<ffffffff810980ef>] sys_init_module+0x7f/0x1f0
2013-02-14T00:52:04.112495+01:00  kernel:  [<ffffffff8143c12a>] system_call_fastpath+0x18/0x1d
2013-02-14T00:52:04.112495+01:00  kernel: Code: 48 89 e5 48 83 ec 60 48 89 5d d8 48 89 fb 4c 89 65 e0 49 89 f4 4c 89 6d e8 4c 89 75 f0 4c 89 7d f8 45 31 ff 4c 8b ab e8 01 00 00 <45> 8b 74 24 3c 49 8b 85 90 00 00 00 48 8b 98 40 02 00 00 65 48
2013-02-14T00:52:04.112498+01:00  kernel: RIP  [<ffffffffa07ca10c>] zrl_is_locked+0xa5c/0x1970 [zfs]
2013-02-14T00:52:04.112498+01:00  kernel:  RSP <ffff880412adba78>
2013-02-14T00:52:04.112498+01:00  kernel: CR2: 000000000000003d
2013-02-14T00:52:04.112498+01:00  kernel: ---[ end trace 08a45a059ed164e3 ]---
Comment 1 Marcin Mirosław 2013-02-14 00:26:02 UTC
Created attachment 338818 [details]
.config
Comment 2 Marcin Mirosław 2013-02-14 00:30:38 UTC
Created attachment 338820 [details]
emerge --info
Comment 3 Matthew Thode ( prometheanfire ) archtester Gentoo Infrastructure gentoo-dev Security 2013-02-14 21:51:37 UTC
can we get the config.log?
Comment 4 Marcin Mirosław 2013-02-14 23:08:51 UTC
Created attachment 338914 [details]
zfs-kmod-9999 config.log

Here it is. I'm wondering could gcc-4.7 be root of this problem?
Comment 5 Richard Yao (RETIRED) gentoo-dev 2013-02-14 23:10:13 UTC
An autotools check in either sys-kernel/spl or sys-fs/zfs-kmod is likely suffering a silent failure. They exist checks need to be modified to fail early by explicitly checking for unknown cases. Eventually, I will find time to do this. In the mean time, I can patch your specific issue provided that I have the config.log files generated by the configure script.

I will take a look at the config.log from sys-fs/zfs-kmod, but it will help if I also had the config.log from sys-kernel/spl. Would you paste that too?
Comment 6 Marcin Mirosław 2013-02-14 23:18:29 UTC
Created attachment 338916 [details]
spl-9999 config.log

Sure, here is config.log from spl.
Comment 7 Richard Yao (RETIRED) gentoo-dev 2013-02-14 23:52:13 UTC
Created attachment 338918 [details, diff]
Proposed patch

This regression was caused by upstream commit 8780c53961e668211682d40ad36946294c3145d8. I have attached a patch that should correct this. Please it put into /etc/portage/patches/sys-fs/zfs-kmod-9999/zfs-kmod-0.6.0_rc14-fix-hardened-support-regression.patch, rebuild sys-fs/zfs-kmod, rebuild any initramfs that you might be using and let me know if this fixes the issue for you.
Comment 8 Anthony Basile gentoo-dev 2013-02-15 00:05:10 UTC
(In reply to comment #7)
> Created attachment 338918 [details, diff] [details, diff]
> Proposed patch
> 
> This regression was caused by upstream commit
> 8780c53961e668211682d40ad36946294c3145d8. I have attached a patch that
> should correct this. Please it put into
> /etc/portage/patches/sys-fs/zfs-kmod-9999/zfs-kmod-0.6.0_rc14-fix-hardened-
> support-regression.patch, rebuild sys-fs/zfs-kmod, rebuild any initramfs
> that you might be using and let me know if this fixes the issue for you.

This looks like a vanilla kernel issue, not hardened specific.  The patch should go into genpatches.

I pushing this one mpagano's way.
Comment 9 Anthony Basile gentoo-dev 2013-02-15 00:08:15 UTC
(In reply to comment #8)
> (In reply to comment #7)
> > Created attachment 338918 [details, diff] [details, diff] [details, diff]
> > Proposed patch
> > 
> > This regression was caused by upstream commit
> > 8780c53961e668211682d40ad36946294c3145d8. I have attached a patch that
> > should correct this. Please it put into
> > /etc/portage/patches/sys-fs/zfs-kmod-9999/zfs-kmod-0.6.0_rc14-fix-hardened-
> > support-regression.patch, rebuild sys-fs/zfs-kmod, rebuild any initramfs
> > that you might be using and let me know if this fixes the issue for you.
> 
> This looks like a vanilla kernel issue, not hardened specific.  The patch
> should go into genpatches.
> 
> I pushing this one mpagano's way.

Oh wait, I take that back.  This kmod is not bundled with the kernel so its just a patch against zfs-kmod.
Comment 10 Richard Yao (RETIRED) gentoo-dev 2013-02-15 01:06:34 UTC
Created attachment 338926 [details, diff]
Proposed patch

I have reproduced this in a Gentoo Hardened installation. My initial patch will not work. I have attached a new patch that I have verified to work.

I will send this upstream and backport this to Gentoo's sys-fs/zfs-kmod ebuilds shortly. Note that I will not make any modifications to the 9999 ebuild. The 9999 ebuild will receive this patch when it is accepted upstream.
Comment 11 Richard Yao (RETIRED) gentoo-dev 2013-02-15 01:48:47 UTC
Actually, this patch might not address your issue. I have stumbled across more autotools checks that silently fail on sys-kernel/hardened-sources. I am working on fixing them now.
Comment 12 Richard Yao (RETIRED) gentoo-dev 2013-02-15 06:50:42 UTC
Created attachment 338956 [details, diff]
Proposed patch

I have written a new patch that eliminates all runtime function pointer modifications from the autotools checks, which should solve your issue. Please apply this to zfs-kmod and let me know if it solve your issue.
Comment 13 Marcin Mirosław 2013-02-15 08:09:59 UTC
Created attachment 338958 [details]
spl-9999 -3.7.5 config.log
Comment 14 Marcin Mirosław 2013-02-15 08:10:55 UTC
Created attachment 338960 [details]
zfs-kmod-9999 -3.7.5 config.log
Comment 15 Marcin Mirosław 2013-02-15 08:15:12 UTC
I've made config.log against hardened-sources-3/7/0 instead 3.7.5. Sorry for this, it's good this bug is reproducible.
Comment 16 Richard Yao (RETIRED) gentoo-dev 2013-02-15 08:25:31 UTC
(In reply to comment #15)
> I've made config.log against hardened-sources-3/7/0 instead 3.7.5. Sorry for
> this, it's good this bug is reproducible.

It looks like you did not patch your kernel correctly. Place the patch in /etc/portage/patches/sys-fs/zfs-kmod-9999/zfs-kmod-0.6.0_rc14-fix-hardened-support-regression.patch.
Comment 17 Richard Yao (RETIRED) gentoo-dev 2013-02-15 08:34:39 UTC
By kernel, I meant zfs-kmod. Sorry for the confusion.
Comment 18 Marcin Mirosław 2013-02-15 13:48:08 UTC
Created attachment 338990 [details]
zfs-kmod-9999 -3.7.5 with patch config.log

Richard, firstly thanks for very quick responses.
About patching, I didn't even try to apply patch I just wanted to attach correct config.log (as I mentioned earlier I didn't change kernel symlink).
Now I'm attaching config.log created with your patch. I can't check now if patch solves my issue, it's on a prodcution server. I've to wait till midnight hours.
Comment 19 Richard Yao (RETIRED) gentoo-dev 2013-02-15 15:40:06 UTC
It appears that the autotools checks are doing the right thing on your system now. Nothing bad stands out from a brief look through the attached log.
Comment 20 Marcin Mirosław 2013-02-15 23:34:55 UTC
Richard, it looks everything is ok. Thanks a lot for creating fix!
Comment 21 Richard Yao (RETIRED) gentoo-dev 2013-02-16 21:25:48 UTC
(In reply to comment #20)
> Richard, it looks everything is ok. Thanks a lot for creating fix!

Thanks for the quick feedback on it.

By the way, building with USE=debug in the future will make these issues into build failures, which is much better than runtime failures.
Comment 22 Richard Yao (RETIRED) gentoo-dev 2013-02-16 22:14:19 UTC
I have committed sys-kernel/zfs-kmod-0.6.0_rc14-r3 to the main tree, which includes this patch. I have also opened an upstream pull request with the patch:

https://github.com/zfsonlinux/zfs/pull/1300

The 9999 ebuild will have the patch as soon as upstream merges it. Afterward, you will want to remove the patch file that you installed on your system. Otherwise, you will see build failures.