Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 496076 - sys-kernel/genkernel-next-50 Unable to resolve root on md array
Summary: sys-kernel/genkernel-next-50 Unable to resolve root on md array
Status: UNCONFIRMED
Alias: None
Product: Gentoo Hosted Projects
Classification: Unclassified
Component: genkernel-next (show other bugs)
Hardware: All Linux
: Normal normal with 1 vote (vote)
Assignee: Ettore Di Giacinto
URL:
Whiteboard:
Keywords: NeedPatch
Depends on:
Blocks:
 
Reported: 2013-12-26 14:53 UTC by Juergen Rose
Modified: 2017-09-12 18:14 UTC (History)
6 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Juergen Rose 2013-12-26 14:53:23 UTC
I tried to convert a further system to work with systemd. Therefore I booted with systemrescuecd-3.5.0, made a backup of all partitions and repartioned my harddisks to merge / and /usr. Then I restored the backup to the new partitions.

At the systemrescuecd system I have now three Raid1 arrays:

root@sysresccd /root % cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md126 : active raid1 sda5[0] sdb5[1]                       -> LV
      1677720440 blocks super 1.2 [2/2] [UU]
      
md125 : active raid1 sda3[0] sdb3[1]                       -> /
      138411968 blocks [2/2] [UU]
      
md127 : active raid1 sda1[0] sdb1[1]                       -> /boot
      263936 blocks [2/2] [UU]


Under /dev/disk/by-uuid I see:

    40bc...                              -> ../../md127
    6d8x1see-1206-4cad-b943-5d7ca81fbe88 -> ../../md125
    ( md126 does not exist ? )

In /mnt/gentoo/etc/fstab I have:

   /dev/md125   /      ...
   /dev/md127   /boot  ...    noauto


I added " init=/usr/lib/systemd/systemd rootfstype=ext4" to GRUB_CMDLINE_LINUX_DEFAULT="domdadm dolvm video=VGA-1:1600x1200M@76" in /mnt/gentoo/etc/default/grub.

I chrooted to /mnt/gentoo, mounted /dev/md127 under /boot and did "LANG=C grub2-mkconfig -o /boot/grub2/grub.cfg".

In /boot/grub2/grub.cfg there are now entries:

   linux /kernel-genkernel-x86_64-3.12.6-gentoo root=UUID=6d8x1see-1206-4cad-b943-5d7ca81fbe88 ...



If I now try to boot this system from harddisk, it hangs after drm_kms_helper: panic occureed, switching back to text console.
Warning CPU: 1PID: 22 at arch/x86/kernel/smp.c


If I hard reboot and press CTL-S at the right time. I can see:

Unable to resolve root:  UUID=6d8x1see-1206-4cad-b943-5d7ca81fbe88



root@sysresccd:/(8)# qlist -Iv systemd
sys-apps/gentoo-systemd-integration-2
sys-apps/systemd-208-r2
root@sysresccd:/(9)# qlist -Iv genkernel
sys-kernel/genkernel-next-50
root@sysresccd:/(10)# qlist -Iv grub
sys-boot/grub-2.02_beta2
Comment 1 Juergen Rose 2013-12-26 15:18:45 UTC
If I activate the line:

GRUB_DISABLE_LINUX_UUID=true

in /etc/default/grub, booting fails with:

/dev/md125 is an invalid root device...
Comment 2 Mike Gilbert gentoo-dev 2013-12-26 18:31:57 UTC
Is 6d8x1see-1206-4cad-b943-5d7ca81fbe88 really the UUID value, or have you altered it?

The characters 'x' and 's' are not valid in a UUID; it's supposed to be a series of hexadecimal digits and hyphens.
Comment 3 David Kredba 2013-12-26 19:26:20 UTC
You need initramfs with mdmon and mdadm in my opinion.


Try genkernel --mdadm --lvm --install initramfs
and rename or copy installed initramfs to initramfs-x.y.z-gentoo-rR.img.

Rebuild grub.cfg, it should find it too. In kernel INITRD support is needed.
Comment 4 Juergen Rose 2013-12-26 20:56:12 UTC
(In reply to Mike Gilbert from comment #2)
> Is 6d8x1see-1206-4cad-b943-5d7ca81fbe88 really the UUID value, or have you
> altered it?
> 
> The characters 'x' and 's' are not valid in a UUID; it's supposed to be a
> series of hexadecimal digits and hyphens.


Sorry it is a typo, I wrote it manually from the screen. The correct UUID is


root@sysresccd /root % ll /dev/disk/by-uuid 
total 0
...
lrwxrwxrwx 1 root root  11 Dec 26 16:50 40bc33e7-c56c-4e1d-b2dd-87af6e00372c -> ../../md127
lrwxrwxrwx 1 root root  11 Dec 26 16:50 6d8a1aee-1306-4cad-b943-5d7ca81f6e88 -> ../../md125
...
Comment 5 Mike Gilbert gentoo-dev 2013-12-26 21:04:49 UTC
It would be interesting to see what the initramfs environment sees.

Can you add "debug" to the kernel command line? This will drop you into a debug shell before it tries to mount your root filesystem. You can then run things like "blkid". You could also check to see that sysfs is mounted and the /dev/md125 device node exists.
Comment 6 Juergen Rose 2013-12-27 07:46:21 UTC
(In reply to David Kredba from comment #3)
> You need initramfs with mdmon and mdadm in my opinion.
> 
> 
> Try genkernel --mdadm --lvm --install initramfs
> and rename or copy installed initramfs to initramfs-x.y.z-gentoo-rR.img.
> 
> Rebuild grub.cfg, it should find it too. In kernel INITRD support is needed.

Usually I generate kernel and initramfs with
"genkernel --oldconfig --udev --mdadm --lvm all". So I do not know, what should be improved with "genkernel --mdadm --lvm --install initramfs". Nevertheless I tested it.

At the start I had:

root@sysresccd:/usr/src/linux(12)# ll /boot/
total 56739
lrwxrwxrwx 1 root root       1 Jan 13  2011 boot -> ./
drwxr-xr-x 6 root root    1024 Dec 26 15:44 grub/
drwxr-xr-x 6 root root    1024 Dec 26 17:33 grub2/
...
-rw-r--r-- 1 root root 4422544 Dec 23 17:08 initramfs-genkernel-x86_64-3.12.6-gentoo
...
-rw-r--r-- 1 root root 4680336 Dec 23 17:03 kernel-genkernel-x86_64-3.12.6-gentoo
...
-rw-r--r-- 1 root root 2831188 Dec 23 17:03 System.map-genkernel-x86_64-3.12.6-gentoo


After "genkernel --mdadm --lvm --install initramfs" I had:
root@sysresccd:/usr/src/linux(13)# genkernel --mdadm --lvm --install initramfs
* Gentoo Linux Genkernel; Version 50
* Running with options: --mdadm --lvm --install initramfs

* Using genkernel.conf from /etc/genkernel.conf
* Sourcing arch-specific config.sh from /usr/share/genkernel/arch/x86_64/config.sh ..
* Sourcing arch-specific modules_load from /usr/share/genkernel/arch/x86_64/modules_load ..

* Linux Kernel 3.12.6-gentoo for x86_64...
* .. with config file /etc/kernels/kernel-config-x86_64-3.12.6-gentoo
...
root@sysresccd:/usr/src/linux(14)# ll /boot/
total 56747
...
-rw-r--r-- 1 root root 4430456 Dec 27 08:47 initramfs-genkernel-x86_64-3.12.6-gentoo


I do not understand which initramfs I should copy or rename. Then I did again "LANG=C grub2-mkconfig -o /boot/grub2/grub.cfg" and the next reboot fails with the same error: "Unable to resolve root:  UUID=6d8a1aee..."
Comment 7 Mike Gilbert gentoo-dev 2013-12-27 16:56:15 UTC
This is almost certainly a problem with your initramfs (genkernel-next), not with grub. I am reassigning this to the maintainer of genkernel-next.

In the mean time, please attempt the debugging procedure I mention in comment 5.

Also, please ignore David's comment 3; you are already using a supported initramfs and do not need to rename anything.
Comment 8 Juergen Rose 2013-12-28 12:55:02 UTC
(In reply to Mike Gilbert from comment #7)
> This is almost certainly a problem with your initramfs (genkernel-next), not
> with grub. I am reassigning this to the maintainer of genkernel-next.
> 
> In the mean time, please attempt the debugging procedure I mention in
> comment 5.
> 
> Also, please ignore David's comment 3; you are already using a supported
> initramfs and do not need to rename anything.

I added debug to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub. Then I regenerated /boot/grub2/grub.cfg:

root@sysresccd:/(16)# grep debug /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="domdadm dolvm video=VGA-1:1600x1200M@76 rootfstype=ext4 debug"

root@sysresccd:/(17)# grep debug /boot/grub2/grub.cfg 
        linux   /kernel-genkernel-x86_64-3.12.6-gentoo root=UUID=6d8a1aee-1306-4cad-b943-5d7ca81f6e88 ro  domdadm dolvm video=VGA-1:1600x1200M@76 rootfstype=ext4 debug
                linux   /kernel-genkernel-x86_64-3.12.6-gentoo root=UUID=6d8a1aee-1306-4cad-b943-5d7ca81f6e88 ro  domdadm dolvm video=VGA-1:1600x1200M@76 rootfstype=ext4 debug
                linux   /kernel-genkernel-x86_64-3.12.5-gentoo root=UUID=6d8a1aee-1306-4cad-b943-5d7ca81f6e88 ro  domdadm dolvm video=VGA-1:1600x1200M@76 rootfstype=ext4 debug
                linux   /kernel-genkernel-x86_64-3.12.4-gentoo root=UUID=6d8a1aee-1306-4cad-b943-5d7ca81f6e88 ro  domdadm dolvm video=VGA-1:1600x1200M@76 rootfstype=ext4 debug
                linux   /kernel-genkernel-x86_64-3.11.6-gentoo root=UUID=6d8a1aee-1306-4cad-b943-5d7ca81f6e88 ro  domdadm dolvm video=VGA-1:1600x1200M@76 rootfstype=ext4 debug
                linux   /kernel-genkernel-x86_64-3.10.10-gentoo root=UUID=6d8a1aee-1306-4cad-b943-5d7ca81f6e88 ro  domdadm dolvm video=VGA-1:1600x1200M@76 rootfstype=ext4 debug

root@sysresccd:/(18)# blkid /dev/md125 
/dev/md125: UUID="6d8a1aee-1306-4cad-b943-5d7ca81f6e88" TYPE="ext4" 
root@sysresccd:/(19)# df -T
...
Filesystem              Type     1K-blocks     Used Available Use% Mounted on
rootfs                  rootfs   136240232 32240968  97078668  25% /
udev                    tmpfs        10240        4     10236   1% /dev
none                    aufs     136240232 32240968  97078668  25% /
/dev/md125              ext4     136240232 32240968  97078668  25% /
...
udev                    devtmpfs     10240        4     10236   1% /dev
/dev/md127              ext2        255595    74668    167731  31% /boot
root@sysresccd:/(20)# 

If I reboot the system the last messages are:

...
>> Starting md devices
[    6.814135] md: md127 stopped.
[    6.844915] md: bind<sdb1>
[    6.848340] md: bind<sda1>
[    6.852938] md/raid1:md127: active with 2 out of 2 mirrors
[    6.856138] md127: detected capacity change from 0 to 270270464
mdadm: /dev/nd/126_0 has been started with 2 drives.
[    9.863663] systemd-udevd (1375) used greatest stack depth: 4712 bytes left
[   11.475209] md127: unknown partition table
>> Activating Logical Volume Groups
  No volume groups found
!! /sbin/cryptsetup not found inside the initramfs
>> Starting debug shell as requested.
>> Stopping by: before setting up the root filesystem
>> Type "exit" to continue with normal bootup.

Busybox v1.20.2 (2013-10-21 11:30:34 CEST) build-in shell (ash)
...

If I try some commands I see that there is only /dev/md127 and no other raid device:

# ls /dev/md*
/dev/md127
# ls /dev/md/*
ls: /dev/md/*: No such file or directory
# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md127 : active raid1 sda1[0] sdb1[1]
      263936 blocks [2/2] [UU]

unused devices: (none)


I can activate /dev/md126 manually:

# mdadm --autodetect
mdadm: unrecognized option '--autodetect'
Usage: mdadm --help

# mdadm --assemble /dev/md126
mdadm: /dev/md126 not identified in config file.
# mdadm --assemble /dev/md126 /dev/sda5 /dev/sdb5
[ 1498.960999] md: md126 stopped.
[ 1498.965934] md: bind<sdb5>
[ 1498.979644] md: bind<sda5>
[ 1498.984965] md/raid1:md126: active with 2 out of 2 mirrors
[ 1498.988770] md126: detected capacity change from 0 to 1717985730560
mdadm: /dev/md125 has been started with 2 drives/

# cat /proc/mdstat
Personalities: [raid1] [raid6] [raid5] [raid4]
md126 : active raid1 sda5[0] sdb5[1]
      1677720440 blocks super 1.2 [2/2] [UU]

md127 : active raid1 sda1[0] sdb1[1]
      263936 blocks [2/2] [UU]

unused devices: (none)

# blkid /dev/md126
[ 2151.027063]  md126: unknown partition table
/dev/md126: UUID="QHbmqE-HF0x-hNWI-bYK7-bjHy-2nfp-GYDGvV" TYPE="LVM2_member"

I scratch this number manually from screen, but I double checked them. 

# mdadm --stop /dev/md126
[ 2794.513371] md126: detected capacity change from 1717985730560 to 0
...
# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md127 : active raid1 sda1[0] sdb1[1]
      263936 blocks [2/2] [UU]

unused devices: (none)

# mdadm --assemble /dev/md125 /dev/sda3 /dev/sdb3
...
# cat /proc/mdstat
Personalities: [raid1] [raid6] [raid5] [raid4]
md125 : active raid1 sda3[0] sdb3[1]
      138411968 blocks [2/2] [UU]

md127 : active raid1 sda1[0] sdb1[1]
      263936 blocks [2/2] [UU]

unused devices: (none)

# blkid /dev/md125
[ 3445.0111631]  md125: unknown partition table
/dev/md125: UUID="6d8a1aee-1306-4cad-b943-5d7ca81f6e88" TYPE="ext4"


So it looks. if the environment is OK but the raid device /dev/md125 is not yet started, when the initial ramdisk tries to mount it as the root device. Or am I wrong?
Comment 9 Juergen Rose 2013-12-28 13:03:42 UTC
BTW., after leaving the debug mode twice, the system boots normally.
Comment 10 Juergen Rose 2013-12-28 14:09:54 UTC
(In reply to Mike Gilbert from comment #7)
> This is almost certainly a problem with your initramfs (genkernel-next), not
> with grub. I am reassigning this to the maintainer of genkernel-next.
> 
> In the mean time, please attempt the debugging procedure I mention in
> comment 5.
> 
> Also, please ignore David's comment 3; you are already using a supported
> initramfs and do not need to rename anything.

It was a problem with initramfs, but not with genkernel-next. It was my fault.

I have the line
MDADM_CONFIG="/etc/mdadm.conf"

in /etc/genkernel.conf. And I had old 

ARRAY /dev/125_0 metadata=0.90 UUID=c7b6eb55:...
ARRAY /dev/126_0 metadata=0.90 UUID=28e56e52:...

entries in /etc/mdadm.conf, which have been copied to initramfs and confused booting. After commenting these lines /dev/md126 with sda3 and sdb3 is now recognized as root device.
Comment 11 Fabio Erculiani (RETIRED) gentoo-dev 2013-12-28 14:43:29 UTC
Honestly, I think that genkernel should never copy block devices configuration files to the initramfs. This should be always able to detect and mount the rootfs without these. As you proved here, copying mdadm.conf does more harm than good.
Comment 12 Mike Gilbert gentoo-dev 2013-12-29 00:28:45 UTC
As an aside, you should be generating /boot/grub/grub.cfg instead of /boot/grub2/grub.cfg. The latter was a mistake on my part when I first took over maintenance of the grub:2 ebuild. It continues to work only because of the symlink that is created in pkg_postinst.
Comment 13 David 2014-03-19 18:40:11 UTC
I have experienced this problem (unable to resolve root in an md-array), though it's not because of an incorrect mdadm.conf (mine has no information in it).  When creating the initramfs with genkernel, all works well.  After unmerging genkernel and merging genkernel-next, then recreating the initramfs with the same parameters (except for --disklabel, which genkernel-next does not recognize), the initramfs is unable to resolve root.  When being dumped to a debug shell, blkid shows the two physical disks, but none of the raid logical volumes.
Comment 14 David 2015-02-06 16:45:23 UTC
This bug still appears to be present in genkernel-next-59 and genkernel-next-60.

See http://forums.gentoo.org/viewtopic-p-7697442.html
Comment 15 David Carlos Manuelda 2015-07-31 20:44:52 UTC
I confirm, genkernel-next does something bad, as I have the very same problems.
With:

emerge genkernel-next -p --nodeps

These are the packages that would be merged, in order:

[ebuild   R    ] sys-kernel/genkernel-next-63::gentoo  USE="mdadm -cryptsetup -dmraid -gpg -iscsi -plymouth (-selinux)" 2.234 KiB
Comment 16 David Carlos Manuelda 2015-07-31 20:47:04 UTC
This is a problem because genkernel does not have btrfs support (at least there is no --btrfs option for it), so just a downgrade to "normal" genkernel could be problematic.
Comment 17 Wes 2016-05-09 07:04:40 UTC
Considering genkernel (old) is masked on the systemd profiles now this is a serious issue.  I'm hitting this now with genkernel-next-64 and am entirely unable to bring up an IMSM array using genkernel-next.  Is this bug being worked on at all?
Comment 18 Pacho Ramos gentoo-dev 2016-05-23 14:56:57 UTC
Try to report it to upstream please and post the link here:
https://github.com/Sabayon/genkernel-next/issues