Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 742527 - latest genkernel (-9999 from git) changed md devices naming causing unbootable system
Summary: latest genkernel (-9999 from git) changed md devices naming causing unbootabl...
Status: RESOLVED WONTFIX
Alias: None
Product: Gentoo Hosted Projects
Classification: Unclassified
Component: genkernel (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Gentoo Genkernel Maintainers
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-09-14 17:10 UTC by Mariusz Ceier
Modified: 2021-03-15 21:42 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Mariusz Ceier 2020-09-14 17:10:43 UTC
On the server I use, I had /dev/md2 (mounted at /boot) and /dev/md3 (mounted at /) since the beginning and it worked fine.

Today I decided to reboot the server with updated kernel and initramfs, built with genkernel from git and it didn't boot - it stopped booting in initramfs because it couldn't find the root partition.

It looks like /dev/md2 and /dev/md3 don't exist anymore, and there's only /dev/md126 and /dev/md127... probably due to udev usage ?

I wish genkernel would warn me about such name changes (or at least *strongly* recommend using UUID), so I could fix the fstab entries or grub configuration before rebooting.

I changed my config to use UUID instead and the server boots now.
Comment 1 Ben Kohler gentoo-dev 2020-09-14 17:13:03 UTC
Aren't you passing --mdadm-config= param to include a config which controls the md device numbering?

I don't think udev renames /dev/md* devices..
Comment 2 Mariusz Ceier 2020-09-14 17:22:55 UTC
(In reply to Ben Kohler from comment #1)
> Aren't you passing --mdadm-config= param to include a config which controls
> the md device numbering?
> 
> I don't think udev renames /dev/md* devices..

No, I'm not passing --mdadm-config= anywhere.
Comment 3 Mariusz Ceier 2020-09-14 17:25:29 UTC
(In reply to Ben Kohler from comment #1)
> Aren't you passing --mdadm-config= param to include a config which controls
> the md device numbering?
> 
> I don't think udev renames /dev/md* devices..

I have /etc/mdadm.conf which specifies /dev/md2 and /dev/md3 arrays (with UUIDs) yet the devices in /dev/ are named /dev/md126 and /dev/md127 after rebooting.
Comment 4 Ben Kohler gentoo-dev 2020-09-14 17:33:55 UTC
You should be passing --mdadm-config=/etc/mdadm.conf so that this mdadm.conf is copied into the initramfs and used when the initramfs calls mdadm.
Comment 5 Mariusz Ceier 2020-09-14 17:36:02 UTC
(In reply to Ben Kohler from comment #4)
> You should be passing --mdadm-config=/etc/mdadm.conf so that this mdadm.conf
> is copied into the initramfs and used when the initramfs calls mdadm.

Now I think so too...

In the genkernel.log I see:

    mdadm: >> --mdadm-config not set; Skipping inclusion of mdadm.conf ...

I guess since I never specified --mdadm-config it was pure coincidence that until now the system was booting correctly.

Thanks for fast response. I wonder if genkernel could warn about consequences of not passing --mdadm-config.
Comment 6 Thomas Deutschmann (RETIRED) gentoo-dev 2020-09-15 00:28:14 UTC
Like you already figured out with Ben, there is no change in genkernel causing that. However, I would like to understand why the problem is new to you:

I just tested with latest genkernel-3.5, without mdadm.conf, my MD device will be named /dev/md126 and /dev/md127. And this is also my experience since I am using Gentoo.

This is coming from kernel/mdadm, see https://bugzilla.redhat.com/show_bug.cgi?id=606481 for example which is already 10 years old.

So I am really curious why you hit this today for the first time...
Comment 7 Mariusz Ceier 2020-09-15 04:43:05 UTC
(In reply to Thomas Deutschmann from comment #6)
> Like you already figured out with Ben, there is no change in genkernel
> causing that. However, I would like to understand why the problem is new to
> you:
> 
> I just tested with latest genkernel-3.5, without mdadm.conf, my MD device
> will be named /dev/md126 and /dev/md127. And this is also my experience
> since I am using Gentoo.
> 
> This is coming from kernel/mdadm, see
> https://bugzilla.redhat.com/show_bug.cgi?id=606481 for example which is
> already 10 years old.
> 
> So I am really curious why you hit this today for the first time...

I have no idea why I hit it today - I restarted system few times in the past (I guess about 6) and had no issue until now.

I tried to fix from the rescue system and at first rescue system also had md2 and md3 after reboots.
Inside rescue system I reinstalled grub since I thought it was broken, regenerated grub.cfg, updated genkernel and rebuilt the kernel 5.8.9 (at first I thought ssh service regenerated the keys after boot but it turned out to be false).
Then I noticed I can login into initramfs rescue shell (so grub was fine) and inside it I found out that that /dev/md2 and /dev/md3 were renamed to /dev/md126 and /dev/md127... when I noticed it, after reboot rescue system also had /dev/md126 and /dev/md127 (maybe the name stays after mounting it, dunno).

I decided to try using UUID in fstab for all partitions and regenerate grub.cfg and it worked - system booted. Now I'm using UUID in fstab and --mdadm-config argument to genkernel, so it shouldn't happen anymore.
Comment 8 Mariusz Ceier 2020-09-15 06:57:20 UTC
Previous kernel I was using was 5.6.13, built on 19.05.2020 with Genkernel 4.0.7-3-gce03050-git; and its initramfs init and initrd.scripts has calls to setup_md_device.

I'm running 5.8.9 kernel now and inside initramfs built with Genkernel 4.1.2-10-g076201c-git, there's no setup_md_device. 

I think this makes the difference... previously grub-mkconfig added root=/dev/md3 to the kernel parameters and setup_md_device was creating /dev/md3 with mknod.




> setup_md_device() {
>        local device
> 
>        [ -z "$1" ] && device="${REAL_ROOT}" || device="$1"
>        [ -z "${device}" ] && return # LiveCD
> 
>        if [ $(echo ${device}|sed -e 's#\(luks:\)\?\(/dev/md\)[[:digit:]]\+#\2#') = "/dev/md" ]
>        then
>                good_msg 'Detected real_root as a md device. Setting up the device node ...'
>                MD_NUMBER=$(echo ${device}|sed -e 's#\(luks:\)\?/dev/md\([[:digit:]]\+\)#\2#')
>                if [ ! -e /dev/md${MD_NUMBER} ]
>                then
>                        run mknod /dev/md${MD_NUMBER} b 9 ${MD_NUMBER} >/dev/null 2>&1
>                        [ $? -ne 0 ] && bad_msg "Creation of /dev/md${MD_NUMBER} failed ..."
>                fi
>                run raidautorun /dev/md${MD_NUMBER}
>          fi
> }
Comment 9 Thomas Deutschmann (RETIRED) gentoo-dev 2021-03-15 21:42:03 UTC
You are right, this change was introduce with commit https://gitweb.gentoo.org/proj/genkernel.git/commit/?id=f6f9384b423e9bb9b8cee294f4ddbeee7c518463 which removed setup_md_device() in favor of udev.

But I am closing this issue as WONTFIX in favor of our move to UDEV. If you depend on fixed device names, please do that via --mdadm-config option, i.e. provide /etc/mdadm.conf which will set up /dev/md/<your name> nodes which works without any third party support.