Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 454780

Summary: sys-kernel/genkernel-3.4.45 - initramfs unable to switch on real-root on /dev/sdax
Product: Gentoo Hosted Projects Reporter: Jacques Landru <jacques.landru>
Component: genkernelAssignee: Gentoo Genkernel Maintainers <genkernel>
Status: RESOLVED OBSOLETE    
Severity: normal CC: toon
Priority: Normal    
Version: unspecified   
Hardware: AMD64   
OS: Linux   
Whiteboard:
Package list:
Runtime testing required: ---

Description Jacques Landru 2013-01-31 08:56:25 UTC
After updating stable genkernel from 3.4.24_p2 to 3.4.45 and rebulding kernel and initamfs with "genkernel --oldconfig all", next reboot halts with that message
"the filesystem mounted at /dev/sda10 does not appear to be a valid /,try again"
"Could not find the root block device in ."

/dev/sda10 is an ext4 filesystem
My /usr dir is on /dev/sda10
My /var dir is on another partition of the same disk

What's wrong with my config ?

Looking in linuxrc script: condition for that message display is

if [ -d ${NEW_ROOT}/dev -a -x ${NEW_ROOT}/sbin/init ] || [ "${REAL_R
OOT}" = "/dev/nfs" ]
                        then
                                break
                        else
                                bad_msg "The filesystem mounted at ${REAL_ROOT} does not appear to be a valid /, try again"
                                got_good_root=0
                                REAL_ROOT=''
                        fi
using shell access choice after error message, I check that condition

/dev/sda10 is mounted on /newroot
/newroot/dev is a directory
/newroot/sbin/init is executable

so condition seems OK, and it works before with initramfs from 3.4.24_p2



Reproducible: Always

Steps to Reproduce:
1. update  genkernel to genkernel-3.4.45
2. rebuild kernel and initramfs "genkernel --oldconfig all"
3. restart the host
Actual Results:  
Next reboot hangs during switching to real-root step



Updating to a newer kernel (from gentoo-sources-3.6.6 to gentoo-sources gentoo-souces-3.7.4) produces the same results.

I downgrade to genkernel-3.4.24_p2, masking genkernel-3.4.45
Comment 1 Tom Wijsman (TomWij) (RETIRED) gentoo-dev 2013-01-31 19:57:55 UTC
Importance is to be set by the maintainers.
Comment 2 Richard Yao (RETIRED) gentoo-dev 2013-01-31 21:45:03 UTC
If your /usr dir is on /dev/sda10, where is your /? Also, what is your kernel commandline?
Comment 3 Jacques Landru 2013-02-01 08:20:18 UTC
(In reply to comment #1)
> Importance is to be set by the maintainers.

Oupsss! Sorry, I 'm not familiar with the procedure for reporting bug.
Comment 4 Jacques Landru 2013-02-01 08:30:14 UTC
(In reply to comment #2)
> If your /usr dir is on /dev/sda10, where is your /? Also, what is your
> kernel commandline?

/boot  on /dev/sda6
swap   on /dev/sda7
/tmp   on /dev/sda8
/var   on /dev/sda9
/      on /dev/sda10


display result from "cat /proc/cmdline"

root=/dev/ram0 init=linuxrc ramdisk=8192 real_root=/dev/sda10 udev i915.modset=1

=========== grub.conf default ===================
...
title=Gentoo Linux x86_64 3.7.4  console KMS
# Partition where the kernel image (or operating system) is located
root (hd0,5)
kernel /boot/kernel-genkernel-x86_64-3.7.4-gentoo root=/dev/ram0 init=linuxrc ramdisk=8192 real_root=/dev/sda10 udev i915.modset=1
initrd /boot/initramfs-genkernel-x86_64-3.7.4-gentoo
...
=========== grub.conf default ===================
Comment 5 Xake 2013-02-01 22:13:25 UTC
Try to add "rootfstype=ext4" to your kernel cmdline.

Also, please remove "root=/dev/ram0" and "ramdisk=" and rename your "real_root" to "root".
Comment 6 Xake 2013-02-01 22:15:24 UTC
(In reply to comment #5)
> Try to add "rootfstype=ext4" to your kernel cmdline.
> 

Btw, I do classify this as a workaround, bt have yet to figure out why it sonetimes is needed...
Comment 7 Jacques Landru 2013-03-01 12:13:42 UTC
(In reply to comment #5)
> Try to add "rootfstype=ext4" to your kernel cmdline.
> 
> Also, please remove "root=/dev/ram0" and "ramdisk=" and rename your
> "real_root" to "root".
I made a test with your suggestion : => Same result 
"the filesystem mounted at /dev/sda10 does not appear to be a valid /,try again"

using that new grub conf default section

=========== grub.conf default ===================
title=Gentoo Linux x86_64 3.7.4  console KMS (genkernel 3.4.45)
# Partition where the kernel image (or operating system) is located
root (hd0,5)
kernel /boot/kernel-genkernel-x86_64-3.7.4-gentoo init=linuxrc rootfstype=ext4 root=/dev/sda10 udev i915.modset=1
initrd /boot/initramfs-genkernel-x86_64-3.7.4-gentoo
=========== grub.conf default ===================
Comment 8 Xake 2013-03-03 13:21:53 UTC
(In reply to comment #7)
> (In reply to comment #5)
> > Try to add "rootfstype=ext4" to your kernel cmdline.
> > 
> > Also, please remove "root=/dev/ram0" and "ramdisk=" and rename your
> > "real_root" to "root".
> I made a test with your suggestion : => Same result 
> "the filesystem mounted at /dev/sda10 does not appear to be a valid /,try
> again"
> 
> using that new grub conf default section
> 
> =========== grub.conf default ===================
> title=Gentoo Linux x86_64 3.7.4  console KMS (genkernel 3.4.45)
> # Partition where the kernel image (or operating system) is located
> root (hd0,5)
> kernel /boot/kernel-genkernel-x86_64-3.7.4-gentoo init=linuxrc
> rootfstype=ext4 root=/dev/sda10 udev i915.modset=1
> initrd /boot/initramfs-genkernel-x86_64-3.7.4-gentoo
> =========== grub.conf default ===================

Also, remove "init=linuxrc".  It is not needed anymore as well. It should not change behavior, however.
I am working on getting my dev-machine back up to check if I can spot the change triggering this, but in the mean time, can you attach your /etc/genkernel.conf, dmesg output and anything else you may think we might need?
Comment 9 Xake 2013-03-03 13:25:45 UTC
(In reply to comment #8)
> (In reply to comment #7)
> Also, remove "init=linuxrc".  It is not needed anymore as well. It should
> not change behavior, however.


On a closer look, this might actually be it. The documentation has stated "/linuxrc" which is what we therefore has safeguarded against, while just "linuxrc" might break things. Try this, and if it helps, I will commit a fix.
Comment 10 Jacques Landru 2013-03-04 07:58:39 UTC
(In reply to comment #9)
> (In reply to comment #8)
> > (In reply to comment #7)
> > Also, remove "init=linuxrc".  It is not needed anymore as well. It should
> > not change behavior, however.
> 
> 
> On a closer look, this might actually be it. The documentation has stated
> "/linuxrc" which is what we therefore has safeguarded against, while just
> "linuxrc" might break things. Try this, and if it helps, I will commit a fix.

Yes, removing "init=linuxrc" did it for me. Boot sequence is now as it was with genkernel-3.4.24_p2.
Thank you, for your help.
Comment 11 Xake 2013-03-04 20:20:27 UTC
(In reply to comment #10)
> Yes, removing "init=linuxrc" did it for me. Boot sequence is now as it was
> with genkernel-3.4.24_p2.
> Thank you, for your help.

@genkernel-team,

As this was a newer working configuration to begin with, and the only reason it has worked before was because "init=" was wrongfully ignored, can we just close this as WORKFORME?
Comment 12 toon 2013-03-07 19:53:24 UTC
Hi, I hit this same problem today.
I was upgrading from gentoo-sources 3.2.1-r2 to 3.7.10.
The new genkernel version was 3.4.45.

I decided to check what the checks were that triggered the error message,
so I took the init-script from the old and the new initramfs file and
ran a unified diff on them.  This is the change that caused the problem:

# diff -u -6 initrd-3.2.1/init initrd-3.7.10/init > init.diff

                # If mount is successful break out of the loop
                # else not a good root and start over.
                if [ "$?" = '0' ]
                then
-                       if [ -d ${NEW_ROOT}/dev -a -x ${NEW_ROOT}/sbin/init ] || [ "${REAL_ROOT}" = "/dev/nfs" ]
+                       if [ -d ${NEW_ROOT}/dev -a -x "${NEW_ROOT}${REAL_INIT:-/sbin/init}" ] || [ "${REAL_ROOT}" = "/dev/nfs" ]
                        then
                                break
                        else
                                bad_msg "The filesystem mounted at ${REAL_ROOT} does not appear to be a valid /, try again"
                                got_good_root=0
                                REAL_ROOT=''

I decided to add "real_init=/sbin/init" to the kernel commandline,
and that did the trick; the system booted again.  I use LILO.
These are the old and the new kernel commandlines:

append = "md=0,/dev/sda,/dev/sdb root=/dev/ram0 real_root=/dev/vg1/root init=linuxrc real_init=/sbin/init domdadm dolvm"
append = "md=0,/dev/sda,/dev/sdb root=/dev/ram0 real_root=/dev/vg1/root init=linuxrc real_init=/sbin/init domdadm dolvm"
Comment 13 toon 2013-03-07 19:56:57 UTC
Oops, I goofed with the old and the new kernel command lines.
They are like follows:

OLD:  append = "md=0,/dev/sda,/dev/sdb root=/dev/ram0 real_root=/dev/vg1/root init=linuxrc domdadm dolvm"

NEW:  append = "md=0,/dev/sda,/dev/sdb root=/dev/ram0 real_root=/dev/vg1/root init=linuxrc real_init=/sbin/init domdadm dolvm"
Comment 14 Xake 2013-03-08 06:17:32 UTC
(In reply to comment #13)
> Oops, I goofed with the old and the new kernel command lines.
> They are like follows:
> 
> OLD:  append = "md=0,/dev/sda,/dev/sdb root=/dev/ram0
> real_root=/dev/vg1/root init=linuxrc domdadm dolvm"
> 
> NEW:  append = "md=0,/dev/sda,/dev/sdb root=/dev/ram0
> real_root=/dev/vg1/root init=linuxrc real_init=/sbin/init domdadm dolvm"

init=linuxrc is your problem.
Will post a legnthy explanation on why real_* is discouraged, why your configuration should  have problems to begin with, and why it sucks that when you fix genkernel to actually do what it is supposed to (init= has been a dead flag for quite some time until this last version) it breaks stuff.



But it seems like we need to safeguard from more bad init= values to help users to not shoot their feets. And if any documentation tells you to use root=*ram*, ramdisksize= and *init=*linuxrc in the same phrase as genkernel, please file bugs about it so we may correct them.
Comment 15 toon 2013-03-08 08:00:05 UTC
Hi Xake, thanks for the interesting update!
I would really appreciate it if you could point me at the one and only correct documentation regarding this topic.  Please educate us.

Thanks in advance!

BTW: As you may have noticed from the command line, my setup is quite exotic.
The root partition is located on a pair of RAID 1 mirrored disks (MD).
This mirrored volume is then partitioned using LVM.
One of the logical volumes contains the root partition.
Here is the status output of mdadm:

# mdadm --detail /dev/md127 
/dev/md127:
        Version : 0.90
  Creation Time : Sat May  9 13:54:07 2009
     Raid Level : raid1
     Array Size : 1953514496 (1863.02 GiB 2000.40 GB)
  Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 127
    Persistence : Superblock is persistent

    Update Time : Fri Mar  8 00:47:05 2013
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : 3d748086:28483d98:0f52ad76:84db52de
         Events : 0.1318

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync   /dev/sda
       1       8       16        1      active sync   /dev/sdb
Comment 16 toon 2013-03-08 08:41:00 UTC
Following your advices in this bug report I reduced the command line to:

append = "md=0,/dev/sda,/dev/sdb root=/dev/vg1/root domdadm dolvm"

It works.  Nice cleanup!
Comment 17 Xake 2013-03-08 23:24:05 UTC
(In reply to comment #15)
> Hi Xake, thanks for the interesting update!
> I would really appreciate it if you could point me at the one and only
> correct documentation regarding this topic.  Please educate us.
> 

Well, a quick google showed that a lot of people has done a lot better jobs in doing detailed writeups. So here is the short version:
With linux-2.4 you used a initrd based on a ramdisk. In linux-2.6 the standard became a initramfs based on a ramfs.
The user visible difference is that with the ramdisk the kernel picks up the ramdisk you specified with "root=" of the fixed size "ramdisksize=" and decompress the fixed size filesystem img specified in the bootloader by "initrd" on top of it. Then it tries to execute the file specified by "init=" in said ramdisk.
This as you can see forced the ramdisks created to set up another filesystem to mount had to be creative, thus the "real_*=" variables was invented so that the ramdisk knew what filesystem to hand over to and what init to execute there.

Now linux-2.6 and ramfs/tmpfs came along. And if it finds out that the file specified by initrd is instead a compressed cpio-archive, then it sets up a ramfs of variable size, decompresses the files in the cpio-archive directly into it and (really very simplified) tells it to take care of boot up. This approach does not clutter up root, init and alike, so the real_*= variables was not needed any longer.

Now genkernel have not been able to create anything but initramfs since 2008, but due to lack of devs and mismatch in communications first now these variables have started to become cleaned up.

So init= has been a dead variable since 2008 when used with genkernel, mostly because genkernel generated initramfs ignored it, and the kernel used "rd.init=" to figure out what file to execute in the ramfs. Now 3.4.45 came along and actually started to use it again to specify what file to execute in "root=" unless it is specified as init=/linuxrc (which is what the gentoo handbook still wrongfully tells people to do) so we would not break too many setups. And every system who had a configuration that may have seemed to work before stopped working since genkernel acctually tried using their configurations.