Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 479730

Summary: sys-kernel/genkernel: systemd and separate /usr causes failing of REAL_ROOT check (summary in comment #6)
Product: Gentoo Hosted Projects Reporter: stareagle
Component: genkernelAssignee: Gentoo Genkernel Maintainers <genkernel>
Status: RESOLVED FIXED    
Severity: normal CC: floppym, pberuto, pravorskii.andrii, rose, systemd, williamh, zerochaos
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
See Also: https://bugs.gentoo.org/show_bug.cgi?id=479274
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 468942, 620846    
Attachments: /usr/local/portage/sys-kernel/genkernel/files/genkernel-3.4.47_systemd.patch
/usr/local/portage/sys-kernel/genkernel/genkernel-3.4.47.ebuild
/usr/local/portage/sys-kernel/genkernel/files/genkernel-3.4.47_systemd.patch

Description stareagle 2013-08-04 17:28:29 UTC
The linuxrc script used by an initramfs created with genkernel uses the following block (located around line 645, location varies betwenn the different genkernel versions) the check if the root filesystem is sane:

if [ -d ${NEW_ROOT}/dev -a -x "${NEW_ROOT}${REAL_INIT:-/sbin/init}" ] || [ "${REAL_ROOT}" = "/dev/nfs" ]
then
    break
else
    bad_msg "The filesystem mounted at ${REAL_ROOT} does not appear to be a valid /, try again"
    got_good_root=0
    REAL_ROOT=''
fi 

When using systemd as init, the $REAL_INIT variable has the value /usr/lib/systemd/systemd. Now if /usr is located on a seperate partion, the check fails because is not mounted when the check is done. The boot then halts with the message: "The filesystem mounted at /dev/xxx does appear to be a valid /, try again.

If I remove the check, the system boots, with systemd as init.

This issues appear with the stable genkernel version, the lastest ~arch version and with the lastest genkernel-next version.
Comment 1 Ulenrich 2013-08-05 09:10:14 UTC
dracut is especially meant for this setup:
module usrmount
Comment 2 Juergen Rose 2013-08-05 16:44:05 UTC
(In reply to stareagle from comment #0)
> The linuxrc script used by an initramfs created with genkernel uses the
> following block (located around line 645, location varies betwenn the
> different genkernel versions) the check if the root filesystem is sane:
...
> When using systemd as init, the $REAL_INIT variable has the value
> /usr/lib/systemd/systemd. Now if /usr is located on a seperate partion, the
> check fails because is not mounted when the check is done. The boot then
> halts with the message: "The filesystem mounted at /dev/xxx does appear to
> be a valid /, try again.
> 
> If I remove the check, the system boots, with systemd as init.
> 
> This issues appear with the stable genkernel version, the lastest ~arch
> version and with the lastest genkernel-next version.

Do you have a patch for genkernel to remove the check? I would like very much to be able to boot my systems with systemd.
Comment 3 mpcww 2013-08-05 19:31:19 UTC
As a workaround I commented out the lines described by stareagle leaving just the break in /usr/share/genkernel/defaults/linuxrc :

643 #     if [ -d ${NEW_ROOT}/dev -a -x "${NEW_ROOT}${REAL_INIT:-/sbin/init}" ] || [ "${REAL_ROOT}" = "/dev/nfs" ]
    644 #                       then
    645                                 break
    646 #                       else
    647 #                               bad_msg "The filesystem mounted at ${REAL_ROOT} does not appear to be a valid /, try again"
    648 #                               got_good_root=0
    649 #                               REAL_ROOT=''
    650 #                       fi
Comment 4 Fabio Erculiani (RETIRED) gentoo-dev 2013-08-05 20:50:33 UTC
So you're using separate /usr with systemd? That's looking for trouble, systemd does not support separate /usr.
However, I also think that genkernel is trying to be too smart here...
Comment 5 Mike Gilbert gentoo-dev 2013-08-05 21:25:28 UTC
(In reply to Fabio Erculiani from comment #4)
> So you're using separate /usr with systemd? That's looking for trouble,
> systemd does not support separate /usr.

Sure it does, assuming you get /usr mounted before you execute systemd.
Comment 6 Mike Gilbert gentoo-dev 2013-08-05 21:29:44 UTC
(In reply to Mike Gilbert from comment #5)

To restate the problem: genkernel's linuxrc is checking for the presence of the specified init just after the root filesystem is mounted, but this is too soon; it needs to mount all of the necessary file systems before checking to see if $REAL_INIT is present and executable.
Comment 7 Juergen Rose 2013-08-06 10:28:43 UTC
(In reply to mpcww from comment #3)
> As a workaround I commented out the lines described by stareagle leaving
> just the break in /usr/share/genkernel/defaults/linuxrc :
> 
> 643 #     if [ -d ${NEW_ROOT}/dev -a -x
> "${NEW_ROOT}${REAL_INIT:-/sbin/init}" ] || [ "${REAL_ROOT}" = "/dev/nfs" ]
>     644 #                       then
>     645                                 break
>     646 #                       else
>     647 #                               bad_msg "The filesystem mounted at
> ${REAL_ROOT} does not appear to be a valid /, try again"
>     648 #                               got_good_root=0
>     649 #                               REAL_ROOT=''
>     650 #                       fi

I tried to convert this changes to a patch file for genkernel. With this patch I first reemerged genkernel and then generated with genkernel a new initramfs. If I boot with this initramfs, it fails with:

>> Using mount -t ext3 -o ro
mount: mounting /dev/md125 on /newroot failed: Device or resource busy
!! Could not mount specified ROOT, try again
!! Please file a bug report with this message
[   **.******] kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
Comment 8 Juergen Rose 2013-08-06 10:30:44 UTC
Created attachment 355224 [details, diff]
/usr/local/portage/sys-kernel/genkernel/files/genkernel-3.4.47_systemd.patch
Comment 9 Juergen Rose 2013-08-06 10:32:07 UTC
Created attachment 355226 [details]
/usr/local/portage/sys-kernel/genkernel/genkernel-3.4.47.ebuild

An ebuild to use the systemd patch.
Comment 10 Fabio Erculiani (RETIRED) gentoo-dev 2013-08-06 11:21:50 UTC
(In reply to Mike Gilbert from comment #5)
> (In reply to Fabio Erculiani from comment #4)
> > So you're using separate /usr with systemd? That's looking for trouble,
> > systemd does not support separate /usr.
> 
> Sure it does, assuming you get /usr mounted before you execute systemd.

Yes, sorry for the mistake, I misread the problem.
Ok, then I believe that the code mounting the entries in initramfs.mount should be moved before (or in the middle of) the REAL_ROOT mount code.
Comment 11 Juergen Rose 2013-08-06 11:43:54 UTC
Created attachment 355232 [details, diff]
/usr/local/portage/sys-kernel/genkernel/files/genkernel-3.4.47_systemd.patch
Comment 12 Juergen Rose 2013-08-06 12:16:31 UTC
(In reply to Juergen Rose from comment #7)
> (In reply to mpcww from comment #3)
> > As a workaround I commented out the lines described by stareagle leaving
> > just the break in /usr/share/genkernel/defaults/linuxrc :
> > 
> > 643 #     if [ -d ${NEW_ROOT}/dev -a -x
> > "${NEW_ROOT}${REAL_INIT:-/sbin/init}" ] || [ "${REAL_ROOT}" = "/dev/nfs" ]
> >     644 #                       then
> >     645                                 break
> >     646 #                       else
> >     647 #                               bad_msg "The filesystem mounted at
> > ${REAL_ROOT} does not appear to be a valid /, try again"
> >     648 #                               got_good_root=0
> >     649 #                               REAL_ROOT=''
> >     650 #                       fi
> 
> I tried to convert this changes to a patch file for genkernel. With this
> patch I first reemerged genkernel and then generated with genkernel a new
> initramfs. If I boot with this initramfs, it fails with:
> 
> >> Using mount -t ext3 -o ro
> mount: mounting /dev/md125 on /newroot failed: Device or resource busy
> !! Could not mount specified ROOT, try again
> !! Please file a bug report with this message
> [   **.******] kernel panic - not syncing: Attempted to kill init!
> exitcode=0x00000100

Sorry, I used the wrong comment symbols in the first submitted patch. With the corrected patch the booting is a little bit better. It comes until
...
[ TIME ] Timed out waiting for device dev-mapper-vg1\x2dsrc.device
[DEPEND] Dependency failed for /usr/src_condor
[DEPEND] Dependency failed for Local File Systems.
[ ***.******] systemd[1]: Job dev-mapper-vg1\x2dportage.device/start timed out
[ ***.******] systemd[1]: Job dev-mapper-vg1\x2dvtmp.device/start timed out
[ ***.******] systemd[1]: Job dev-mapper-vg1\x2dftp.device/start timed out
[ ***.******] systemd[1]: Job dev-mapper-vg1\x2dopt.device/start timed out
[ ***.******] systemd[1]: Job dev-mapper-vg1\x2dtmp.device/start timed out
Welcome to emergency mode! After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" to try again 
to boot into default mode.
Give root password for maintenance


If I login as root, I see that /, /dev/mapper/vg1-usr as /usr and two further disk partitions are mounted.

Performing 'journalctl -xb' I see:
...
/etc/mtab is not a symlink or not pointing to /proc/self/mount...
...
Cannot add dependency job for unit display-manager.service, ignoring: Unit display-manager.service failed to load: No such file or directory.
...
failed to execute '/lib/udev/usbseat.sh' 'usbseat.sh': No such file or directory
...
Job dev-mapper-vg1\x2dsrc.device/start timed out.
Timed out waiting for device dev-mapper-vg1\x2dsrc.device
Dependency failed for /usr/src_condor
Dependency failed for Local File Systems
Triggering OnFailure= dependencies of local-fs.target.
Dependency failed for File Systems Check on /dev/mapper/vg1-src/
...

followed by similar messages for the other logical volumes. I.e., also /var is not mounted

...
Failed to start Console System Startup Logging.



'systemctl default' reports again that Job dev-mapper-vg1\x2dusr.device/start timed out, followed by corresponding messages about the other logical volumes


Any hint is appreciated.
Comment 13 Fabio Erculiani (RETIRED) gentoo-dev 2013-08-06 12:19:51 UTC
(In reply to Juergen Rose from comment #12)
Please, one bug = one issue. This is a separate bug and has to do with genkernel using mdev instead of udev, which makes lvm kaputt. See bug #424637, the only way to get it to work is (at this time) using genkernel-next and run it with --udev so that udev gets embedded into the initramfs and enabled at runtime.
Comment 14 Juergen Rose 2013-08-06 12:28:03 UTC
Thanks Fabio for the hint.
Comment 15 Fabio Erculiani (RETIRED) gentoo-dev 2013-08-26 10:37:46 UTC
This should be fixed in the latest genkernel-next release (v24).
Comment 16 Pacho Ramos gentoo-dev 2013-09-11 11:22:14 UTC
(This was solved in genkernel-next)
Comment 17 Pacho Ramos gentoo-dev 2013-09-15 19:20:05 UTC
(In reply to Pacho Ramos from comment #16)
> (This was solved in genkernel-next)

(Drop blocker as we will tell people to use genkernel-next when using systemd)
Comment 18 Mike Gilbert gentoo-dev 2014-04-26 13:04:02 UTC
*** Bug 508678 has been marked as a duplicate of this bug. ***
Comment 19 Rick Farina (Zero_Chaos) gentoo-dev 2015-08-20 16:26:39 UTC
I would accept a patch to move the check until after /usr is mounted, by simply removing the check doesn't seem like a great idea just to support such an usual setup.
Comment 20 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2017-07-14 18:38:16 UTC
Yes, that specific check should be deferred until /usr is mounted for systemd, but I think we need to replace it's present position with some other test to test that we found a filesystem usable for booting. It should be a test that ALSO fails if the blockdev for /usr is accidentally passed as rootfs.

I suggest:
test -x /bin/sh && test -e /etc/fstab
Comment 21 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2018-08-09 20:33:02 UTC
williamh:
opinion on detecting if a filesystem is a good candidate for the rootfs (before /usr gets mounted into it.

Existing test:
test -d ${NEWFS}/dev && \
test -x ${NEWFS}${REAL_INIT:-/sbin/init}

Proposed:
test -x ${NEWFS}${REAL_INIT:-/sbin/init} || \
test -L ${NEWFS}${REAL_INIT:-/sbin/init} || \
test -x ${NEWFS}/bin/sh
(followed by a better test for init after /usr is mounted)
Comment 22 Larry the Git Cow gentoo-dev 2019-07-15 15:37:08 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/proj/genkernel.git/commit/?id=76a837bc21ffd8a7d350cd3c648947f45e6a07a7

commit 76a837bc21ffd8a7d350cd3c648947f45e6a07a7
Author:     Thomas Deutschmann <whissi@gentoo.org>
AuthorDate: 2019-07-15 15:30:39 +0000
Commit:     Thomas Deutschmann <whissi@gentoo.org>
CommitDate: 2019-07-15 15:30:39 +0000

    linuxrc: Relax NEWROOT validation
    
    If the root device specified by the user is invalid then all
    we are going to lose is the possibility to ask for a new root
    device in a nice way. So no need to be smart here.
    
    Closes: https://bugs.gentoo.org/479730
    Signed-off-by: Thomas Deutschmann <whissi@gentoo.org>

 defaults/linuxrc | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)