Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 424637

Summary: sys-apps/systemd won't work with lvm due to udev and genkernel problems
Product: Gentoo Linux Reporter: Kirill Elagin <kirelagin>
Component: [OLD] Core systemAssignee: Gentoo systemd Team <systemd>
Status: RESOLVED NEEDINFO    
Severity: normal CC: alexander, erkiferenc, genkernel, gentoo-bugs, gentoo, granger.damien, kirelagin, nikoli, pacho, renegart, yamadharma
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
See Also: https://bugs.freedesktop.org/show_bug.cgi?id=50365
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on: 453594, 468942    
Bug Blocks: 468898, 478252    

Description Kirill Elagin 2012-07-03 19:02:04 UTC
I can't boot my system with systemd as activating lvm volumes times out.

I'm still not sure, but, it seems to me, that the problem is that lvm-volumes-related entries in udev db are not populated as they sould be and they have SYSTEMD_READY=0.

===
P: /devices/virtual/block/dm-4
N: dm-4
E: DEVNAME=/dev/dm-4
E: DEVPATH=/devices/virtual/block/dm-4
E: DEVTYPE=disk
E: DM_SBIN_PATH=/sbin
E: DM_UDEV_DISABLE_DISK_RULES_FLAG=1
E: DM_UDEV_DISABLE_OTHER_RULES_FLAG=1
E: DM_UDEV_DISABLE_SUBSYSTEM_RULES_FLAG=1
E: MAJOR=254
E: MINOR=4
E: SUBSYSTEM=block
E: SYSTEMD_READY=0
E: TAGS=:systemd:
E: UDISKS_PRESENTATION_NOPOLICY=1
E: USEC_INITIALIZED=13164341
===

This seems to be a Gentoo-specific issue (please, see upstream bug https://bugs.freedesktop.org/show_bug.cgi?id=50365) and, since lvm2 is clearly built with USE="udev", sems that this bug is closely related to bug #330651 which is IN_PROGRESS for more than a year.

I'm pretty sure that the best way to resolve this is replacing mdev with udev in genkernel's initramfs since udev is a really important part of any system and activating some part of hardware without udev is not, IMHO, a good idea.

Maybe there are other ways to fix this, I don't know. Some missing symlinks and empty db are fine and no one notices this, unless you want to use systemd (and I hope Gentoo will switch to it one day).
Comment 1 Kirill Elagin 2012-07-03 22:14:34 UTC
Ok, so, I'm trying to use dracut, but, guess what, its lvm module relies on udev db to detect lvm volumes and enable itself.
I'm a bit confused right now, because to get a good initramfs I need a good initramfs. I guess I'll insert dirty hacks into dracut's lvm module to make it believe it is needed. But, man, if Gentoo decides to support systemd with lvm one day, this migration won't be easy!
Comment 2 Xake 2012-07-06 11:17:41 UTC
(In reply to comment #1)
> Ok, so, I'm trying to use dracut, but, guess what, its lvm module relies on
> udev db to detect lvm volumes and enable itself.


Try run "udevadm trigger" as root before running dracut.

Also, try running "systemctl enable udev-settle.device".
Comment 3 Kirill Elagin 2012-07-06 11:24:48 UTC
(In reply to comment #2)
> Try run "udevadm trigger" as root before running dracut.
If you read through bug #330651, you'll understand why this doesn't help.

> Also, try running "systemctl enable udev-settle.device".
Actually, /etc/init.d/udev does `udevadm settle` and, of course, this doesn't help.


By the way, I've already successfully switched to dracut+systemd and everything works very well except for udev-settle.service which I had to enable to make lvm work. I hope to get rid of this.
Comment 4 Xake 2012-07-07 06:53:34 UTC
(In reply to comment #3)
> (In reply to comment #2)
> > Try run "udevadm trigger" as root before running dracut.
> If you read through bug #330651, you'll understand why this doesn't help.
> 

Do you use devtmpfs?
Comment 5 Kirill Elagin 2012-07-07 09:32:46 UTC
(In reply to comment #4)
> Do you use devtmpfs?
I do. And I have CONFIG_DEVTMPFS_MOUNT=y.
Comment 6 Fabio Erculiani (RETIRED) gentoo-dev 2013-05-07 20:17:55 UTC
I have created a new package called sys-kernel/genkernel-next (which is now sitting in the systemd-love overlay and will be moved to tree soon).

genkernel-next is an improved version of genkernel, which implements (and drops) support for:

+ udev
+ plymouth
+ embedding system packages rather than "hand" compiling them
+ speed improvements
+ improved code quality and stability
- cross compilation (broken for ages anyway)
Comment 7 Marcin Kowalski 2013-07-07 11:56:12 UTC
perhaps this might solve the behavior ? 

https://bugs.gentoo.org/show_bug.cgi?id=453594
Comment 8 Fabio Erculiani (RETIRED) gentoo-dev 2013-07-08 06:21:41 UTC
No, you need udev in the initramfs to get lvm working.
Comment 9 Pacho Ramos gentoo-dev 2013-07-27 06:51:29 UTC
If I don't misremember, genkernel team was happy to implement your patches, how did it end finally?
Comment 10 Richard Yao (RETIRED) gentoo-dev 2013-07-27 07:00:41 UTC
(In reply to Pacho Ramos from comment #9)
> If I don't misremember, genkernel team was happy to implement your patches,
> how did it end finally?

It has not ended yet. Right now, one of us needs to sit down, take the patch and adapt it to genkernel HEAD. I will try to do that tomorrow. Poke me in IRC if I do not.
Comment 11 Pacho Ramos gentoo-dev 2013-07-27 07:05:27 UTC
Thanks a lot ryao
Comment 12 Damien Granger 2013-07-30 18:47:23 UTC
Hi, to bring this bug back on systemd a bit, my feeling is that genkernel-next can't fix it alone. Just tried today to update to systemd-206 with genkernel-next-19 and I just broke my box... (gentoo-sources-3.10.0)

The error was "/dev/mapper/vg00-root does not appear to be a valid /".

This error lies (IMHO, please correct me if I'm wrong) in a bad initrd. I tried also the stable systemd-204 with no more success.

So I'm wondering what are the differences between systemd in main tree and working systemd with my root-on-lvm box in systemd-love overlay.

   * >=sys-apps/systemd-205 dropped static-libs USE flag :
Obviously not the problem here as 204 still has it, but I think it might be an issue anyway in the long run

   * Main tree systemd doesn't provide 40-gentoo.rules
Can't see why it would be an issue as this file only provides usb and mem subsystem rules

   * Inclusion of a specific blacklist.conf
Can't see why either, only blacklisting specific modules that have nothing to do with LVM

These two are the tswo things genkernel complains about when building the initrd...

I won't talk about all the stuff about the init itself because obviously I don't seem to get to that point (and the symlink still points to systemd binary)

So if you have any idea of what's causing this unwanted behavior, I'd gladly switch back to main tree to continue and try a pure systemd box without overlay.

Of course if you need any info, just ask!
Comment 13 Martin Wegner 2013-08-01 10:50:00 UTC
I also have root (and other partitions, e.g. for /var) on LVM and experienced this issue and I solved it by booting an initramfs generated with genkernel-next.

However, this stopped working for me with (at least) updating the kernel to gentoo-sources-3.10.4 (I do not know if other updates might have been involved). I'm currently getting the timeouts for the LVM devices again.

All dm devices in the udev db show SYSTEMD_READY=0 (again? can't verify), e.g.:

P: /devices/virtual/block/dm-0
N: dm-0
E: DEVNAME=/dev/dm-0
E: DEVPATH=/devices/virtual/block/dm-0
E: DEVTYPE=disk
E: DM_UDEV_DISABLE_DISK_RULES_FLAG=1
E: DM_UDEV_DISABLE_OTHER_RULES_FLAG=1
E: DM_UDEV_DISABLE_SUBSYSTEM_RULES_FLAG=1
E: MAJOR=254
E: MINOR=0
E: SUBSYSTEM=block
E: SYSTEMD_READY=0
E: TAGS=:systemd:
E: UDISKS_PRESENTATION_NOPOLICY=1
E: USEC_INITIALIZED=283616

So I don't know exactly what changed (despite kernel version) but I booted successfully with initramfs generated by genkernel-next and gentoo-sources-3.10.3 yesterday, and today it is broken again.

If I can do anything to debug this further or to provide more or more accurate information, let me know.

$ emerge -pv systemd genkernel-next 
[ebuild   R    ] sys-kernel/genkernel-next-22::systemd-love  USE="crypt cryptsetup -dmraid -gpg (-ibm) -iscsi -plymouth (-selinux)" 2,264 kB
[ebuild   R    ] sys-apps/systemd-206-r1  USE="cryptsetup filecaps firmware-loader gudev introspection kmod pam policykit tcpd -acl -audit -doc -gcrypt -http -lzma -openrc -python -qrcode (-selinux) {-test} -vanilla -xattr" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7" 2,286 kB

Despite the systemd version above, this also happened with systemd-206 before.
Comment 14 Martin Wegner 2013-08-01 11:17:42 UTC
Nevermind, it was my fault: I compared the ramdisks and found out that I forgot to pass the --udev switch when generating the initramfs for gentoo-sources-3.10.4. Sorry for the noise.

So with genkernel-next and including udev in the initramfs, it works for me and the systemd timeouts on the LVM devices disappear.
Comment 15 Damien Granger 2013-08-01 11:23:28 UTC
Okay, so I'll give another shot at it if it works for you, but I still can't find out why I get this error.

I compile my kernel with genkernel --lvm --udev --install --kernel-config=<...> all.

The message "/dev/mapper/vg00-root does not appear to be a valid /" is not verbose enough for me.
Comment 16 Damien Granger 2013-08-01 12:07:03 UTC
Tried it, it works actually. I still have a lot in systemd-love overlay (mainly for service files) but it's slowly getting into the main tree!

I found out that the symlink in /sbin/init still existed but pointed to a location where there was no files, hence the "not valid /" as there was no init binary.

So I guess the next step will be to include eselect-init in main-tree but that's another story ;)
Comment 17 Juergen Rose 2013-08-06 20:27:29 UTC
(In reply to Damien Granger from comment #15)
> Okay, so I'll give another shot at it if it works for you, but I still can't
> find out why I get this error.
> 
> I compile my kernel with genkernel --lvm --udev --install
> --kernel-config=<...> all.
> 
> The message "/dev/mapper/vg00-root does not appear to be a valid /" is not
> verbose enough for me.

I used today genkernel-next from the normal portage tree (genkernel-next-18).
I generated the kernel and the initramfs with

genkernel --oldconfig --udev all

In /etc/default/grub I have:

GRUB_CMDLINE_LINUX_DEFAULT="dolvm root=/dev/sda3 real_init=/usr/lib/systemd/systemd rootfstype=ext4"

The next booting fails with as before with:
!! The filesystem mounted at /dev/sda3 does not appear to be a valid /, try again
!! Please file a bug report...

I.e., I am again at https://bugs.gentoo.org/show_bug.cgi?id=479730 .
Comment 18 Juergen Rose 2013-08-08 19:40:52 UTC
(In reply to Juergen Rose from comment #17)
...
> I used today genkernel-next from the normal portage tree (genkernel-next-18).
> I generated the kernel and the initramfs with
> 
> genkernel --oldconfig --udev all
> 
> In /etc/default/grub I have:
> 
> GRUB_CMDLINE_LINUX_DEFAULT="dolvm root=/dev/sda3
> real_init=/usr/lib/systemd/systemd rootfstype=ext4"
> 
> The next booting fails with as before with:
> !! The filesystem mounted at /dev/sda3 does not appear to be a valid /, try
> again
> !! Please file a bug report...
> 
> I.e., I am again at https://bugs.gentoo.org/show_bug.cgi?id=479730 .

I tried to modify the patch from https://bugs.gentoo.org/show_bug.cgi?id=479730 and apply now for genkernel-next. If I then generate a new kernel and a initramfs with this patched genkernel-next and boot with the corresponding default kernel and initramfs, the issue with the invalid root disappeared and I get now:
...
[   *.****] udevd (856) used greatest stack depth: 4208 bytes left
[   *.****] udevd (854) used greatest stack depth: 4016 bytes left
switch_root: can't execute '/usr/lib/systemd/systemd': No such file or directory
[   *.****] switch_root (1) used greatest stack depth: 3504 bytes left
[   *.****] Kernel panic - not syncing: Attempted to kill init! exitcode =0x00000100
[   *.****]
[   *.****] CPU: 1 PID: 1 Comm: switch_root Not tainted 3.10.5-gentoo-r1 #1
...


If I boot with the same kernel in the recovery mode, the first error message seems to be:
[   *.****] EXT3-fs (sda3): error: couldn't mount because of unsupported optional features (240)
[   *.****] EXT2-fs (sda3): error: couldn't mount because of unsupported optional features (240)
[   *.****] EXT4-fs (sda3): mounted filesystem with ordered data mode. Opts: (null)
>> Mounting /dev/mapper/vg0-usr as /usr: mount -t ext4 -o noatime,ro /dev/mapper/vg0-usr /newroot/usr
mount: mounting /dev/mapper/vg0-usr on /newroot/usr failed: No such file or directory
!! Unable to mount /dev/mapper/vg0-usr for /usr
>> Booting (initramfs)
...
INIT: version 2.88 booting

  OpenRC 0.11.8 is starting up Gentoo Linux (x86_64)

* /proc is already mounted
* /run/openrc: creating directory
...
* Setting up the Logical Volume Manager...
/sbin/lvm: error while loading shared libraries: libudev.so.1: ... No such file or directory
* Failed to setup the LVM
...
This is followed by statements about missing LVM devices (/dev/vg0/home etc.)
The I get a lot of messages about missing files (e.g. /usr/sbin/acpid, /usr/bin/dbus-uuidgen, libgssglue.so.1) and the failing start of several services.

At the end I get login prompt and can login. But as expected only the root device is mount and the most commands (e.g. pvscan) are failing due to missing of shared libraries.
Comment 19 Juergen Rose 2013-08-14 11:39:10 UTC
Is here anybody, who was able to boot with systemd and with a separate /usr partition on a lvm volume?
Comment 20 Mike Gilbert gentoo-dev 2013-08-14 14:43:26 UTC
(In reply to Juergen Rose from comment #19)
> Is here anybody, who was able to boot with systemd and with a separate /usr
> partition on a lvm volume?

Bugzilla is not a support forum.

You are running into a combination of this bug and bug 479730, neither of which has been resolved.
Comment 21 Pacho Ramos gentoo-dev 2013-09-11 11:21:22 UTC
Going back to original problem, what is pending here when using genkernel-next?
Comment 22 Pacho Ramos gentoo-dev 2013-09-15 19:04:33 UTC
(In reply to Pacho Ramos from comment #21)
> Going back to original problem, what is pending here when using
> genkernel-next?