Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 706434 - sys-kernel/genkernel-4.0.2: mount on boot failed for device timeout, related to udev and systemd?
Summary: sys-kernel/genkernel-4.0.2: mount on boot failed for device timeout, related ...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal with 4 votes (vote)
Assignee: Gentoo Genkernel Maintainers
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-01-26 13:01 UTC by giskard
Modified: 2020-08-16 16:25 UTC (History)
19 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description giskard 2020-01-26 13:01:27 UTC
issue:
	systemd mount for fstab failed for device timeout and drop to the emergency
	shell, and the lvm partition uuid not present at the time

	to recover normal boot (from the emergency shell):
	. lvchange -an vg_rt/home
	. lvchange -ay vg_rt/home
		- then the symbol link under /dev/disk/by-*  get auto created
	. systemctl default
		- evrything works well


	the issue happend after I resize the partion size of "root" and recreate
	"home" online (btrfs resize + lvm resize), and still exist after recovered
	from previous subvolume snapshot of btrfs

	it seems to be related to udev support (but it indeed works several times
	since the fresh installation several days ago). I have tried to re-generate
	initramfs with genkernel-next, it boot successful (but not works well after
	log into gnome, hang freequently)


partition latout:
	efi, boot, luks/lvm/{root,home,swap}

	sda                8:0    0 119.2G  0 disk
	├─sda1             8:1    0   127M  0 part	/boot/efi
	├─sda2             8:2    0   128M  0 part	/boot
	└─sda3             8:3    0   119G  0 part
	  └─root         252:0    0   119G  0 crypt
	    ├─vg_rt-root 252:1    0    48G  0 lvm   /
	    ├─vg_rt-home 252:2    0    64G  0 lvm   /home
	    └─vg_rt-swap 252:3    0     4G  0 lvm   [SWAP]


related info (maybe):
. lsblk -f:
	- no UUID for lvm LVs, even for the mounted root

. ls -l /dev/disk/by-*:
	- no links to the lvm LVs

	- mount manually from the emergency shell works if the mountpoint is
	  different to the path in fstab (systemd private mount?)

. journalctl -u systemd-udevd.service:
	- dm-0: Conflicting device node '/dev/mapper/root' found, link to
	  '/dev/dm-0' will not be created.


maybe related
. https://github.com/systemd/systemd/issues/11255
. https://unix.stackexchange.com/questions/330094/udev-rule-to-mount-disk-does-not-work
. https://github.com/lvmteam/lvm2/issues/24

installed packages:
	sys-fs/lvm2-2.02.184-r5
	sys-apps/systemd-243-r2
	sys-kernel/genkernel-4.0.2
	sys-kernel/gentoo-sources-5.4.13
Comment 1 Jonas Stein gentoo-dev 2020-01-26 13:20:11 UTC
It is sad to read that you have problems with the software. The situation seems to be a bit more complicate and requires some analysis.
We can not help you efficiently via bug tracker. The bug tracker aims rather on specific problems in .ebuilds and less on individual systems. 

I have had very good experience on the gentoo IRC [1] with questions like this. Of course there are also forums and mailing lists [2,3].
I hope you understand, that I will close the bug here therefore and wish you good luck on one of the mentioned channels [4].
Please reopen the ticket in order to provide an indication for an specific error in an ebuild or any gentoo related product.

[1] https://www.gentoo.org/get-involved/irc-channels/
[2] https://forums.gentoo.org/
[3] https://www.gentoo.org/get-involved/mailing-lists/all-lists.html
[4] https://www.gentoo.org/support/
Comment 2 giskard 2020-01-26 13:59:47 UTC
(In reply to Jonas Stein from comment #1)
> It is sad to read that you have problems with the software. The situation
> seems to be a bit more complicate and requires some analysis.
> We can not help you efficiently via bug tracker. The bug tracker aims rather
> on specific problems in .ebuilds and less on individual systems. 
> 
> I have had very good experience on the gentoo IRC [1] with questions like
> this. Of course there are also forums and mailing lists [2,3].
> I hope you understand, that I will close the bug here therefore and wish you
> good luck on one of the mentioned channels [4].
> Please reopen the ticket in order to provide an indication for an specific
> error in an ebuild or any gentoo related product.
> 
> [1] https://www.gentoo.org/get-involved/irc-channels/
> [2] https://forums.gentoo.org/
> [3] https://www.gentoo.org/get-involved/mailing-lists/all-lists.html
> [4] https://www.gentoo.org/support/

sorry there is some issue with the network and I can't connect the irc channel. And it's very likely to be a bug with genkernel, thus it's here, it might be a reference for the other guys too
Comment 3 Thomas Deutschmann (RETIRED) gentoo-dev 2020-01-26 15:23:39 UTC
It's unclear if you used genkernel or genkernel-next. Genkernel-next has *nothing* to do with genkernel.

You are saying that your sda3 is encrypted. Because systemd was able to take control, which resides on LVM volume within that LUKS volume, genkernel did its job (making / available to hand off control to real init system, systemd in your case). So you have a systemd problem (or to be more precise: a configuration problem). Because this is not a bug and you are looking for user support, I am closing this report as INVALID.

Please use appropriate channels like gentoo-user mailing list, forums or IRC.
Comment 4 giskard 2020-01-26 15:50:02 UTC
(In reply to Thomas Deutschmann from comment #3)
> It's unclear if you used genkernel or genkernel-next. Genkernel-next has
> *nothing* to do with genkernel.
> 
> You are saying that your sda3 is encrypted. Because systemd was able to take
> control, which resides on LVM volume within that LUKS volume, genkernel did
> its job (making / available to hand off control to real init system, systemd
> in your case). So you have a systemd problem (or to be more precise: a
> configuration problem). Because this is not a bug and you are looking for
> user support, I am closing this report as INVALID.
> 
> Please use appropriate channels like gentoo-user mailing list, forums or IRC.

thanks! Indeed initramfs' job is to mount fs and exec init program , it was done there at the time, but strange that everything be the same with only initramfs from genkernel and genkernel-next, with genkernel everything works well except for the boot mount process, with genkernel-next boot successful but hang frequently..

Well, looks like I can can enter the IRC channel with another webclient now, I will  go there for some help after having the dinner, thank you!
Comment 5 giskard 2020-01-29 16:24:55 UTC
(In reply to Thomas Deutschmann from comment #3)
> It's unclear if you used genkernel or genkernel-next. Genkernel-next has
> *nothing* to do with genkernel.
> 
> You are saying that your sda3 is encrypted. Because systemd was able to take
> control, which resides on LVM volume within that LUKS volume, genkernel did
> its job (making / available to hand off control to real init system, systemd
> in your case). So you have a systemd problem (or to be more precise: a
> configuration problem). Because this is not a bug and you are looking for
> user support, I am closing this report as INVALID.
> 
> Please use appropriate channels like gentoo-user mailing list, forums or IRC.

The issue is likely related to udev and there is also some other related post which might be years ago, that's why this post is created here, but indeed it's not a bug.

From what I have got, udev support in initramfs for lvm + system setup is important, but there might be some other way to have it work with non-udev initramfs, which I have got no idea for now. From genkernel's wiki, systemd is already supported, so long as lvm, but from systemd's wiki , genkernel-next is recommended, and there is no more details over it, I'm just stucked here..

- https://wiki.gentoo.org/wiki/Systemd#Using_LVM_and_initramfs
- https://wiki.gentoo.org/wiki/Custom_Initramfs#LVM
- https://wiki.gentoo.org/wiki/Genkernel#Genkernel.3F_Genkernel-next.3F_Dracut.3F


For the IRC channel, I entered there and asked several times but hardly got response, and tody it's not available again ... I have just created a post in the forum, looking forward to the help
Comment 6 Mike Gilbert gentoo-dev 2020-01-29 16:28:07 UTC
This is a real bug that happens when the encrypted volume is set up using tools that are built without udev support (as happens with genkernel).
Comment 7 Thomas Deutschmann (RETIRED) gentoo-dev 2020-01-29 17:27:22 UTC
It cannot be a genkernel problem because we don't use udev.

If systemd can't gain control without some udev nodes created you must configure systemd or some helper to do that.

Assigning to systemd because from genkernel POV there is nothing we can do about this and genkernel will *not* start using udev.

And to add some data points: Genkernel project is aware of several users using systemd and genkernel with setup like 

sda                                  8:0    0 223,1G  0 disk
├─sda1                               8:1    0     1M  0 part
├─sda2                               8:2    0   100M  0 part
├─sda3                               8:3    0   180M  0 part  /boot
└─sda4                               8:4    0 222,8G  0 part
  └─root                           253:0    0 222,8G  0 crypt
    ├─Storage-volSwap    253:1    0     4G  0 lvm   [SWAP]
    ├─Storage-volLog     253:3    0    10G  0 lvm   /var/log
    └─Storage-volRoot    253:5    0   140G  0 lvm   /

...so if others are able to get a system like shown working, from my personal POV, this must be a configuration problem.
Comment 8 Mike Gilbert gentoo-dev 2020-01-29 17:42:28 UTC
(In reply to Thomas Deutschmann from comment #7)
> It cannot be a genkernel problem because we don't use udev.

That's exactly the problem.

> If systemd can't gain control without some udev nodes created you must
> configure systemd or some helper to do that.

The device-mapper udev rules do not function correctly if the crypt devices are opened without udev support enabled.

> Assigning to systemd because from genkernel POV there is nothing we can do
> about this and genkernel will *not* start using udev.

systemd is operating as expected here.

The fault is possibly with the device-mapper (LVM2) udev-related code/rules. But then it is rather unexpected to use lvm2 compiled with a different feature set (no udev) in the initramfs.

I don't really understand your outright refusal to support udev in genkernel. So long as you are not willing to budge on this, I see no path forward.

If you don't want to address the issue, please close it as WONTFIX or CANTFIX.

> And to add some data points: Genkernel project is aware of several users
> using systemd and genkernel with setup like 
>
> sda                                  8:0    0 223,1G  0 disk
> ├─sda1                               8:1    0     1M  0 part
> ├─sda2                               8:2    0   100M  0 part
> ├─sda3                               8:3    0   180M  0 part  /boot
> └─sda4                               8:4    0 222,8G  0 part
>   └─root                           253:0    0 222,8G  0 crypt
>     ├─Storage-volSwap    253:1    0     4G  0 lvm   [SWAP]
>     ├─Storage-volLog     253:3    0    10G  0 lvm   /var/log
>     └─Storage-volRoot    253:5    0   140G  0 lvm   /
> 
> ...so if others are able to get a system like shown working, from my
> personal POV, this must be a configuration problem.

I'm really doubtful this actually works for anybody out of the box. Could you point me to these users with working luks with genkernel and systemd? Perhaps we could figure out what they are doing differently.
Comment 9 giskard 2020-01-29 18:10:18 UTC
I agree that genkernel do not include udev to minimize the dependency, as initramfs is mainly used for the mount of fs, for which this is indeed not a bug.

As the rootfs has be mounted and everything has been present, there must be some way to hack the system, but for this common scenario, there should be detailed document
Comment 10 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2020-01-29 18:33:12 UTC
As a bystander, do I understand correctly that this problem does not occur with dracut?
Comment 11 Mike Gilbert gentoo-dev 2020-01-29 19:08:22 UTC
(In reply to Michał Górny from comment #10)
> As a bystander, do I understand correctly that this problem does not occur
> with dracut?

dracut + systemd + luks works as far as I know.
Comment 12 giskard 2020-01-31 14:32:15 UTC
(In reply to Mike Gilbert from comment #11)
> (In reply to Michał Górny from comment #10)
> > As a bystander, do I understand correctly that this problem does not occur
> > with dracut?
> 
> dracut + systemd + luks works as far as I know.

dracut seems to have udev support in initramfs. As far as I know, currently udev support is necessary for such kind of setup, which is a problem for genkernel.

Here is some reference from systemd, but I'm not clear about the details. For now as the issue has taken too much time and still with no solution, I'll just leave it aside, I may switch back to genkernel-next or try to customize an initramfs later after I have the other works done.

[1] https://systemd.io/INITRD_INTERFACE/
[2] https://systemd.io/ROOT_STORAGE_DAEMONS/
Comment 13 giskard 2020-02-10 09:33:35 UTC
after check through the udev rule of dm related, it looks like DM_UDEV_PRIMARY_SOURCE_FLAG check in 10-dm.rules is the key point,
it might be added by dmsetup or lvm when setup the block device,
thus I tried to add an udev rule:

/etc/udev/rules.d/09-dm_hack.rules:
....
KERNEL=="device-mapper", NAME="mapper/control"
SUBSYSTEM!="block", GOTO="dm_end"
KERNEL!="dm-[0-9]*", GOTO="dm_end"
ACTION!="add|change", GOTO="dm_end"

ENV{DM_UDEV_PRIMARY_SOURCE_FLAG}="1"

LABEL="dm_end"
....

then it works.. , everything seems to be fine now,
but NOT SURE if there are any potential issues
Comment 14 Ed Santiago 2020-04-16 19:14:46 UTC
(In reply to giskard from comment #13)
> thus I tried to add an udev rule:

**THANK YOU**. I've been seeing the same problem; your udev hack solved it for me.

Since this is a real problem but it's very difficult to find help, I'm taking the liberty of hijacking this comment in hopes that it will help others in the future.

Keywords: genkernel lvm luks systemd fstab

Synopsis: LVM on top of LUKS on top of RAID results in hung boot.

Symptom: boot hangs for 90 seconds displaying:

[ *** ] (1 of 2) A start job is running for /dev/vg0/opt (xxx / 1min 30s)

Then:

[ TIME ] Timed out waiting for device /dev/vg0/opt
[DEPEND] Dependency failed for File System Check on /dev/vg0/opt
[DEPEND] Dependency failed for /opt.
[DEPEND] Dependency failed for Local File Systems

(Repeat for swap and any other logical volumes referenced in /etc/fstab)

You are in emergency mode [...]

After logging in:

# systemctl list-units --all | grep vg
  dev-mapper-vg0\x2droot.device             loaded  activating tentative /dev/mapper/vg0-root
  dev-vg0-opt.device                        loaded  inactive   dead       /dev/vg0/opt
  [repeat for all logival volumes in /etc/fstab]

# systemctl status dev-mapper-vg0\x2droot.device
* dev-mapper-vg0\x2droot.device - /dev/mapper/vg0-root
     Loaded: loaded
     Active: activating (tentative) since ...
     Device: /sys/devices/virtual/block/dm-1

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.


Manual start attempt just times out again. The only way to boot is to comment out all the non-root entries from /etc/fstab and reboot. That is undesirable.


Using @giskard's udev rule from comment 13 fixes everything: the system comes up as expected.

My partition layout (trimmed to remove redundant sdb/c/d):

NAME             MAJ:MIN RM   SIZE RO TYPE   MOUNTPOINT
sda                8:0    0 931.5G  0 disk
├─sda1             8:1    0   200M  0 part
├─sda2             8:2    0   1.5G  0 part
│ └─md2            9:2    0   2.9G  0 raid10
└─sda3             8:3    0 929.9G  0 part
  └─md3            9:3    0   1.8T  0 raid10
    └─root       253:0    0   1.8T  0 crypt
      ├─vg0-root 253:1    0    20G  0 lvm    /
      ├─vg0-swap 253:2    0    64G  0 lvm    [SWAP]
      ├─vg0-esm  253:3    0   100G  0 lvm
      └─vg0-opt  253:4    0   100G  0 lvm    /opt
Comment 15 Thomas Deutschmann (RETIRED) gentoo-dev 2020-04-17 16:25:39 UTC
Can everyone affected by this please grep their udev rules, i.e.

/{etc,lib}/udev/rules.d/

for 'DM_UDEV_PRIMARY_SOURCE_FLAG'?

I am trying to understand which rules are making use of this attribute.
Comment 16 Craig Andrews gentoo-dev 2020-04-17 16:42:46 UTC
(In reply to Thomas Deutschmann from comment #15)
> Can everyone affected by this please grep their udev rules, i.e.
> 
> /{etc,lib}/udev/rules.d/
> 
> for 'DM_UDEV_PRIMARY_SOURCE_FLAG'?
> 
> I am trying to understand which rules are making use of this attribute.

$ grep -R DM_UDEV_PRIMARY_SOURCE_FLAG /{etc,lib}/udev/rules.d/
/lib/udev/rules.d/10-dm.rules:ENV{DM_UDEV_PRIMARY_SOURCE_FLAG}=="1", ENV{DM_ACTIVATION}="1", GOTO="dm_flags_done"
/lib/udev/rules.d/10-dm.rules:IMPORT{db}="DM_UDEV_PRIMARY_SOURCE_FLAG"
/lib/udev/rules.d/10-dm.rules:# DM_UDEV_PRIMARY_SOURCE_FLAG to see if the device was activated correctly
/lib/udev/rules.d/10-dm.rules:ENV{DM_UDEV_RULES_VSN}!="1", ENV{DM_UDEV_PRIMARY_SOURCE_FLAG}!="1", GOTO="dm_disable"
/lib/udev/rules.d/10-dm.rules:#        EVENT              DM_UDEV_PRIMARY_SOURCE_FLAG   DM_ACTIVATION
/lib/udev/rules.d/69-dm-lvm-metad.rules:ENV{DM_UDEV_PRIMARY_SOURCE_FLAG}=="1", ENV{DM_ACTIVATION}=="1", GOTO="lvm_scan"
Comment 17 Craig Andrews gentoo-dev 2020-04-17 16:43:33 UTC
(In reply to giskard from comment #13)
> after check through the udev rule of dm related, it looks like
> DM_UDEV_PRIMARY_SOURCE_FLAG check in 10-dm.rules is the key point,
> it might be added by dmsetup or lvm when setup the block device,
> thus I tried to add an udev rule:
> 
> /etc/udev/rules.d/09-dm_hack.rules:
> ....
> KERNEL=="device-mapper", NAME="mapper/control"
> SUBSYSTEM!="block", GOTO="dm_end"
> KERNEL!="dm-[0-9]*", GOTO="dm_end"
> ACTION!="add|change", GOTO="dm_end"
> 
> ENV{DM_UDEV_PRIMARY_SOURCE_FLAG}="1"
> 
> LABEL="dm_end"
> ....
> 
> then it works.. , everything seems to be fine now,
> but NOT SURE if there are any potential issues

I confirm that this workaround worked for me (btrfs on luks)
Comment 18 Craig Andrews gentoo-dev 2020-04-17 17:02:20 UTC
CC'ing the maintainers of sys-fs/lvm2 as that owns /lib/udev/rules.d/*dm*.rules
Comment 19 Thomas Deutschmann (RETIRED) gentoo-dev 2020-04-17 18:03:48 UTC
Note: While the rules are owned by lvm2, candrews isn't using LVM at all.
Comment 20 Ed Santiago 2020-04-17 20:09:50 UTC
(In reply to Thomas Deutschmann from comment #15)
> Can everyone affected by this please grep their udev rules, i.e.

My results are identical to Craig Andrews's in comment 16, confirmed via diff (having copy/pasted his results). I'm attaching mine anyway in case anyone else finds `ack` format more readable with long filename prefixes:

# ack DM_UDEV_PRIMARY_SOURCE_FLAG /{etc,lib}/udev/rules.d
/lib/udev/rules.d/10-dm.rules
62:ENV{DM_UDEV_PRIMARY_SOURCE_FLAG}=="1", ENV{DM_ACTIVATION}="1", GOTO="dm_flags_done"
69:IMPORT{db}="DM_UDEV_PRIMARY_SOURCE_FLAG"
78:# DM_UDEV_PRIMARY_SOURCE_FLAG to see if the device was activated correctly
87:ENV{DM_UDEV_RULES_VSN}!="1", ENV{DM_UDEV_PRIMARY_SOURCE_FLAG}!="1", GOTO="dm_disable"
96:#        EVENT              DM_UDEV_PRIMARY_SOURCE_FLAG   DM_ACTIVATION

/lib/udev/rules.d/69-dm-lvm-metad.rules
50:ENV{DM_UDEV_PRIMARY_SOURCE_FLAG}=="1", ENV{DM_ACTIVATION}=="1", GOTO="lvm_scan"

I'm embarrassed not to have included versions in my earlier comment. sys-fs/lvm2-2.02.184-r5  sys-kernel/genkernel-4.0.5  sys-kernel/gentoo-sources-5.4.28
Comment 21 giskard 2020-04-19 06:06:46 UTC
I'm glad that it do some help, it works quiet well for me since then, this is the simplified udev rule I'm using:
....
rt ~ # cat /etc/udev/rules.d/09-hack.rules 
#: fix for non-udev initramfs boot with luks/lvm/systemd
#
# 10-dm.rules checks DM_UDEV_PRIMARY_SOURCE_FLAG
#
SUBSYSTEM=="block", KERNEL=="dm-[0-9]*", ACTION=="add|change", \
	ENV{DM_UDEV_PRIMARY_SOURCE_FLAG}="1"
....


@Thomas Deutschmann, here is the related rules on my laptop:
....
rt ~ # ag DM_UDEV_PRIMARY_SOURCE_FLAG /{etc,lib}/udev/rules.d/
/etc/udev/rules.d/09-hack.rules
3:# 10-dm.rules checks DM_UDEV_PRIMARY_SOURCE_FLAG
6:	ENV{DM_UDEV_PRIMARY_SOURCE_FLAG}="1"

/lib/udev/rules.d/69-dm-lvm-metad.rules
50:ENV{DM_UDEV_PRIMARY_SOURCE_FLAG}=="1", ENV{DM_ACTIVATION}=="1", GOTO="lvm_scan"

/lib/udev/rules.d/10-dm.rules
62:ENV{DM_UDEV_PRIMARY_SOURCE_FLAG}=="1", ENV{DM_ACTIVATION}="1", GOTO="dm_flags_done"
69:IMPORT{db}="DM_UDEV_PRIMARY_SOURCE_FLAG"
78:# DM_UDEV_PRIMARY_SOURCE_FLAG to see if the device was activated correctly
87:ENV{DM_UDEV_RULES_VSN}!="1", ENV{DM_UDEV_PRIMARY_SOURCE_FLAG}!="1", GOTO="dm_disable"
96:#        EVENT              DM_UDEV_PRIMARY_SOURCE_FLAG   DM_ACTIVATION

/lib/udev/rules.d/66-kpartx.rules
39:ENV{ENV{DM_UDEV_PRIMARY_SOURCE_FLAG}!="1", IMPORT{db}="DM_SUBSYSTEM_UDEV_FLAG1"
....

sys-fs/lvm2-2.02.184-r5:
. /lib/udev/rules.d/10-dm.rules
. /lib/udev/rules.d/69-dm-lvm-metad.rules
sys-fs/multipath-tools-0.6.4-r1
. /lib/udev/rules.d/66-kpartx.rules

[the fix I'm using]
. /etc/udev/rules.d/09-hack.rules


And this link may provide more information with related to DM_UDEV_PRIMARY_SOURCE_FLAG:
- https://bugs.gentoo.org/330651#c8
Comment 22 Tommaso Visconti 2020-04-23 17:48:55 UTC
Hi all, I'm having the same exact problem on a fresh install with both root and home on lvm, using systemd and udev, no luks. I can boot only if the home partition is commented out from fstab. I resolved adding the /etc/udev/rules.d/09-hack.rules file as in the message from giskard.
If I should provide more info, please ask (I'm new to gentoo).
Thanks :)
Comment 23 Christopher Byrne 2020-04-24 18:32:53 UTC
I've also hit the same problem, in this case with a thin-provisioned LVM volume. Everything except the root times out, and adding the udev rules provided seems to fix it.
Comment 24 William Breathitt Gray 2020-05-01 14:52:16 UTC
I'm chiming in to mention that giskard's 09-hack.rules fixed this issue for me as well on my luks+lvm+systemd setup. The ebuilds I have installed are:

sys-kernel/genkernel-4.0.7-r1
sys-fs/lvm2-2.02.187-r2
sys-apps/systemd-245-r5
Comment 25 Martin Dummer 2020-05-10 09:37:19 UTC
https://bugs.gentoo.org/706434

At first, I met this problem on a fresh installed gentoo stable desktop last week. Because I thought of a systemd misconfiguration as my personal fault, and a running linux system was urgently needed, I installed an ubuntu desktop (sorry....).

But then I hit this bug yesterday on my personal notebook, ACCEPT_KEYWORDS=amd64

This brought me near a heart attack, and I started searching - so I found this bug.

This is my history: 

On the boot at 2020-04-24   14:29:15 everything was fine.
On boot 2020-05-08 13:09:05 systemd startup timed out as described - root fs on logical volume  was mounted by initrd, timeout occurs on all other LVM devices.

Disk layout, kernel and initrd  is/was unchanged in between, some packages were updated.

Disk layout:

martin@bln9716cm ~/tmp $ lsblk 
NAME               MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
nvme1n1            259:0    0 238,5G  0 disk  
├─nvme1n1p1        259:6    0   529M  0 part  
├─nvme1n1p2        259:7    0    99M  0 part  /boot/efi
├─nvme1n1p3        259:8    0    16M  0 part  
├─nvme1n1p4        259:9    0  63,3G  0 part  
├─nvme1n1p5        259:10   0   200M  0 part  /boot
└─nvme1n1p6        259:11   0 174,3G  0 part  
  ├─vg256G-root    253:0    0    50G  0 lvm   /
  ├─vg256G-tmp     253:1    0     2G  0 lvm   
  ├─vg256G-var     253:2    0    20G  0 lvm   /var
  ├─vg256G-portage 253:3    0    10G  0 lvm   
  └─vg256G-swap    253:4    0    16G  0 lvm   
    └─cryptSwap    253:8    0    16G  0 crypt [SWAP]
nvme0n1            259:1    0 953,9G  0 disk  
├─nvme0n1p1        259:2    0     2M  0 part  
├─nvme0n1p2        259:3    0   100M  0 part  
├─nvme0n1p3        259:4    0   200M  0 part  
└─nvme0n1p4        259:5    0 953,6G  0 part  
  ├─vg1T-home      253:5    0   100G  0 lvm   
  │ └─cryptHome    253:9    0   100G  0 crypt /home
  ├─vg1T-virt      253:6    0   200G  0 lvm   /data/virt
  └─vg1T-binpkg    253:7    0     5G  0 lvm   /var/cache/binpkgs



Kernel+Initrd files:

martin@bln9716cm ~ $ ls -l /boot/*5.4.28*
-rw-r--r-- 1 root root  4588451  3. Apr 19:48 /boot/System.map-5.4.28-gentoo-x86_64-cel1
-rw-r--r-- 1 root root  7454580  3. Apr 20:19 /boot/initramfs-5.4.28-gentoo-x86_64-cel1.img
-rw-r--r-- 1 root root 11245952  3. Apr 19:48 /boot/vmlinuz-5.4.28-gentoo-x86_64-cel1

So IMHO the problem was introduced with one of the updated packages between 20200424 and 20200508.

I GREP through emerge.log and find two possible suspects:


1587821905:  === (14 of 67) Merging (sys-kernel/genkernel-4.0.7-r1::/var/db/repos/gentoo/sys-kernel/genkernel/genkernel-4.0.7-r1.ebuild)
1587821909:  === Unmerging... (sys-kernel/genkernel-4.0.5)
1587822237:  === (19 of 67) Merging (sys-fs/lvm2-2.02.187-r2::/var/db/repos/gentoo/sys-fs/lvm2/lvm2-2.02.187-r2.ebuild)
1587822239:  === Unmerging... (sys-fs/lvm2-2.02.184-r5)
1588749686:  === (39 of 86) Merging (sys-fs/lvm2-2.02.187-r2::/var/db/repos/gentoo/sys-fs/lvm2/lvm2-2.02.187-r2.ebuild)
1588749688:  === Unmerging... (sys-fs/lvm2-2.02.187-r2)



My current versions:

sys-apps/systemd-244.3
sys-kernel/genkernel-4.0.7-r1
sys-fs/lvm2-2.02.187-r2

Note: the initrd currently IN USE was not build with genkernel-4.0.7-r1

I continue to investigate.
Comment 26 Martin Dummer 2020-05-10 10:11:35 UTC
(In reply to Martin Dummer from comment #25)
> Note: the initrd currently IN USE was not build with genkernel-4.0.7-r1
> 

Now I rebuilt the initrd with the installed genkernel-4.0.7-r1 (which uses LVM2.2.02.187) -> boot still failing.
Comment 27 Aleksey Kunitskiy 2020-07-13 16:47:01 UTC
I encountered same issue after sys-kernel/genkernel-next has been masked and I moved to the sys-kernel/genkernel. It was quite unexpected, spent several hours trying to understand what went wrong in configuration but found this bug report.
Thanks giskard for the hack.

Probably someone should notice that masking of sys-kernel/genkernel-next is not in time while this bug is not fixed
Comment 28 Larry the Git Cow gentoo-dev 2020-07-23 23:57:33 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/genkernel.git/commit/?id=c792697d0ec5e54ee9fcf1536f04f2267dff699d

commit c792697d0ec5e54ee9fcf1536f04f2267dff699d
Author:     Thomas Deutschmann <whissi@gentoo.org>
AuthorDate: 2020-07-23 20:25:44 +0000
Commit:     Thomas Deutschmann <whissi@gentoo.org>
CommitDate: 2020-07-23 22:56:47 +0000

    Switch from MDEV to UDEV
    
    We need to switch from using MDEV to UDEV to avoid boot problems
    due to timeouts caused by some UDEV rules from real system when
    real system is using systemd.
    
    Bug: https://bugs.gentoo.org/706434
    Signed-off-by: Thomas Deutschmann <whissi@gentoo.org>

 defaults/initrd.defaults                     |   1 +
 defaults/initrd.scripts                      |  62 +++++++++-----
 defaults/linuxrc                             |  68 +++++++++++----
 defaults/software.sh                         |  16 +++-
 defaults/unlock-luks.sh                      |   2 +
 doc/genkernel.8.txt                          |   4 +
 gen_funcs.sh                                 |  18 ++++
 gen_initramfs.sh                             | 121 +++++++++++++++++++++++----
 gkbuilds/eudev.gkbuild                       |  40 +++++++++
 gkbuilds/hwids.gkbuild                       |  35 ++++++++
 gkbuilds/lvm.gkbuild                         |   8 +-
 mdev/helpers/nvme                            |  21 -----
 mdev/helpers/storage-device                  |  52 ------------
 mdev/mdev.conf                               |  40 ---------
 patches/eudev/3.2.9/eudev-3.2.9-static.patch |  97 +++++++++++++++++++++
 worker_modules/gkbuild.sh                    |   2 +-
 16 files changed, 416 insertions(+), 171 deletions(-)
Comment 29 Thomas Deutschmann (RETIRED) gentoo-dev 2020-07-24 00:00:48 UTC
To support systemd users we decided to switch from MDEV to UDEV.

Please test >=sys-kernel/genkernel-4.1.0_beta1 which is now in repository and the first version using udev.
Comment 30 Martin Dummer 2020-07-24 20:42:41 UTC
(In reply to Thomas Deutschmann from comment #29)
> 
> Please test >=sys-kernel/genkernel-4.1.0_beta1 which is now in repository
> and the first version using udev.

Hey, on my system the new genkernel produces a initamfs which boots WITHOUT the wait-for-device-timeout and the need for manual LV deactivte/activate !!!!


Excellent! Thanks!