While building today's kernels, I've noticed that they fail to boot on arm64. The last lines of output are: [ 14.103927] dracut: Gentoo-2.15 Starting systemd-udevd version 256 [ 555.757460] dracut Warning: Could not boot. dracut Warning: Could not boot. [ 556.175725] dracut Warning: /dev/disk/by-label/groot does not exist dracut Warning: /dev/disk/by-label/groot does not exist i.e. udev starts, then dracut waits for a long time and then abandons all hope. I suspect block devices don't appear. I've reproduced this on arm64 but not amd64. It happens with systemd-256 but not 255. I don't know about ppc64le nor x86 since these two were built with -255. I'd really appreciate if someone could quickly test regenerating initramfs (i.e. `emerge --config`) against systemd-256 on arm64 and checking if it boots, so we'd know whether it's specific to our src_test() configuration and/or qemu, or affects real systems as well.
Was able to reproduce! - VM Host: Windows 10, VirtualBox v7.0.18. - Kernel: sys-kernel/gentoo-kernel-bin-6.6.30 - SystemD: sys-apps/systemd-256.1-r3 Same behavior, cannot find the root filesystem.
Forgot to mention, host and vm are both amd64.
(In reply to Anthony Mendez from comment #2) > Forgot to mention, host and vm are both amd64. So it happens on amd64 too? That's weird, because src_test() worked there, and it's the same setup AFAIR...
I rolled back systemd to 255.7-r1 and emerge --config gentoo-kernel-bin again and then the vm booted again with no issues.
Unable to reproduce here. ~amd64 sys-kernel/dracut-102 sys-apps/systemd-256.1-r3 sys-kernel/vanilla-kernel-6.6.35 I am able to boot real hardware using grub, and src_test in vanilla-kernel works fine.
Could someone who can reproduce this please re-test with Dracut 103?
(In reply to Andrew Nowa Ammerlaan from comment #6) > Could someone who can reproduce this please re-test with Dracut 103? Still happens. I have exported the docker image, and will try booting it interactively in qemu.
Ok, so got into the debug shell, and there were no block devices at all. I had to manually: $ modprobe virtio_blk for them to appear. So apparently that's what stopped happening in systemd-256.
Maybe missing libkmod.so.2 in initramfs due to [1]? [1] https://github.com/systemd/systemd/commit/3abc3671f56d0d124c93acef20f7e7a1c8efe72b
(In reply to Alexander Tsoy from comment #9) > Maybe missing libkmod.so.2 in initramfs due to [1]? Please ignore. dracut-102 and -103 should handle this.
If you want to look at it, the last set is archived at: https://dev.gentoo.org/~mgorny/tmp/kernetest.tar (45M) (warning: no top-level directory) vmlinuz.efi is the kernel, initrd is the initramfs, fs.img is the test disk image. run.sh is the original script used to run tests; you will need to adjust paths in it. (In reply to Alexander Tsoy from comment #9) > Maybe missing libkmod.so.2 in initramfs due to [1]? > > [1] > https://github.com/systemd/systemd/commit/ > 3abc3671f56d0d124c93acef20f7e7a1c8efe72b That's an interesting suggestion. $ lsinitrd initrd | grep kmod -rwxr-xr-x 1 root root 133024 lip 14 17:55 usr/bin/kmod lrwxrwxrwx 1 root root 4 lip 14 18:10 usr/bin/modprobe -> kmod lrwxrwxrwx 1 root root 4 lip 14 18:10 usr/bin/rmmod -> kmod So indeed it doesn't seem to be there. Curious enough, modprobe(1) works, so... kmod(1) doesn't require it?
And indeed, with 255 the initramfs includes libkmod: -rwxr-xr-x 1 root root 132832 Jul 14 17:27 usr/lib64/libkmod.so.2.4.2 lrwxrwxrwx 1 root root 16 Jul 15 06:39 usr/lib64/libkmod.so.2 -> libkmod.so.2.4.2 lrwxrwxrwx 1 root root 16 Jul 15 06:39 usr/lib64/libkmod.so -> libkmod.so.2.4.2
(In reply to Michał Górny from comment #11) > If you want to look at it, the last set is archived at: > > https://dev.gentoo.org/~mgorny/tmp/kernetest.tar (45M) Looks like a typo there. Should be https://dev.gentoo.org/~mgorny/tmp/kerntest.tar > So indeed it doesn't seem to be there. Curious enough, modprobe(1) works, > so... kmod(1) doesn't require it? Yes, kmod statically links to libkmod. That's interesting.. Initramfs is generated with dracut-103 which contains a fix [1]: $ lsinitrd initrd usr/lib/initrd-release | grep ^DRACUT_VERSION DRACUT_VERSION="103" but there is no libkmod indeed: $ lsinitrd initrd | grep libkmod $ [1] https://github.com/dracut-ng/dracut-ng/commit/04b362d713235459cff1f370efb4cd5e36e4a358
And I wonder how this works on amd64
> [1] https://github.com/dracut-ng/dracut-ng/commit/04b362d713235459cff1f370efb4cd5e36e4a358 This fix changes the systemd dracut module, but we explicitly omit that module in src_test. Systemd seems to have some additional conditions that need to be fulfilled for it to consider the root partition valid, at the time I could not figure out what exactly was missing so instead simply omitted the module. Presumably if we fix booting the src_test with the systemd module included we also fix this problem (due to [1]). But then still it seems to me that there is some additional bug in Dracut that needs fixing upstream. What still puzzles me though is why this bug is arch dependent.
Does this resolve the issue on your end? diff --git a/modules.d/01systemd-udevd/module-setup.sh b/modules.d/01systemd-udevd/module-setup.sh index 8d4a8b90..f3a24807 100755 --- a/modules.d/01systemd-udevd/module-setup.sh +++ b/modules.d/01systemd-udevd/module-setup.sh @@ -71,6 +71,8 @@ install() { # Install required libraries. _arch=${DRACUT_ARCH:-$(uname -m)} - inst_libdir_file {"tls/$_arch/",tls/,"$_arch/",}"libudev.so.*" + inst_libdir_file \ + {"tls/$_arch/",tls/,"$_arch/",}"libkmod.so*" \ + {"tls/$_arch/",tls/,"$_arch/",}"libudev.so.*" }
(In reply to Andrew Nowa Ammerlaan from comment #16) > diff --git a/modules.d/01systemd-udevd/module-setup.sh > b/modules.d/01systemd-udevd/module-setup.sh > index 8d4a8b90..f3a24807 100755 > --- a/modules.d/01systemd-udevd/module-setup.sh > +++ b/modules.d/01systemd-udevd/module-setup.sh > @@ -71,6 +71,8 @@ install() { > > # Install required libraries. > _arch=${DRACUT_ARCH:-$(uname -m)} > - inst_libdir_file {"tls/$_arch/",tls/,"$_arch/",}"libudev.so.*" > + inst_libdir_file \ > + {"tls/$_arch/",tls/,"$_arch/",}"libkmod.so*" \ > + {"tls/$_arch/",tls/,"$_arch/",}"libudev.so.*" > > } Note that systemd-udevd module depends on systemd module. I suspect that libkmod.so should also be installed by udev-rules module. Otherwise openrc users will also face the same problem when sys-apps/systemd-utils is updated.
0.036 patching file /usr/lib/dracut/modules.d/01systemd-udevd/module-setup.sh 0.036 Hunk #1 FAILED at 71. 0.037 1 out of 1 hunk FAILED -- saving rejects to file /usr/lib/dracut/modules.d/01systemd-udevd/module-setup.sh.rej Of course, it's possible that I've messed it up while trying to fix it after it being messed up by Bugzilla paste.
I can reproduce on amd64 $ sudo lsinitrd | grep libkmod -rwxr-xr-x 1 root root 104456 Jun 22 19:26 usr/lib64/libkmod.so.2.4.2 lrwxrwxrwx 1 root root 16 Jul 15 11:22 usr/lib64/libkmod.so.2 -> libkmod.so.2.4.2 lrwxrwxrwx 1 root root 16 Jul 15 11:22 usr/lib64/libkmod.so -> libkmod.so.2.4.2 $ sudo dracut -f -o " systemd " ... $ sudo lsinitrd | grep libkmod $ (In reply to Andrew Nowa Ammerlaan from comment #16) > Does this resolve the issue on your end? > > diff --git a/modules.d/01systemd-udevd/module-setup.sh > b/modules.d/01systemd-udevd/module-setup.sh > index 8d4a8b90..f3a24807 100755 > --- a/modules.d/01systemd-udevd/module-setup.sh > +++ b/modules.d/01systemd-udevd/module-setup.sh > @@ -71,6 +71,8 @@ install() { > > # Install required libraries. > _arch=${DRACUT_ARCH:-$(uname -m)} > - inst_libdir_file {"tls/$_arch/",tls/,"$_arch/",}"libudev.so.*" > + inst_libdir_file \ > + {"tls/$_arch/",tls/,"$_arch/",}"libkmod.so*" \ > + {"tls/$_arch/",tls/,"$_arch/",}"libudev.so.*" > > } And this patch doesn't help here. The only module that installs udevd when systemd module is omitted is udev-rules. I'll prepare a patch.
Created attachment 897660 [details] kmod.patch
Created attachment 897661 [details] kmod.patch Added a guard to prevent installing same things multiple times
Created attachment 897662 [details] kmod.patch Fixed copy-paste error
(In reply to Alexander Tsoy from comment #20) > Created attachment 897660 [details] > kmod.patch I can confirm that this one fixed the issue for me.
https://github.com/dracut-ng/dracut-ng/pull/507
Many thanks!
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=286138c22e437f22d1ed42726c4988745d8cf5cf commit 286138c22e437f22d1ed42726c4988745d8cf5cf Author: Andrew Ammerlaan <andrewammerlaan@gentoo.org> AuthorDate: 2024-07-16 16:51:56 +0000 Commit: Andrew Ammerlaan <andrewammerlaan@gentoo.org> CommitDate: 2024-07-16 16:51:56 +0000 sys-kernel/dracut: backport fix for systemd 256 Closes: https://bugs.gentoo.org/935548 Signed-off-by: Andrew Ammerlaan <andrewammerlaan@gentoo.org> .../{dracut-103.ebuild => dracut-103-r1.ebuild} | 2 ++ .../files/dracut-103-systemd-udev-256-kmod.patch | 41 ++++++++++++++++++++++ 2 files changed, 43 insertions(+)
