Summary: | sys-fs/lvm2-2.02.184-r4 delayed boot with multiple "WARNING: Device /dev/sdaX not initialized in udev database even after waiting 10000000 microseconds" | ||
---|---|---|---|
Product: | Gentoo Linux | Reporter: | Norman Back <gentoo3> |
Component: | Current packages | Assignee: | Gentoo's Team for Core System packages <base-system> |
Status: | UNCONFIRMED --- | ||
Severity: | normal | CC: | agk, cardoe, evert.gentoo, robbat2 |
Priority: | Normal | ||
Version: | unspecified | ||
Hardware: | All | ||
OS: | Linux | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Attachments: | console output @ boot |
Description
Norman Back
2019-07-23 09:22:28 UTC
Booting via initramfs created by sys-kernel/dracut-048-r1 with command dracut --host-only --no-hostonly-cmdline --force /boot/initramfs-5.1.18-gentoo.img 5.1.18-gentoo kernel command line: root=/dev/mapper/sys_ssd1_5-slash vga=845 dolvm console=tty1 rootfstype=ext4 rd.lvm.vg=sys_ssd1_5 Boots without delay with sys-fs/lvm2-2.02.145-r2 confirmed boot breaks for me on both lvm2-2.02.184-r5 and lvm2-2.02.183 - I do *not* use initrd (simply grub -> kernel -> init) - I use systemd - kernel 4.19.82 (doesn't seem kernel related, same on happens on 4.19.69) [ TIME ] Timed out waiting for device /dev/disk/by-label/data. [DEPEND] Dependency failed for /data. [DEPEND] Dependency failed for /root. [DEPEND] Dependency failed for Local File Systems. [DEPEND] Dependency failed for /home. [DEPEND] Dependency failed for File System Check on /dev/disk/by-label/data. [ OK ] Started Emergency Shell. [ OK ] Reached target Emergency Mode. You are in emergency mode. After logging in, type "journalctl -xb" to view system logs, "systemctl reboot" to reboot, "systemctl default" or "exit" to boot into default mode. Give root password for maintenance (or press Control-D to continue): So let's check ... # time lvs WARNING: Device /dev/sdb16 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb17 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb18 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb19 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb10 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb11 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb12 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb13 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb14 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb15 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb16 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb17 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb18 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb19 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb10 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb11 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb12 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb13 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb14 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb15 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb16 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb17 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb18 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb19 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb10 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb11 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb12 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb13 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb14 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb15 not initialized in udev database even after waiting 10000000 microseconds. device-mapper: ioctl: 4.39.0-ioctl (2018-04-03) initialised: dm-devel@redhat.com LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert data data -wi------- <2.00t real 5m0.590s user 0m0.130s sys 0m0.130s # time pvs WARNING: Device /dev/sdb16 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb17 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb18 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb19 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb10 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb11 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb12 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb13 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb14 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb15 not initialized in udev database even after waiting 10000000 microseconds. PV VG Fmt Attr PSize PFree /dev/sdb10 data lvm2 a-- <256.00g 0 /dev/sdb11 data lvm2 a-- <256.00g 0 /dev/sdb12 data lvm2 a-- <256.00g 0 /dev/sdb13 data lvm2 a-- <256.00g 0 /dev/sdb14 data lvm2 a-- <256.00g 0 /dev/sdb15 data lvm2 a-- <256.00g 0 /dev/sdb16 data lvm2 a-- <256.00g 0 /dev/sdb17 data lvm2 a-- <256.00g 0 /dev/sdb18 data lvm2 a-- <256.00g <256.00g /dev/sdb19 lvm2 --- 256.00g 256.00g real 1m40.228s user 0m0.056s sys 0m0.047s After downgrading to lvm2-2.02.145-r2 everything is fine. My LV is created on 2013-12-25 (according to /etc/lvm/backup/) so I thought maybe a vgconvert would help. So I disabled use_lvmetad and stopped lvmetad (required for vgconvert) but it seems it already uses the lastest metadata format: # vgconvert -M2 data Volume group "data" already uses format lvm2 My guess is I hit https://bugzilla.redhat.com/show_bug.cgi?id=1676612#c3 - MD metadata are placed at the end of the disk (current LVM disk-reading code caches/considers only start of the disk) However it's unclear to me how to solve this. Do I need to backup the data, recreate the pv/vg/lv and restore the data or ...? Does this still happen with lvm2-2.02.186? I want to test but ... sys-fs/lvm2-2.02.145-r2 has now been removed from portage. This is the latest version known to work without issue. If things don't work out, I'd like to be able to revert to sys-fs/lvm2-2.02.145-r2 but without that ebuild, that's impossible. This makes testing a huge risk. Can you please restore sys-fs/lvm2-2.02.145-r2 into portage? lvm2-2.02.186 resolved a similar issue for me see: https://bugs.gentoo.org/show_bug.cgi?id=701074 So I saved lvm2-2.02.145-r2.ebuild from the pkg db and took the risk. lvm2-2.02.186-r2 does not solve my issue. I needed to revert to lvm2-2.02.145-r2, however files were also missing in the files/ dir. Luckily I found them somewhere on the internet. I restored - lvm2-2.02.145-r2.ebuild - files/lvm.rc-2.02.116-r6 - files/lvm2-2.02.129-static-pkgconfig-libs.patch - files/lvm2-2.02.130-pthread-pkgconfig.patch - files/lvm2-2.02.139-dynamic-static-ldflags.patch - files/lvm2-2.02.145-mkdev.patch and was able to revert to lvm2-2.02.145-r2. Until this issue is solved, can you please restore those files in the gentoo repo? Note: I use systemd - this uses an alternate udev: /lib/systemd/systemd-udevd Please let me know how I can help to fix this issue. The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=0a7dcaff7c920619ffce4c0195223cbd9db02411 commit 0a7dcaff7c920619ffce4c0195223cbd9db02411 Author: Robin H. Johnson <robbat2@gentoo.org> AuthorDate: 2019-11-28 22:43:05 +0000 Commit: Robin H. Johnson <robbat2@gentoo.org> CommitDate: 2019-11-28 22:44:35 +0000 sys-fs/lvm2: partial revert of 2161b3a104c9a5664131b9297766db0538e7464e Bug 690494 shows that the regression is still not fixed in 2.02.186-r2, so 2.02.145-r2 is still required for some users. Bug: https://bugs.gentoo.org/show_bug.cgi?id=690494 Package-Manager: Portage-2.3.79, Repoman-2.3.18 Signed-off-by: Robin H. Johnson <robbat2@gentoo.org> sys-fs/lvm2/Manifest | 3 + .../lvm2-2.02.129-static-pkgconfig-libs.patch | 102 +++++++++ .../files/lvm2-2.02.130-pthread-pkgconfig.patch | 32 +++ .../lvm2-2.02.139-dynamic-static-ldflags.patch | 63 +++++ sys-fs/lvm2/files/lvm2-2.02.145-mkdev.patch | 127 +++++++++++ sys-fs/lvm2/lvm2-2.02.145-r2.ebuild | 254 +++++++++++++++++++++ 6 files changed, 581 insertions(+) Please share your udev version with us. Created attachment 597834 [details]
console output @ boot
udev is implemented in systemd
$ qlist -ISRv udev systemd
acct-group/systemd-coredump-0:0::gentoo
acct-group/systemd-journal-0:0::gentoo
acct-group/systemd-journal-remote-0:0::gentoo
acct-group/systemd-network-0:0::gentoo
acct-group/systemd-resolve-0:0::gentoo
acct-group/systemd-timesync-0:0::gentoo
acct-user/systemd-coredump-0:0::gentoo
acct-user/systemd-journal-remote-0:0::gentoo
acct-user/systemd-network-0:0::gentoo
acct-user/systemd-resolve-0:0::gentoo
acct-user/systemd-timesync-0:0::gentoo
dev-libs/libgudev-233-r1:0::gentoo
sys-apps/gentoo-systemd-integration-7:0::gentoo
sys-apps/systemd-243-r2:0::gentoo
sys-fs/udev-init-scripts-33:0::gentoo
virtual/libgudev-232:0::gentoo
virtual/libudev-232-r3:0::gentoo
virtual/udev-217:0::gentoo
I don't use initrd, so the sequence here is:
grub -> kernel-4.19.86 -> init=/lib/systemd/systemd
At boot time on the console, I also see 9 times segfaults like
udisks-lvm-pv-e[252]: segfault at 0 ip 000055a9671cf28f sp 00007ffc0853bdd0 error 4 in udisks-lvm-pv-export[55a9671cf000+1000]
I have 9 PVs of 256 GiB each (+ one PV of 10 GiB)
Full console output -> see attachment - includes segfaults and waiting
Good news: sys-fs/lvm2-2.02.186-r2 is running fine here too. Solution was an emerge --depclean. The cause was segfault in udisks-lvm-pv-export. The real solution was removal of sys-fs/udisks-1.0.5-r3:0 # locate udisks-lvm-pv-export /lib64/udev/udisks-lvm-pv-export # qfile -Sv udisks-lvm-pv-export sys-fs/udisks-1.0.5-r3:0: /lib/udev/udisks-lvm-pv-export # qlist -ISRv udisks sys-fs/udisks-2.8.4:2::gentoo sys-fs/udisks-1.0.5-r3:0::gentoo So the depclean removed udisks-1.0.5-r3 which makes lvm2-2.02.186-r2 happy. Now, if I remerge udisks-1.0.5-r3 again then it downgrades lvm2 too so obviously there's some dependency there. However, after that, it lets me happily upgrade lvm2 anyway without removing udisks-1.0.5-r3 (in which case the issue is back again). # emerge -av1 =udisks-1.0.5-r3 These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild N ] app-admin/killproc-2.13-r1::gentoo 0 KiB [ebuild UD ] sys-fs/lvm2-2.02.145-r2::gentoo [2.02.186-r2::gentoo] USE="readline systemd thin udev -device-mapper-only -lvm1% -lvm2create_initrd (-selinux) -static -static-libs (-sanlock%)" 0 KiB [ebuild N ] sys-apps/sg3_utils-1.42::gentoo USE="-static-libs" 0 KiB [ebuild N ] sys-apps/rescan-scsi-bus-1.57-r1::gentoo 0 KiB [ebuild NS ] sys-fs/udisks-1.0.5-r3:0::gentoo [2.8.4:2::gentoo] USE="lvm* nls -debug -remote-access% (-selinux) (-acl%*) (-elogind%) (-introspection%*) (-systemd%*) (-vdo%)" 0 KiB Total: 5 packages (1 downgrade, 3 new, 1 in new slot), Size of downloads: 0 KiB # emerge -av lvm2 These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild U ~] sys-fs/lvm2-2.02.186-r2::gentoo [2.02.145-r2::gentoo] USE="readline systemd thin udev -device-mapper-only -lvm2create_initrd -sanlock% (-selinux) -static -static-libs (-lvm1%)" 0 KiB Total: 1 package (1 upgrade), Size of downloads: 0 KiB So IMHO, sys-fs/lvm2-2.02.186-r2 should have a conflict with sys-fs/udisks-1.0.5-r3:0 or something similar. Btw: again lvm2-2.02.145-r2 did not emerge because files/lvm.rc-2.02.116-r6 was still missing. More specifically, the issue was: sys-fs/udisks-1.0.5-r3:0: /lib/udev/rules.d/80-udisks.rules ENV{ID_FS_TYPE}=="LVM2_member", TEST=="/lib/udev/udisks-lvm-pv-export", IMPORT{program}="udisks-lvm-pv-export $env{ID_FS_UUID}" Commenting this like with a # also resolves the issue. Anyway, I don't need udisks-1.0.5-r3 so I removed the package. Still happening with 2.02.187-r2. Are we certain lvm2 is causing this? sys-fs/lvm2-2.02.187-r2::gentoo was built with the following: USE="readline static-libs thin udev -device-mapper-only -lvm2create_initrd -sanlock (-selinux) -static -systemd" ABI_X86="(64)" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg candy cgroup compress-build-logs compressdebug config-protect-if-modified distlocks ebuild-locks fixlafiles installsources ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned sandbox sfperms split-elog split-log splitdebug strict strict-keepdir unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" sys-fs/udisks-2.9.0::gentoo was built with the following: USE="acl daemon elogind introspection lvm nls -debug (-selinux) -systemd -vdo" ABI_X86="(64)" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg candy cgroup compress-build-logs compressdebug config-protect-if-modified distlocks ebuild-locks fixlafiles installsources ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms split-elog split-log splitdebug strict strict-keepdir unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" |