Booting genkernel 4.1.0-r2 initramfs does not mount separate /usr partition during sysinit and boot runlevels, causing several daemons to fail finding /usr/lib64/libcap-ng.so.0. Same configuration using genkernel 4.0.10 works fine. Reproducible: Always Steps to Reproduce: 1. Upgrade to genkernel 4.1.0-r2 2. Run 'genkernel initramfs --luks --lvm --kernel-config=/usr/src/linux/.config' 3. Reboot Actual Results: sysinit and boot runlevel daemons fail to start Expected Results: System should boot normally like before * Genkernel was only used to generate initramfs, kernel itself was manually built * System uses dm-crypt full root disk encryption, encrypted disk contains a separate /usr partition emerge --info: Portage 3.0.4 (python 3.7.9-final-0, default/linux/amd64/17.1/hardened, gcc-10.2.0, glibc-2.32, 5.8.3-gentoo x86_64) ================================================================= System uname: Linux-5.8.3-gentoo-x86_64-AMD_Ryzen_9_3900X_12-Core_Processor-with-gentoo-2.7 KiB Mem: 32855868 total, 30236336 free KiB Swap: 10485756 total, 10485756 free Timestamp of repository gentoo: Sun, 23 Aug 2020 22:30:01 +0000 Head commit of repository gentoo: 158312052d73276281ba9e49ddaf4c792fe25cd3 sh bash 5.0_p18 ld GNU ld (Gentoo 2.34 p6) 2.34.0 ccache version 3.7.11 [enabled] app-shells/bash: 5.0_p18::gentoo dev-lang/perl: 5.30.3-r1::gentoo dev-lang/python: 2.7.18-r1::gentoo, 3.7.9::gentoo, 3.8.5::gentoo, 3.9.0_rc1::gentoo dev-util/ccache: 3.7.11::gentoo dev-util/cmake: 3.18.1::gentoo sys-apps/baselayout: 2.7::gentoo sys-apps/openrc: 0.42.1::gentoo sys-apps/sandbox: 2.20::gentoo sys-devel/autoconf: 2.13-r1::gentoo, 2.69-r5::gentoo sys-devel/automake: 1.16.2::gentoo sys-devel/binutils: 2.34-r2::gentoo sys-devel/gcc: 10.2.0-r1::gentoo sys-devel/gcc-config: 2.3.1::gentoo sys-devel/libtool: 2.4.6-r6::gentoo sys-devel/make: 4.3::gentoo sys-kernel/linux-headers: 5.8::gentoo (virtual/os-headers) sys-libs/glibc: 2.32::gentoo Repositories: gentoo location: /var/db/repos/gentoo sync-type: rsync sync-uri: rsync://gntrepo.internal.mnet/gentoo-portage priority: -1000 sync-rsync-verify-metamanifest: yes sync-rsync-extra-opts: sync-rsync-verify-jobs: 1 sync-rsync-verify-max-age: 24 local location: /usr/local/portage masters: gentoo guru location: /var/lib/layman/guru masters: gentoo priority: 50 steam-overlay location: /var/lib/layman/steam-overlay masters: gentoo priority: 50 ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="@FREE" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -pipe -march=znver2" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -pipe -march=znver2" DISTDIR="/var/cache/distfiles" ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR" FCFLAGS="-O2 -pipe -march=znver2" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs ccache config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe -march=znver2" GENTOO_MIRRORS="https://ftp.halifax.rwth-aachen.de/gentoo/ http://ftp.halifax.rwth-aachen.de/gentoo/ ftp://ftp.halifax.rwth-aachen.de/gentoo/ rsync://ftp.halifax.rwth-aachen.de/gentoo/ ftp://mirror.mdfnet.se/gentoo https://mirror.mdfnet.se/gentoo http://mirror.mdfnet.se/gentoo" LANG="en_DK.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j25" PKGDIR="/var/cache/binpkgs" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" USE="X acl alsa amd64 apparmor audit bzip2 crypt dbus elogind flac hardened iconv ipv6 jack libglvnd libtirpc multilib ncurses nls nptl openmp pam pcre pie pulseaudio readline seccomp split-usr ssl ssp unicode vorbis vulkan wayland xattr xtpax zlib" ABI_X86="64 32" ADA_TARGET="gnat_2018" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt sha sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64" INPUT_DEVICES="libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2 php7-3" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_7" PYTHON_TARGETS="python2_7 python3_7" RUBY_TARGETS="ruby25" USERLAND="GNU" VIDEO_CARDS="amdgpu radeonsi" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Created attachment 656582 [details] rc.log (sysinit runlevel)
Created attachment 656584 [details] /etc/initramfs.mounts
Set gk.log.keep=yes command-line argument and reboot. This will create /genkernel-boot.log. Please attach this file to this bug.
Created attachment 656618 [details] genkernel-boot.log using genkernel 4.1.0
Created attachment 656620 [details] genkernel-boot.log using genkernel 4.0.10
Please show us your fstab.
Created attachment 656622 [details] /etc/fstab
Like you can see in log, > [2020-08-25 14:33:26] [OK] Mounting ../dm-2 as /usr: mount -t ext4 -o relatime,nodev,ro,ro ../dm-2 /newroot/usr > [2020-08-25 14:33:26] [!!] Unable to mount ../dm-2 for /usr it's failing to mount for some reason. I have no idea at the moment what could cause this (it's ext4 like root mountpoint, on same volume/group like root which was successfully mounted just seconds before). 1) Please apply https://gitweb.gentoo.org/proj/genkernel.git/commit/?id=848f9d5eb48f6fe7cf1d11a2ccb2df9198f474f1 to see if this will already address your issue (but I doubt that this is related). 2) If not, are you able to debug? Add "debug" to kernel command-line which will give you multiple prompts. Do "exit" until you reached breakpoint before switch_root. Try manually mounting... maybe you will spot the issue and can report back. You can also use SSH feature for remote debugging...
Wait, it's "../dm-2" -- sounds like another case of https://gitweb.gentoo.org/proj/genkernel.git/commit/?id=0048f44c081dce2e296b48c71a208abf2a815c84
The issue no longer happens after applying https://gitweb.gentoo.org/proj/genkernel.git/commit/?id=848f9d5eb48f6fe7cf1d11a2ccb2df9198f474f1, I guess process_initramfs_mounts introduced there also handles symlinks correctly.
Yeah, I am now able to confirm bug in <=4.0.10 and that it's fixed in git: > - [ -L ${dev} ] && dev=$(readlink ${dev}) > + [ -L "${dev}" ] && dev=$(realpath "${dev}") I.e. the patch is changing from readlink to realpath which will make it work. And the issue is present in genkernel-4.1 at all because LVM's udev rule will create relative symlinks.
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=66c1419df0c7adcd7a6436199f5229de8e6b8ece commit 66c1419df0c7adcd7a6436199f5229de8e6b8ece Author: Thomas Deutschmann <whissi@gentoo.org> AuthorDate: 2020-08-26 22:59:23 +0000 Commit: Thomas Deutschmann <whissi@gentoo.org> CommitDate: 2020-08-26 22:59:31 +0000 sys-kernel/genkernel: bump to v4.1.1 Closes: https://bugs.gentoo.org/738922 Closes: https://bugs.gentoo.org/738740 Package-Manager: Portage-3.0.4, Repoman-3.0.1 Signed-off-by: Thomas Deutschmann <whissi@gentoo.org> sys-kernel/genkernel/Manifest | 3 + sys-kernel/genkernel/genkernel-4.1.1.ebuild | 303 ++++++++++++++++++++++++++++ sys-kernel/genkernel/genkernel-9999.ebuild | 5 +- 3 files changed, 310 insertions(+), 1 deletion(-)
Just upgraded to genkernel 4.1.2-r2. It seems, that issue is not resolved - on my system, all LVM volumes (except root volume) is not available during boot process, even if kernel itself, before starting initd, writes to console that LVM volumes are detected. They are actually not available until LVM daemon is started. Boot process can not check and mount volumes, listed in fstab, and because of it, some services, scheduled to start in default boot level, are broken. After system started, I need to login, manually mount volumes and start broken services. On version 4.0 I just need to edit initrd.scripts to move detection of bcache partitions before LVM, because on my system bcache partitions is physical volumes for LVM. On version 4.1 bcache detection is moved away from that place. bcache partitions are actually correctly detected, LVM somehow partially working, because root partition is in LVM side. But other non root volumes in LVM is not available until LVM daemon starts.
Please boot genkernel initramfs created by >=genkernel-4.1.2 with "debug" in kernel command-line. Boot will stop multiple times when hitting a debug breakpoint. Wait for the debug breakpoint before "switch_root" call. In rescue shell, run "gksosreport". If system is unable to continue booting, please save /run/initramfs/gksosreport.txt to /boot or USB stick. If boot is able to continue, /run will be preserved. Please attach this report to this bug.
Created attachment 659136 [details] progmachine gksosreport
Strange situation: in rescue shell, all LVM things seems to be normal, directories /dev/vgroup0, /dev/mapper, all with symlinks exists, but during boot stage that LVM things is not available. Root filesystem mounts without problems, because it specified with UUID, and kernel successfully uses direct /dev/dm-XX file. Now I forced to do the same thing in fstab - all filesystems in it now specified with UUIDs, and it works fine.
How did /etc/fstab looked before? If you didn't use strong names (i.e. used /dev/dm-4), it might be possible that names have changed due to udev usage. I cannot spot any error in gksosreport so I assume it wasn't a problem within genkernel itself, just a problem on real system when taking control which you now addressed?
>> How did /etc/fstab looked before? The previous version of fstab used paths like /dev/vgroup0/progmachine-ws0-root or /dev/vgroup0/progmachine-ws0-home, where "vgroup0" is LVM volume group and "progmachine-ws0-root" or "progmachine-ws0-home" is LVM volumes in that group. For example, /dev/vgroup0/root is symlink to "../mapper/vgroup0-progmachine--ws0--root", and that is symlink to "../dm-0" in initrd environment. Pretty standard paths for LVM. It works very well on genkernel 4.0, but is broken in genkernel 4.1, possibly due to changes in chroot-transition from initrd to real root filesystem. >> I cannot spot any error in gksosreport so I assume it wasn't a problem within genkernel itself, just a problem on real system when taking control which you now addressed? Now I use workaround for this problem - all block device paths in fstab is changed to UUIDs, and it helps for me.
