When booting with an initramfs generated with >=sys-kernel/genkernel-4.1.0-r2 I see mysterious problems with at least NetworkManager and Xorg: * NetworkManager lists all interfaces as "unmanaged" and can not activate any connection * xorg does not detect any input devices My best guess was that it was something udev-related, but as far as I can tell /dev is properly populated so I'm a bit lost. Everythin works as expected if I use an initramfs generated with genkernel-4.0.10, so I'm pretty sure the issue is caused by the initramfs, even as the system appears to boot properly. Reproducible: Always Steps to Reproduce: 1. generate an initramfs with genkernel-4.1.1 2. boot using the generated initramfs Actual Results: When starting NetowrkManager and running nmcli: NetworkManager reports all interfaces to be "unmanaged" and I'm unable to activate any connection When starting X: no input device works (log does not list any input devices) Expected Results: nmcli should report interfaces as "disconnected" xorg should register mouse and keyboard as input devices. possibly uncommon setup that may or may not be related: * I use zfs on root * both /boot and the zfs pool is encrypted; /boot is unlocked by passphrase to grub and the zfs pool is unlocked by a key embedded in the initramfs. # emerge --info Portage 2.3.103 (python 3.7.8-final-0, default/linux/amd64/17.1/hardened/selinux, gcc-9.3.0, glibc-2.31-r6, 5.4.60-gentoo-x86_64 x86_64) ================================================================= System uname: Linux-5.4.60-gentoo-x86_64-x86_64-Intel-R-_Core-TM-_i7-7500U_CPU_@_2.70GHz-with-gentoo-2.6 KiB Mem: 16162704 total, 11741248 free KiB Swap: 0 total, 0 free Timestamp of repository gentoo: Fri, 28 Aug 2020 14:45:01 +0000 Head commit of repository gentoo: 303324e498b671b34acf4e08a780e61f2a27b0b9 sh bash 5.0_p18 ld GNU ld (Gentoo 2.33.1 p2) 2.33.1 app-shells/bash: 5.0_p18::gentoo dev-java/java-config: 2.3.1::gentoo dev-lang/perl: 5.30.3::gentoo dev-lang/python: 2.7.18-r1::gentoo, 3.6.11-r2::gentoo, 3.7.8-r2::gentoo, 3.8.5::gentoo dev-util/cmake: 3.16.5::gentoo dev-util/pkgconfig: 0.29.2::gentoo sys-apps/baselayout: 2.6-r1::gentoo sys-apps/openrc: 0.42.1::gentoo sys-apps/sandbox: 2.18::gentoo sys-devel/autoconf: 2.13-r1::gentoo, 2.69-r4::gentoo sys-devel/automake: 1.13.4-r2::gentoo, 1.16.1-r1::gentoo sys-devel/binutils: 2.33.1-r1::gentoo sys-devel/gcc: 9.3.0-r1::gentoo sys-devel/gcc-config: 2.3.1::gentoo sys-devel/libtool: 2.4.6-r6::gentoo sys-devel/make: 4.2.1-r4::gentoo sys-kernel/linux-headers: 5.4-r1::gentoo (virtual/os-headers) sys-libs/glibc: 2.31-r6::gentoo Repositories: gentoo location: /usr/portage sync-type: rsync sync-uri: rsync://rsync.europe.gentoo.org/gentoo-portage priority: -1000 sync-rsync-verify-max-age: 24 sync-rsync-verify-metamanifest: yes sync-rsync-extra-opts: sync-rsync-verify-jobs: 1 local location: /usr/local/portage masters: gentoo ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=native -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-march=native -O2 -pipe" DISTDIR="/usr/portage/distfiles" ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch preserve-libs protect-owned qa-unresolved-soname-deps sandbox selinux sesandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://distfiles.gentoo.org" LANG="sv_SE.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j5 -l7" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" USE="X acl alsa amd64 apng audit bindist bluetooth bzip2 cacert cairo cjk client conntrack corefonts crypt cryptsetup cups curl dbus device-mapper dri elogind exif gd glamor gpg gssapi gtk gui hardened hpijs hwaccel iconv icu idn infinality inspector ipset ipv6 jpeg kerberos ldap libglvnd libtirpc libzfs lightning lua luajit lzo mbim minizip modemmanager multilib ncurses nftables nls nptl nvme opengl openldap openmp pam pcre pcre16 pie png pulseaudio python raw readline relp seccomp selinux smartcard spice split-usr sqlite ssh ssl ssp svg text tiff truetype udev uefi unicode usb usbredir vaapi vim-syntax virt-network widgets xattr xinerama xinetd xkb xtpax zlib" ABI_X86="64" ADA_TARGET="gnat_2018" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 fma3 mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev libinput" KERNEL="linux" L10N="sv sv_SE en en_GB en_US" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LLVM_TARGETS="X86" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2 php7-3" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_7" PYTHON_TARGETS="python2_7 python3_7" RUBY_TARGETS="ruby25" USERLAND="GNU" VIDEO_CARDS="intel i965" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Created attachment 657304 [details] genkernel.conf used
Created attachment 657306 [details] log from "genkernel initramfs"
Created attachment 657308 [details] log from starting xorg when no input devices are found
This sounds similar to another issue I heard about-- can you see if you are missing /run/utmp ? Do your /run contents look the same on working vs non-working boot?
Created attachment 657336 [details, diff] diff of listings of /run for working and not working environment I generated a listing of /run in both a working and a broken environment using find /run | sort > list and created the attached diff. utmp is present in both environments, but you're probably on to something because the broken environment is missing lots of files compared to the working environment. Most of them seems to be udev-related.
Looks like udev isn't starting at all... can you try to start the udev service on the "broken" setup and see if it starts up successfully?
Getting closer, but unfortunately it looks like udev is running; 'rc-service udev status' reports the service as started and there is a systemd-udevd process in the process list. Trying to restart udev curiously results in that I now have two systemd-udevd-processes, but /run has not been more populated than before and the problem persists. For the record: when I noticed the problem I was using sys-fs/udev-245.5-r1, but right now I use sys-fs/udev-246-r1 as I unmasked it during troubleshooting. I tried to upgrade another machine which uses pretty much the same setup, except for using sys-fs/eudev instead of sys-fs/udev. The problem does not appear on that machine, so it's probably restricted to systems using sys-fs/udev.
I have exactly the same problem on my system. Network devices won’t come up and X.org won’t start due to missing devices. With initramfs images from earlier genkernel versions, booting works fine. I diff’ed the two initramfs and noticed that the newer ones lack mdev-related stuff. Maybe genkernel omits something important?
Created attachment 659024 [details] diff between working (583) and non-working (585) initramfs
Created attachment 659026 [details] emerge --info (as an attachment because it’s too large)
(In reply to matthias.grobarek from comment #8) > I diff’ed the two initramfs and noticed that the newer ones lack > mdev-related stuff. Maybe genkernel omits something important? No :) We switched to udev with v4.1.0+.
I can reproduce this and having problems with wifi not being enabled (and impossible to) in NetworkManager: nmcli says: (...) wlo1: unmanaged "Intel Wireless-AC 9560" wifi (iwlwifi), 30:24:32:69:B7:84, plugin missing, hw, mtu 1500 That "plugin missing" is strange, since I compiled it with wifi USE flag: Installed versions: 1.26.2^t(19:29:37 16/09/20)(dhcpcd elogind introspection iwd json ncurses nss policykit resolvconf wext wifi -audit -bluetooth -connection-sharing -consolekit -dhclient -gnutls -modemmanager -ofono -ovs -ppp -selinux -systemd -teamd -test -vala ABI_MIPS="-n32 -n64 -o32" ABI_RISCV="-ilp32 -ilp32d -lp64 -lp64d" ABI_S390="-32 -64" ABI_X86="64 -32 -x32" KERNEL="linux") But according to equery files networkmanager I can see that the plugin is in fact there: /usr/lib64/NetworkManager/1.26.2/libnm-device-plugin-wifi.so So I don't think it is a NetworkManager ebuild problem but rather seems related to this genkernel update. Did not tried though reverting back to previous version without any other change to double confirm.
(In reply to David Carlos Manuelda from comment #12) > I can reproduce this and having problems with wifi not being enabled (and > impossible to) in NetworkManager: > > nmcli says: > > (...) > wlo1: unmanaged > "Intel Wireless-AC 9560" > wifi (iwlwifi), 30:24:32:69:B7:84, plugin missing, hw, mtu 1500 > > That "plugin missing" is strange, since I compiled it with wifi USE flag: > > Installed versions: 1.26.2^t(19:29:37 16/09/20)(dhcpcd elogind > introspection iwd json ncurses nss policykit resolvconf wext wifi -audit > -bluetooth -connection-sharing -consolekit -dhclient -gnutls -modemmanager > -ofono -ovs -ppp -selinux -systemd -teamd -test -vala ABI_MIPS="-n32 -n64 > -o32" ABI_RISCV="-ilp32 -ilp32d -lp64 -lp64d" ABI_S390="-32 -64" ABI_X86="64 > -32 -x32" KERNEL="linux") > > But according to equery files networkmanager I can see that the plugin is in > fact there: > > /usr/lib64/NetworkManager/1.26.2/libnm-device-plugin-wifi.so > > So I don't think it is a NetworkManager ebuild problem but rather seems > related to this genkernel update. > > Did not tried though reverting back to previous version without any other > change to double confirm. Important to say that this problem is not related to kernel or other software, because wifi can be correctly associated by hand using wpa_supplicant
I have reproduced this problem on every kernel build I have done since I upgraded genkernel to 4.1.2-r3 on Sep 28 - being gentoo-sources 5.8.12, 5.8.13, and 5.8.14. I could successfully start Xorg, and use NetworkManager on 5.8.11 (built on Sep 26). I have downgraded genkernel to 4.0.10, and rebuilt the initramfs of 5.8.14. This boots successfully, and I get into X, and Network Manager brings up my network connections, so the problem has to be with what genkernel is doing when it builds the initramfs.
Bug 740576 looks like it is related. Is everyone on this bug running selinux? It looks to me like it could be this issue only affects selinux systems booted with the new genkernel release.
(In reply to Eddie Chapman from comment #15) > Bug 740576 looks like it is related. Is everyone on this bug running > selinux? It looks to me like it could be this issue only affects selinux > systems booted with the new genkernel release. Yes... and no... or maybe just a little? Both my systems (one working and one not) use the hardened/selinux profile, but both are configured to run in permissive mode. The difference between the two systems regarding selinux is that the system that does not work has had selinux in enforcing mode previously, while the system that works have I never had in enforcing mode. The labelling is probably in various state of broken on both systems...
(In reply to Fredrik Eriksson from comment #16) > Both my systems (one working and one not) use the hardened/selinux profile, > but both are configured to run in permissive mode. Both my affected systems are selinux permissive but, if the problem is what I think it is, it doesn't matter which selinux mode you are running. From my own observations and testing, and what the OP of the other bug suggests, I believe the problem exists because in the initramfs /run gets mounted by the linuxrc script *without* these options needed for all selinux systems: rootcontext=system_u:object_r:var_run_t,seclabel After the initramfs, OpenRC then mounts on top of /run with the selinux options (from the /run entry in the system's /etc/fstab). So on an selinux system you will have 2 /run mounts showing in /proc/mounts. But the selinux options used on the 2nd mount have no effect, your system runs as if you did not have them (which is correct and intended, I believe). This causes all kinds of problems on an selinux system for e.g. X, NetworkManager and OpenRC itself. I'm working on a change to the linuxrc script to try out myself, to mount /run correctly on selinux systems inside the initramfs.
I modified the linuxrc script in the initramfs to mount /run with rootcontext=system_u:object_r:var_run_t,seclabel on selinux systems, and tried it on one of my such systems. But unfortunately the mount command fails when you ask it to use rootcontext= So I enabled the recently added Dropbear SSH feature - very nice, works really well by the way :-) - and SSH'd into the initrd environment while it was paused on bootup to poke around. The mount command is a busybox's, not util-linux, so it just complains of an invalid argument if you try to mount anything using rootcontext= . I tried both remounting /run and mounting a separate tmpfs on a made up directory, both fail. Then I copied /bin/mount (util-linux) plus its needed /lib64 libraries (including libselinux.so.1) from my rootfs which was already mounted at /newroot, into the initramfs environment. I was then able to run the util-linux mount command using rootcontext= and it did not emit any error, either with remount option for /run or when mounting a separate new tmpfs. However, although appearing to work, the resulting mountpoints did not have rootcontext= both in the output from the mount command and in /proc/mounts, so it simply did not take effect. Then I realised of course it wont work as no selinux policies are loaded yet. At that point I realised I'd opened a can of worms and getting selinux support into the initramfs environment is not going to be straighforward :-) Unless I'm barking up the wrong tree completely, I've concluded that >=sys-kernel/genkernel-4.1.0 is unusable for selinux systems and should not be used on them until we are able to mount /run using rootcontext= at the beginning inside the initramfs. Alternatively, which might be a better approach, on selinux systems at least, perhaps clean up the environment completely, including unmounting /run (maybe in the "cleanup" function in initrd.scripts?), before running switch_root. Then /run can be freshly mounted by OpenRC with the rootcontext= option. I say unusable as even on selinux systems without X or NetworkManager the /run situation still messes up OpenRC, which tries to run services multiple times on boot as a result (I'm seeing this on multiple selinux systems). I don't run any systemd systems so maybe there are no problems there. But anyway running a Gentoo selinux system without correct rootcontext= on /run is by definition a broken selinux system anyway. Workarounds that I can think of: disable selinux completely until this is fixed (my systems are permissive anyway, though there are still security benefits to lose by disabling a permissive system), use another initramfs solution, or one hack I might try now is insert a very early OpenRC service which unmounts /run and remounts it again after selinux policies are loaded.
An update: I'm happy and relieved as I finally have a correctly booting selinux machine using sys-kernel/genkernel-4.1.2-r3 and it's nice new features :-) Albeit with a nasty but simple hack. First I tried inserting a command to unmount /run in one of the early scripts in the OpenRC sysinit runlevel, that runs right after genkernel. but umount failed (busy error) as turns out OpenRC puts stuff in /run to track bootup as soon as it is launched from init. Then I tried a /bin/sh wrapper around the "/sbin/openrc sysinit" command that init launches as its first child. umount failed here as well as turns out, from what I could gather from debugging, that /sbin/init itself pops a single file into /run as soon as it is launched by genkernel. So I used a simple wrapper for /sbin/init containing just: #!/bin/sh echo "Trying to unmount leftover /run from genkernel" umount -n /run # sleep briefly so we can see any output from umount sleep 5 exec /sbin/init and added init=/sbin/myinit to kernel command line in grub (genkernel picks that up and launches that instead of /sbin/init). Now umount succeeded in unmounting /run and OpenRC booted perfectly without any of the mess created by the leftover /run, and I have a fully working selinux system again. OpenRC handled mounting /run itself as it had done before, and /proc/mounts now has only a single /run mounted with the correct rootcontext= option. This confirms to me without a doubt that the problem in this bug report is simply caused by the /run mountpoint leftover by genkernel without the needed rootcontext= on selinux systems. Simple solution IMHO is to simply unmount that in genkernel linuxrc script before running /sbin/init - but I'll leave that to genkernel devs.
One quick thing I forgot to add in case anyone tries what I did, make sure you set correct context on your wrapper: chcon --reference /sbin/init /sbin/myinit and of course don't forget: chmod 0755 /sbin/myinit
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/genkernel.git/commit/?id=ab6d73225f21be7d55649363ceb460d91270638d commit ab6d73225f21be7d55649363ceb460d91270638d Author: Thomas Deutschmann <whissi@gentoo.org> AuthorDate: 2021-02-08 01:25:50 +0000 Commit: Thomas Deutschmann <whissi@gentoo.org> CommitDate: 2021-02-08 21:20:28 +0000 linuxrc: Add gk.preserverun.disabled When this boolean option is set and enabled, genkernel initramfs will unmount /run before calling switch_root. This can help in SELinux context for example where labeling is required which is not supported by genkernel. Bug: https://bugs.gentoo.org/739424 Bug: https://bugs.gentoo.org/740576 Signed-off-by: Thomas Deutschmann <whissi@gentoo.org> defaults/initrd.defaults | 1 + defaults/linuxrc | 15 +++++++++++ doc/genkernel.8.txt | 6 +++++ ....1-switch_root-check-if-mountpoint-exists.patch | 31 ++++++++++++++++++++++ 4 files changed, 53 insertions(+)
Merging with bug 740576. *** This bug has been marked as a duplicate of bug 740576 ***
I’m sorry to bother you but when I saw this ticket being closed and genkernel-4.2.1-r1 being stable in Portage, I gave it another try. Sadly, also that version creates non-working initramfs for me. Just as 4.1.2-r3 did. When I go back to 4.0.10, I get working an initramfs. The symptoms haven’t changed: • X.org does not detect any input devices • the network interfaces are not started properly (especially IPv4) Should this ticket be reopened (because it seems to be closely related) or open another one? 740576 doesn’t really tackle my problem either because I don’t run SELinux.
Have you booted with gk.preserverun.disabled=yes?
Ahhh I missed that option. With gk.preserverun.disabled=yes, booting from the initramfs created by genkernel >=4.1.2-r3 works again. Thank you! But since I don’t use SELinux, why is it that I need that flag at all? You can find my emerge --info in comment #10.
