Current busybox built-in modprobe unable to load zstd compressed modules and will give error "key was rejected by service". This causes systems that require additional modules to mount root fail to boot. For example, LUKS and ZFS. Reproducible: Always
Created attachment 734722 [details, diff] Patch enable native kmod and support zstd compressed moduels This patch enable native kmod tool (along with xz, zlib, zstd libraries) that able to load zstd compressed moduels from initramfs.
You are right that modutils from busybox don't support zstd (yet). But (e)udev in genkernel is linked against (lib)kmod with zstd support enabled and should take care of loading modules via udev builtin... What I tested: > # modinfo raid1 > filename: /lib/modules/5.13.12-gentoo-x86_64/kernel/drivers/md/raid1.ko.zst > # cat /usr/lib/udev/rules.d/99-custom.rules > KERNEL=="sd?*", RUN{builtin}+="kmod load libata raid1" From /run/initramfs/udevd.log: > [...] > RUN 'kmod load libata raid1' /usr/lib/udev/rules.d/99-custom.rules:1 > handling device node '/dev/sdb', devnum=b8:16, mode=0600, uid=0, gid=0 > preserve already existing symlink '/dev/block/8:16' to '../sdb' > found 'b8:16' claiming '/run/udev/links/\x2fdisk\x2fby-id\x2fscsi-36000c29725ea06fd6154d38cb4c59284' > creating link '/dev/disk/by-id/scsi-36000c29725ea06fd6154d38cb4c59284' to '/dev/sdb' > preserve already existing symlink '/dev/disk/by-id/scsi-36000c29725ea06fd6154d38cb4c59284' to '../../sdb' > found 'b8:16' claiming '/run/udev/links/\x2fdisk\x2fby-id\x2fwwn-0x6000c29725ea06fd6154d38cb4c59284' > creating link '/dev/disk/by-id/wwn-0x6000c29725ea06fd6154d38cb4c59284' to '/dev/sdb' > preserve already existing symlink '/dev/disk/by-id/wwn-0x6000c29725ea06fd6154d38cb4c59284' to '../../sdb' > found 'b8:16' claiming '/run/udev/links/\x2fdisk\x2fby-path\x2fpci-0000:00:10.0-scsi-0:0:1:0' > creating link '/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:1:0' to '/dev/sdb' > preserve already existing symlink '/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:1:0' to '../../sdb' > created db file '/run/udev/data/b8:16' for '/devices/pci0000:00/0000:00:10.0/host0/target0:0:1/0:0:1:0/block/sdb' > Execute 'load' 'libata' > Inserted 'libata' > Execute 'load' 'raid1' > Inserted 'raid1' > adding watch on '/dev/sdb' > # lsmod > Module Size Used by Not tainted > raid1 49152 2 > md_mod 188416 1 raid1 > libata 237568 0 > crc32_pclmul 16384 0 > crc32c_intel 24576 0 > mptspi 28672 4 > mptscsih 49152 1 mptspi > mptbase 102400 2 mptspi,mptscsih Could you please check size/content of modules.dep file in your initramfs for example? I suspect that kmod from host system has no USE=zstd enabled so you end up with empty modules index in initramfs with the result that udev in initramfs believes there aren't any modules available (and because it is empty but present it won't get re-created in initramfs).
I do have globle zstd USE enabled, and I did checked modules.dep in initramfs, and it looks ok. lsmod did show some modules were loaded by eudev but others are not worked, included LUKS (dm-crypt, dm-mod, etc.), ZFS, and I suspect that they are loaded by cryptsetup and zpool command which will try to call modprobe / insmod. I also just found I cannot mount ext4 fs due to mount cannot load the module. But they all fixed when using native kmod. > ... > kernel/drivers/md/linear.ko.zst: kernel/drivers/md/md-mod.ko.zst > kernel/drivers/md/raid0.ko.zst: kernel/drivers/md/md-mod.ko.zst > kernel/drivers/md/raid1.ko.zst: kernel/drivers/md/md-mod.ko.zst > kernel/drivers/md/raid10.ko.zst: kernel/drivers/md/md-mod.ko.zst > kernel/drivers/md/raid456.ko.zst: kernel/crypto/async_tx/async_raid6_recov.ko.zst kernel/crypto/async_tx/async_pq.ko.zst kernel/crypto/async_tx/async_xor.ko.zst kernel/crypto/async_tx/async_memcpy.ko.zst kernel/crypto/async_tx/async_tx.ko.zst kernel/drivers/md/md-mod.ko.zst kernel/lib/raid6/raid6_pq.ko.zst kernel/crypto/xor.ko.zst kernel/lib/libcrc32c.ko.zst > kernel/drivers/md/multipath.ko.zst: kernel/drivers/md/md-mod.ko.zst > kernel/drivers/md/md-mod.ko.zst: > kernel/drivers/md/dm-mod.ko.zst: > kernel/drivers/md/dm-bufio.ko.zst: kernel/drivers/md/dm-mod.ko.zst > kernel/drivers/md/dm-bio-prison.ko.zst: > kernel/drivers/md/dm-crypt.ko.zst: kernel/security/keys/trusted-keys/trusted.ko.zst kernel/lib/asn1_encoder.ko.zst kernel/drivers/md/dm-mod.ko.zst > kernel/drivers/md/dm-delay.ko.zst: kernel/drivers/md/dm-mod.ko.zst > kernel/drivers/md/dm-flakey.ko.zst: kernel/drivers/md/dm-mod.ko.zst > kernel/drivers/md/dm-multipath.ko.zst: kernel/drivers/md/dm-mod.ko.zst > kernel/drivers/md/dm-round-robin.ko.zst: kernel/drivers/md/dm-multipath.ko.zst kernel/drivers/md/dm-mod.ko.zst > kernel/drivers/md/dm-queue-length.ko.zst: kernel/drivers/md/dm-multipath.ko.zst kernel/drivers/md/dm-mod.ko.zst > kernel/drivers/md/dm-service-time.ko.zst: kernel/drivers/md/dm-multipath.ko.zst kernel/drivers/md/dm-mod.ko.zst > kernel/drivers/md/dm-switch.ko.zst: kernel/drivers/md/dm-mod.ko.zst > kernel/drivers/md/dm-snapshot.ko.zst: kernel/drivers/md/dm-bufio.ko.zst kernel/drivers/md/dm-mod.ko.zst > kernel/drivers/md/persistent-data/dm-persistent-data.ko.zst: kernel/drivers/md/dm-bufio.ko.zst kernel/drivers/md/dm-mod.ko.zst kernel/lib/libcrc32c.ko.zst > kernel/drivers/md/dm-mirror.ko.zst: kernel/drivers/md/dm-region-hash.ko.zst kernel/drivers/md/dm-log.ko.zst kernel/drivers/md/dm-mod.ko.zst > kernel/drivers/md/dm-log.ko.zst: kernel/drivers/md/dm-mod.ko.zst > kernel/drivers/md/dm-region-hash.ko.zst: kernel/drivers/md/dm-log.ko.zst kernel/drivers/md/dm-mod.ko.zst > kernel/drivers/md/dm-log-userspace.ko.zst: kernel/drivers/md/dm-log.ko.zst kernel/drivers/md/dm-mod.ko.zst > kernel/drivers/md/dm-zero.ko.zst: kernel/drivers/md/dm-mod.ko.zst > kernel/drivers/md/dm-raid.ko.zst: kernel/drivers/md/raid456.ko.zst kernel/crypto/async_tx/async_raid6_recov.ko.zst kernel/crypto/async_tx/async_pq.ko.zst kernel/crypto/async_tx/async_xor.ko.zst kernel/crypto/async_tx/async_memcpy.ko.zst kernel/crypto/async_tx/async_tx.ko.zst kernel/drivers/md/dm-mod.ko.zst kernel/drivers/md/md-mod.ko.zst kernel/lib/raid6/raid6_pq.ko.zst kernel/crypto/xor.ko.zst kernel/lib/libcrc32c.ko.zst > kernel/drivers/md/dm-thin-pool.ko.zst: kernel/drivers/md/persistent-data/dm-persistent-data.ko.zst kernel/drivers/md/dm-bio-prison.ko.zst kernel/drivers/md/dm-bufio.ko.zst kernel/drivers/md/dm-mod.ko.zst kernel/lib/libcrc32c.ko.zst > kernel/drivers/md/dm-verity.ko.zst: kernel/lib/reed_solomon/reed_solomon.ko.zst kernel/drivers/md/dm-bufio.ko.zst kernel/drivers/md/dm-mod.ko.zst > kernel/drivers/md/dm-cache.ko.zst: kernel/drivers/md/persistent-data/dm-persistent-data.ko.zst kernel/drivers/md/dm-bio-prison.ko.zst kernel/drivers/md/dm-bufio.ko.zst kernel/drivers/md/dm-mod.ko.zst kernel/lib/libcrc32c.ko.zst > kernel/drivers/md/dm-cache-smq.ko.zst: kernel/drivers/md/dm-cache.ko.zst kernel/drivers/md/persistent-data/dm-persistent-data.ko.zst kernel/drivers/md/dm-bio-prison.ko.zst kernel/drivers/md/dm-bufio.ko.zst kernel/drivers/md/dm-mod.ko.zst kernel/lib/libcrc32c.ko.zst > kernel/drivers/md/dm-log-writes.ko.zst: kernel/drivers/md/dm-mod.ko.zst > kernel/drivers/md/dm-integrity.ko.zst: kernel/crypto/async_tx/async_xor.ko.zst kernel/crypto/async_tx/async_tx.ko.zst kernel/drivers/md/dm-bufio.ko.zst kernel/drivers/md/dm-mod.ko.zst kernel/crypto/xor.ko.zst > ... > extra/zstd/zzstd.ko.zst: extra/spl/spl.ko.zst > extra/unicode/zunicode.ko.zst: > extra/nvpair/znvpair.ko.zst: extra/spl/spl.ko.zst > extra/spl/spl.ko.zst: > extra/zcommon/zcommon.ko.zst: extra/nvpair/znvpair.ko.zst extra/spl/spl.ko.zst > extra/icp/icp.ko.zst: extra/zcommon/zcommon.ko.zst extra/nvpair/znvpair.ko.zst extra/spl/spl.ko.zst > extra/avl/zavl.ko.zst: extra/spl/spl.ko.zst > extra/lua/zlua.ko.zst: > extra/zfs/zfs.ko.zst: extra/lua/zlua.ko.zst extra/unicode/zunicode.ko.zst extra/icp/icp.ko.zst extra/zcommon/zcommon.ko.zst extra/nvpair/znvpair.ko.zst extra/avl/zavl.ko.zst extra/zstd/zzstd.ko.zst extra/spl/spl.ko.zst
Looks like we need to address two problems: Switch to kmod in initramfs because busybox doesn't support zstd (yet) but also don't use depmod from host system which could lack zstd support, too.
Or just warning user they didn't enable zstd on kmod when kernel configured using zstd?
Created attachment 736969 [details, diff] Patch enable native kmod and support zstd compressed moduels v2 v2 patch Enable native kmod by default, can be disabled by set "KMOD=no" in genkernel.conf. Check if the current module compression algorithm is incompatible with the current system's kmod when compiling the kernel and/or initramfs, and gives an error message.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/genkernel.git/commit/?id=153a877d333d3b85920267535aef950056c92192 commit 153a877d333d3b85920267535aef950056c92192 Author: Thomas Deutschmann <whissi@gentoo.org> AuthorDate: 2021-09-09 00:16:42 +0000 Commit: Thomas Deutschmann <whissi@gentoo.org> CommitDate: 2021-09-09 00:59:03 +0000 Refactor (compressed) kernel module handling To support a specific module compression algorithm, two things are needed: Used depmod utility on host system building the kernel must support chosen module compression algorithm to generate proper modules.dep file or genkernel would be unable to read module dependencies when copying modules to initramfs. At runtime, used modprobe utility must be able to handle chosen module compression algorithm or modules would be unloadable. To address the first requirement, genkernel will now check if used kmod utility on host system supports chosen module compression algorithm. To address the runtime requirement, this commit will switch from BusyBox's modutils implementation to kmod because BusyBox does not support ZSTD compression (yet). Bug: https://bugs.gentoo.org/809344 Signed-off-by: Thomas Deutschmann <whissi@gentoo.org> gen_configkernel.sh | 7 ++++ gen_determineargs.sh | 6 +++ gen_funcs.sh | 72 ++++++++++++++++++++++++++++++++++++ gen_initramfs.sh | 22 ++++++++++- gkbuilds/kmod.gkbuild | 26 ++++++++++++- patches/kmod/29/kmod-29-static.patch | 12 ++++++ 6 files changed, 143 insertions(+), 2 deletions(-)
Note: If you created kernel with sys-apps/kmod[-zstd] before this change, it is not enough to just re-generate initramfs. You have to rebuild kernel with sys-apps/kmod[zstd] (genkernel will now error out if zstd was chosen for module compression but used kmod doesn't support that now) to get proper modules.deps file.
It works well under the new patch.
Release via genkernel-4.2.4.
