Hi, do not see how to decrypt multiple LUKS2 partitions before mounting my raid1 btrfs volume. emerge info: http://dpaste.com/381SAH6 GRUB_CMDLINE_LINUX="rootfs=btrfs dobtrfs dolvm root_trim=yes crypt_root=UUID=25597508-6959-40a1-9eff-2fc0f5027b47"
Please describe your disk/volume layout.
Similar to below: nvme0n1 => p3 Luks => device 1 btrfs raid1 nvme1n1 => p3 Luks => device 2 btrfs raid2 nvme0n1 259:0 0 953.9G 0 disk ├─nvme0n1p1 259:1 0 260M 0 part /boot/efi ├─nvme0n1p2 259:2 0 256M 0 part /boot ├─nvme0n1p3 259:3 0 937.8G 0 part │ └─root 253:0 0 937.8G 0 crypt / nvme0n1 259:0 0 953.9G 0 disk ├─nvme1n1p1 259:1 0 260M 0 part /boot/efi ├─nvme1n1p2 259:2 0 256M 0 part /boot ├─nvme1n1p3 259:3 0 937.8G 0 part │ └─root 253:0 0 937.8G 0 crypt /
Any other information needed? Basically need to decrypt two lucks block devices during root before the btrfs dev scan happens.
Are there any plans to merge this from genkernel-next?
No, there are no plans to support multiple crypt_roots. Also, I still don't understand your problem. That looks like a basic setup which is already supported and doesn't even require multiple crypt_roots.
If you have 2 disks which are part of a raid/pool and they are neccessary to mount the rootfs, how would you do that with genkernel?
A typical RAID setup with genkernel looks like > # cat /proc/mdstat > Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] > md126 : active raid1 nvme3n1p2[0] nvme2n1p2[1] > 4190208 blocks super 1.2 [2/2] [UU] > > md127 : active raid1 nvme3n1p3[0] nvme2n1p3[1] > 1623336128 blocks super 1.2 [2/2] [UU] > bitmap: 0/13 pages [0KB], 65536KB chunk > # lsblk > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > nvme2n1 259:6 0 1,8T 0 disk > ├─nvme2n1p1 259:9 0 260M 0 part > ├─nvme2n1p2 259:10 0 4G 0 part > │ └─md126 9:126 0 4G 0 raid1 /boot > └─nvme2n1p3 259:11 0 1,5T 0 part > └─md127 9:127 0 1,5T 0 raid1 > └─root 253:0 0 1,5T 0 crypt > ├─dev1Storage-volSwap 253:1 0 16G 0 lvm [SWAP] > ├─dev1Storage-volLog 253:2 0 20G 0 lvm /var/log > └─dev1Storage-volRoot 253:3 0 1,4T 0 lvm / > nvme3n1 259:13 0 1,8T 0 disk > ├─nvme3n1p1 259:14 0 260M 0 part > ├─nvme3n1p2 259:15 0 4G 0 part > │ └─md126 9:126 0 4G 0 raid1 /boot > └─nvme3n1p3 259:16 0 1,5T 0 part > └─md127 9:127 0 1,5T 0 raid1 > └─root 253:0 0 1,5T 0 crypt > ├─dev1Storage-volSwap 253:1 0 16G 0 lvm [SWAP] > ├─dev1Storage-volLog 253:2 0 20G 0 lvm /var/log > └─dev1Storage-volRoot 253:3 0 1,4T 0 lvm / In this system I have two NVMe devices with 3 partitions (EFI, /boot and my 'vault'). You will create the RAID, first (in my case I created RAID for /boot and LUKS volume). On top of that RAID volume, you will add LUKS (md127 in my case). On top of that LUKS volume, you can add LVM for example like shown. But this is just one example. Please read source code: https://gitweb.gentoo.org/proj/genkernel.git/tree/defaults/linuxrc?h=v4.0.5#n588 We basically try to start everything first. Then we will scan for root device. And once we detect a root device, we will re-run most functions to allow volumes stored on root volume to become available (in my example the LVM thing).
So you have luks on top of mdadm, therefore just 1 crypt_root. But for my setup, with ZFS, i have multiple partitions that are LUKS. After decrypting all those, i can import my root pool. So its ZFS on LUKS. Any chances to get this working with normal genkernel? Im currently using genkernel-next to specify multiple LUKS devices to be unlocked.
Same for my setup which is BTRFS. I have multiple LUKS that need to be unlocked so that btrfs device scan can detect all members of the array.
When you require multiple LUKS devices, there is no chance. We have several ZFS users, also using encryption. You maybe have to adjust your setup.
Do you mean change or chance? The block at line 605 calls a do_LUKS function does that detect other devices? How does genkernel detect the other LUKS devices or how can we help him do it. Currently I have 2 and only one is picked up by genkernel. If you know of a way to detect these please help us out. I have read the wiki and the man page and do not see a way to do this currently.
her is the resoning genkernel-next has for the multiple crypt_roots: # The first entry will be the one that # is going to be mapped to ${REAL_ROOT}. # Multiple "roots" devices are needed # in order to support multiple dmcrypts # aggregated through software raid arrays. CRYPT_ROOTS="${CRYPT_ROOTS} ${x#*=}"
(In reply to Thomas Deutschmann from comment #10) > When you require multiple LUKS devices, there is no chance. > > We have several ZFS users, also using encryption. You maybe have to adjust > your setup. Hence this request to have multiple LUKS to decrypt with crypt_roots in genkernel. Is there a specific reason to not support multiple LUKS devices?
It's against the intention of genkernel. Please read the note in https://wiki.gentoo.org/wiki/Genkernel about genkernel's primary job. Everything we add has to be maintained/tested. Nobody is doing that so we keep feature set limited. Just saying, "but my setup will require..." is not enough. Explain why your setup is common and why you can't do it like most people are doing for years who don't have the need for multiple crypt_roots. I am sorry but until there's a *real* technical reason why we must support something like that the answer is: No, we are not going to support that. Please fix/adjust your setup if you want to use genkernel.
Unless I am mistaken and there is some sort of other way that we are missing using genkernel to activate a LUKS multi volume BTRFS/ZFS root then is that not enough of a reason to qualify for mainstreem? If people using btrfs raid1 + luks are unable to boot getntoo then that should be a a bug no?
https://forums.gentoo.org/viewtopic-t-1024794-start-0.html this forum post seems to also run into this issue.
From the genkernel description: >Its primary job is to bring up only the basic stuff needed to mount a (block) >device containing the root filesystem so that it can hand off control to real >system as soon as possible. Without multiple crypt devices, you cannot import some rootfs to hand off control for the real system to boot up. Meanwhile i just switched away from genkernel/genkernel-next for initramfs to dracut. Its sad that a tool that should make all this easy makes such a basic thing so complicated. I can understand if you still dont want to implement this especially since you dont see a "*real* technical reason" to have multiple LUKS for the rootfs. So in the end, i did as you suggested. I adjusted my setup to not have this genkernel issue anymore and im so far happy with dracut...
Here is what the boot looks like with dracut and what we are trying to get with genkernel.. ❯ sudo blkid /dev/nvme1n1p1: UUID="1435079933A60EB4" TYPE="ntfs" PTTYPE="dos" PARTUUID="1783a775-8b86-3644-9c92-8ecd5ef1e451" /dev/nvme1n1p2: UUID="25597508-6959-40a1-9eff-2fc0f5027b47" TYPE="crypto_LUKS" PARTUUID="656c957b-d4dd-734e-a84f-610e10b4f6ea" /dev/nvme1n1p3: LABEL="cryptswap" UUID="f0ba1391-3970-421d-9938-f1b078cd4c4a" TYPE="ext2" PARTUUID="e9ac5fdd-0951-8140-b820-b97cdb4e81b2" /dev/nvme1n1p4: LABEL="boot" UUID="6b1a2cc1-bdfc-4785-86dd-1e4ba7b125fe" TYPE="ext4" PARTUUID="e7055681-a5ff-ac42-9ec8-e732cb8b6a2e" /dev/nvme0n1p1: LABEL="Recovery" UUID="A6802D31802D08FF" TYPE="ntfs" PARTLABEL="Basic data partition" PARTUUID="76daf90e-ef80-4c47-b20e-cb3184c55ee8" /dev/nvme0n1p2: UUID="6E2D-B681" TYPE="vfat" PARTLABEL="EFI system partition" PARTUUID="140ba496-73bc-491a-8a46-fe1e6bfbde93" /dev/nvme0n1p3: LABEL="crypto-swap" UUID="e95ab8f6-82bf-41f1-8a4a-e1b144079573" TYPE="swap" PARTLABEL="Microsoft reserved partition" PARTUUID="1cfbbd97-8ef7-4c9b-85fb-6874ddb45dcb" /dev/nvme0n1p4: UUID="901A47491A472B92" TYPE="ntfs" PARTLABEL="Basic data partition" PARTUUID="34fb6b38-b8e6-439b-b7f4-bcf7c7efc355" /dev/nvme0n1p5: UUID="ec2e18a1-3a59-49f6-b7ec-1e3e4fbd8065" TYPE="crypto_LUKS" PARTUUID="2846b6d8-5f95-40fc-b2d1-c509f178d0b0" /dev/mapper/luks-25597508-6959-40a1-9eff-2fc0f5027b47: LABEL="root" UUID="ed8ec199-1fba-4905-a63f-d2b8427a0f55" UUID_SUB="e120284f-0113-403d-bd47-85aa85625d10" TYPE="btrfs" /dev/mapper/luks-ec2e18a1-3a59-49f6-b7ec-1e3e4fbd8065: LABEL="root" UUID="ed8ec199-1fba-4905-a63f-d2b8427a0f55" UUID_SUB="21a3ee31-961f-4d9c-a7a1-46cee63db7cb" TYPE="btrfs" /dev/nvme1n1: PTUUID="49e48905-aad9-11e9-abb3-00d86153fb3e" PTTYPE="gpt" /dev/nvme0n1: PTUUID="c66534c8-966a-4dd1-9f15-34d8b8e786ef" PTTYPE="gpt" ~ ❯ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT nvme1n1 259:0 0 953.9G 0 disk ├─nvme1n1p1 259:2 0 450G 0 part ├─nvme1n1p2 259:3 0 488G 0 part │ └─luks-25597508-6959-40a1-9eff-2fc0f5027b47 254:1 0 488G 0 crypt /home ├─nvme1n1p3 259:4 0 15.4G 0 part └─nvme1n1p4 259:5 0 489.3M 0 part /boot nvme0n1 259:1 0 953.9G 0 disk ├─nvme0n1p1 259:6 0 499M 0 part ├─nvme0n1p2 259:7 0 99M 0 part ├─nvme0n1p3 259:8 0 16M 0 part ├─nvme0n1p4 259:9 0 488.4G 0 part └─nvme0n1p5 259:10 0 464.9G 0 part └─luks-ec2e18a1-3a59-49f6-b7ec-1e3e4fbd8065 254:0 0 464.9G 0 crypt ~ ❯ cat /proc/cmdline BOOT_IMAGE=/vmlinuz-5.6.0 root=UUID=ed8ec199-1fba-4905-a63f-d2b8427a0f55 ro rootflags=subvol=@root rd.luks.uuid=ec2e18a1-3a59-49f6-b7ec-1e3e4fbd8065 rd.luks.uuid=25597508-6959-40a1-9eff-2fc0f5027b47 rootfs=btrfs ~ ❯ sudo btrfs fi show Label: 'root' uuid: ed8ec199-1fba-4905-a63f-d2b8427a0f55 Total devices 2 FS bytes used 301.08GiB devid 1 size 488.00GiB used 310.02GiB path /dev/mapper/luks-25597508-6959-40a1-9eff-2fc0f5027b47 devid 2 size 464.84GiB used 25.00GiB path /dev/mapper/luks-ec2e18a1-3a59-49f6-b7ec-1e3e4fbd8065
We didn't need to support multiple LUKS volumes in the past 15 years and sorry, I still don't see that this has changed. Just fix your setup if you want to use genkernel or don't use genkernel like there are alternatives: From your example, /root resides on > └─nvme0n1p5 259:10 0 464.9G 0 part > └─luks-ec2e18a1-3a59-49f6-b7ec-1e3e4fbd8065 254:0 0 464.9G 0 crypt That's the volume genkernel is supposed to open during boot. Now configure your system services to open additional LUKS volume > ├─nvme1n1p2 259:3 0 488G 0 part > │ └─luks-25597508-6959-40a1-9eff-2fc0f5027b47 254:1 0 488G 0 crypt /home which contains your /home. When using OpenRC you would configure dmcrypt service to do that using a key file which can be located on encrypted root device. It's not complicated to do that.
@Thomas Deutschmann, the root and home are on the same btrfs volume which is a raid1: ❯ sudo btrfs subvol list / ID 256 gen 33127 top level 5 path @root ID 258 gen 33127 top level 5 path @home so if I only had one of my luks partitions at boot like you suggest btrfs would fail to mount because there are two devices part of the btrfs raid1: ❯ sudo btrfs fi show Label: 'root' uuid: ed8ec199-1fba-4905-a63f-d2b8427a0f55 Total devices 2 FS bytes used 307.20GiB devid 1 size 488.00GiB used 309.03GiB path /dev/mapper/luks-25597508-6959-40a1-9eff-2fc0f5027b47 devid 2 size 464.84GiB used 309.03GiB path /dev/mapper/luks-ec2e18a1-3a59-49f6-b7ec-1e3e4fbd8065 those two devices are needed at the initramfs level for btrfs to mount and then in fstab I mount my home and root: UUID=ed8ec199-1fba-4905-a63f-d2b8427a0f55 / btrfs defaults,noatime,subvol=@root 0 0 UUID=ed8ec199-1fba-4905-a63f-d2b8427a0f55 /home btrfs defaults,noatime,subvol=@home,nofail 0 0 so as you are asking to use dmcrypt at boot that means I need my root to be already mounted which it wont because of the missing luks volume that is part of the btrfs raid1 group.
In my ZFS setup, it looks like this: lsblk: /dev/sda1: LABEL="bpool" UUID="2188703416925228116" UUID_SUB="17643639836280086655" TYPE="zfs_member" PARTUUID="04ce98b2-01" /dev/sda2: UUID="2ecdef65-5d0b-5e82-a465-8921b5c17ebc" UUID_SUB="03c7f590-b604-2ed8-3fa9-06b13e7f35fe" LABEL="T520:swap" TYPE="linux_raid_member" PARTUUID="04ce98b2-02" /dev/sda3: UUID="3bc3119a-23ec-4675-9268-986700943d80" TYPE="crypto_LUKS" PARTUUID="04ce98b2-03" /dev/sdb1: LABEL="bpool" UUID="2188703416925228116" UUID_SUB="10115771453936582708" TYPE="zfs_member" PARTUUID="8ca73bef-01" /dev/sdb2: UUID="2ecdef65-5d0b-5e82-a465-8921b5c17ebc" UUID_SUB="5af5ce1d-f25e-3ee5-c126-da9c1d541408" LABEL="T520:swap" TYPE="linux_raid_member" PARTUUID="8ca73bef-02" /dev/sdb3: UUID="aab7bf38-a8f7-4757-acca-c93061941707" TYPE="crypto_LUKS" PARTUUID="8ca73bef-03" /dev/md127: UUID="f18e83a2-af0d-4cf0-a2ff-0dd15aa50ccb" TYPE="crypto_LUKS" /dev/mapper/luks-f18e83a2-af0d-4cf0-a2ff-0dd15aa50ccb: UUID="17166d30-de0e-4727-adac-99c6e3f52f6d" TYPE="swap" /dev/sdc: UUID="34207630-c450-4b81-9595-153decd6b8a9" TYPE="crypto_LUKS" /dev/mapper/luks-3bc3119a-23ec-4675-9268-986700943d80: LABEL="rpool" UUID="17445744107752055202" UUID_SUB="16402149316153681789" TYPE="zfs_member" /dev/mapper/luks-aab7bf38-a8f7-4757-acca-c93061941707: LABEL="rpool" UUID="17445744107752055202" UUID_SUB="16725569833090245865" TYPE="zfs_member" /dev/mapper/luks-34207630-c450-4b81-9595-153decd6b8a9: UUID_SUB="14194125744820836073" TYPE="zfs_member" zpool status: pool: bpool state: ONLINE status: Some supported features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(5) for details. scan: scrub repaired 0B in 0 days 00:00:02 with 0 errors on Wed Apr 1 17:39:58 2020 config: NAME STATE READ WRITE CKSUM bpool ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 /dev/disk/by-id/ata-HITACHI_HTS727550A9E364_J3320082GRM5VA-part1 ONLINE 0 0 0 /dev/disk/by-id/ata-ST500LM000-SSHD-8GB_W764KNPY-part1 ONLINE 0 0 0 errors: No known data errors pool: rpool state: ONLINE action: Export this pool on all systems on which it is imported. Then import it to correct the mismatch. scan: scrub repaired 0 in 1h14m with 0 errors on Sun Mar 8 15:38:32 2020 config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 /dev/mapper/luks-3bc3119a-23ec-4675-9268-986700943d80 ONLINE 0 0 0 /dev/mapper/luks-aab7bf38-a8f7-4757-acca-c93061941707 ONLINE 0 0 0 cache /dev/mapper/luks-34207630-c450-4b81-9595-153decd6b8a9 ONLINE 0 0 0 errors: No known data errors lsblk: NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 465.8G 0 disk ├─sda1 8:1 0 5G 0 part ├─sda2 8:2 0 16G 0 part │ └─md127 9:127 0 16G 0 raid1 │ └─luks-f18e83a2-af0d-4cf0-a2ff-0dd15aa50ccb 253:0 0 16G 0 crypt [SWAP] └─sda3 8:3 0 444.8G 0 part └─luks-3bc3119a-23ec-4675-9268-986700943d80 253:1 0 444.8G 0 crypt sdb 8:16 0 465.8G 0 disk ├─sdb1 8:17 0 5G 0 part ├─sdb2 8:18 0 16G 0 part │ └─md127 9:127 0 16G 0 raid1 │ └─luks-f18e83a2-af0d-4cf0-a2ff-0dd15aa50ccb 253:0 0 16G 0 crypt [SWAP] └─sdb3 8:19 0 444.8G 0 part └─luks-aab7bf38-a8f7-4757-acca-c93061941707 253:2 0 444.8G 0 crypt sdc 8:32 0 60G 0 disk └─luks-34207630-c450-4b81-9595-153decd6b8a9 253:3 0 60G 0 crypt zram0 252:0 0 1.4G 0 disk [SWAP] zram1 252:1 0 1.4G 0 disk [SWAP] zram2 252:2 0 1.4G 0 disk [SWAP] zram3 252:3 0 1.4G 0 disk [SWAP] So i have 2 LUKS devices that are mirrored for the root pool that need unlocking: sda3 and sdb3. sdc is a fast SSD cache which could be mounted later to the zfs pool. Then theres a mdadm with LUKS on top for swap ( could also be done later). I think its quite ignorant to say that "we" didnt need it for 15 years. Just cause you dont have this need dosnt mean it applies to everyone else. The others had the same issues and just ditched genkernel initramfs like we did now... This dosnt mean there is no need for it, just cause it gets ignored or worked around. If you are arguing like this, i could argue: *sarcasm* There is no need for genkernel initramfs since their are alternatives and no one had a "*real* technical reason" to depend on genkernel initramfs. From what i've seen in other bug reports and requests, this seems to be more of a political thing to block everything that's been done in genkernel-next. TLDR; there is a "*real* technical reason" for this setup and ignoring it dosnt mean it dosnt exist.
You are seeing this now because gentoo-next has been masked and everyone who was running btrfs raid on encrypted block devices is looking at moving back to genkernel. Btrfs does not support internal encryption and the recommended way to encrypt btrfs filesystems is to run them on at least 2 luks encrypted block devices (you need at least 2 devices to restore data caused by bit level corruption on the devices). If there is not going to be support for multiple encrypted block devices I will need to either fork genkernel or work on updating genkernel-next. If the issue is the work in adding this I am happy to look at it but there having not looked at the genkernel code in years and it might take me a bit of time to get started.
That should be genkernel-next not gentoo-next.
Created attachment 649158 [details, diff] patch to add support for crypt_roots to genkernel 4.0.9 This is a first pass at adding crypt_roots to genkernel 4.0.9 it could do with some cleanup. I have tested it with a btrfs root on two luks encrypted block devices.
The above patch breaks the single device case. I am looking at that now.
Created attachment 649244 [details, diff] minimal patch to add support for crypt_roots to genkernel 4.0.9 This is a minimal change to add support for crypt_roots support. It allows the use of root filesystems built on multiple encrypted block devices, which is the recommended way of implementing encryption for btrfs. It has has been tested on configurations with single and multiple encrypted devices.
(In reply to Thomas Deutschmann from comment #7) > A typical RAID setup with genkernel looks like > > > # cat /proc/mdstat > > Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] > > md126 : active raid1 nvme3n1p2[0] nvme2n1p2[1] > > 4190208 blocks super 1.2 [2/2] [UU] > > > > md127 : active raid1 nvme3n1p3[0] nvme2n1p3[1] > > 1623336128 blocks super 1.2 [2/2] [UU] > > bitmap: 0/13 pages [0KB], 65536KB chunk > > > # lsblk > > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > > nvme2n1 259:6 0 1,8T 0 disk > > ├─nvme2n1p1 259:9 0 260M 0 part > > ├─nvme2n1p2 259:10 0 4G 0 part > > │ └─md126 9:126 0 4G 0 raid1 /boot > > └─nvme2n1p3 259:11 0 1,5T 0 part > > └─md127 9:127 0 1,5T 0 raid1 > > └─root 253:0 0 1,5T 0 crypt > > ├─dev1Storage-volSwap 253:1 0 16G 0 lvm [SWAP] > > ├─dev1Storage-volLog 253:2 0 20G 0 lvm /var/log > > └─dev1Storage-volRoot 253:3 0 1,4T 0 lvm / > > nvme3n1 259:13 0 1,8T 0 disk > > ├─nvme3n1p1 259:14 0 260M 0 part > > ├─nvme3n1p2 259:15 0 4G 0 part > > │ └─md126 9:126 0 4G 0 raid1 /boot > > └─nvme3n1p3 259:16 0 1,5T 0 part > > └─md127 9:127 0 1,5T 0 raid1 > > └─root 253:0 0 1,5T 0 crypt > > ├─dev1Storage-volSwap 253:1 0 16G 0 lvm [SWAP] > > ├─dev1Storage-volLog 253:2 0 20G 0 lvm /var/log > > └─dev1Storage-volRoot 253:3 0 1,4T 0 lvm / > > In this system I have two NVMe devices with 3 partitions (EFI, /boot and my > 'vault'). > You will create the RAID, first (in my case I created RAID for /boot and > LUKS volume). > On top of that RAID volume, you will add LUKS (md127 in my case). > On top of that LUKS volume, you can add LVM for example like shown. > > But this is just one example. Please read source code: > https://gitweb.gentoo.org/proj/genkernel.git/tree/defaults/linuxrc?h=v4.0. > 5#n588 > > We basically try to start everything first. Then we will scan for root > device. And once we detect a root device, we will re-run most functions to > allow volumes stored on root volume to become available (in my example the > LVM thing). btrfs supports raid1 in filesystem which allows you to recover from bit errors in the underlying block devices. This can happen when you have a mirrored raid and the same blocks contain different data between two mirrors. Linux software raid can't tell which block is correct if the mirrors are different. btrfs raid1 stores checksums which allow the filesystem to determine which version is corrupted and replace it. To do this btrfs needs to handle raid the underlying devices together. I have had this bit level corruption occur with Linux software raid and it is very hard to recover from because there is no general way to know which data is corrupted. I believe zfs raid works in a similar way.
Created attachment 649314 [details] boot failes with multi luks + btrfs raid1 unable to boot with my raid1 btrfs setup and applied patch..
(In reply to Vladi from comment #28) > Created attachment 649314 [details] > boot failes with multi luks + btrfs raid1 > > unable to boot with my raid1 btrfs setup and applied patch.. This looks like it caused by something else. The patch sets up the first device as /dev/mapper/root and any subsequent devices as /dev/mapper/root_n which is what it has done. I assume /sbin/switch_root moves you into /newroot so you would need to # ls /usr/lib/systemd/systemd to check if it was there. I have tested multi-disk setups using systemd and using regular openrc and it worked for both. Is this a systemd setup or openrc?
I have the rootfstype=btrfs set so it doesn't probe for the filesystem type.
Yup, it was my fault. I uncommented my genkernel line but did not remove the systemd line but now I use openrc. Thanks for getting this done works great!
Created attachment 650442 [details, diff] minimal patch to add support for crypt_roots to genkernel 4.0.10 This is the updated patch for 4.0.10. If there is interest in merging this I can created a patch based the git repo.
(In reply to Edward Middleton from comment #22) > You are seeing this now because gentoo-next has been masked and everyone who > was running btrfs raid on encrypted block devices is looking at moving back > to genkernel. This is the case for me. I've been booting my systems with LUKS -> mdadm -> lvm -> root for years without issues using genkernel-next. Since this is masked now, I am looking into genkernel and it does not work for my setup. @Thomas Deutschmann: genkernel did not need to support multiple LUKS volumes, because there was genkernl-next! Please consider adding Edward Middelton's patches.
I am not saying no but adding something which *can* break stuff if you don't have a setup to test on your own is challenging.
Maybe I misread your posts, but your previous comments pretty much sounded like "no" to me: "No, there are no plans to support multiple crypt_roots.", "When you require multiple LUKS devices, there is no chance.", "It's against the intention of genkernel.", "We didn't need to support multiple LUKS volumes in the past 15 years and sorry, I still don't see that this has changed." I am glad to hear that this is not the case. I will test the patch, will report back later.
(In reply to Christian Becke from comment #33) > (In reply to Edward Middleton from comment #22) > > You are seeing this now because gentoo-next has been masked and everyone who > > was running btrfs raid on encrypted block devices is looking at moving back > > to genkernel. > This is the case for me. I've been booting my systems with LUKS -> mdadm -> > lvm -> root for years without issues using genkernel-next. Since this is > masked now, I am looking into genkernel and it does not work for my setup. I haven't tested this configuration. If the raid setup occurs after the luks devices are setup it should work. Is there a reason for running luks on the raw block devices rather then on the mdadm raid? I have only ever run with mdadm (raid1) -> luks -> lvm
Well, I still don't really understand why this must be supported. I still believe that you can get it working when you change something in your existing setup... but the "WONTFIX" attitude is gone given that genkernel-next is going away... But before I can have a look I must understand the use case and also have time to setup a test system...
(In reply to Thomas Deutschmann from comment #37) > Well, I still don't really understand why this must be supported. I still > believe that you can get it working when you change something in your > existing setup... but the "WONTFIX" attitude is gone given that > genkernel-next is going away... > > But before I can have a look I must understand the use case and also have > time to setup a test system... For btrfs its the recommended setup. It gives you * recovery from single disk hardware failure * recovery from bit level hardware corruption (through checksums) * disclosure of data due to resale of rma's drives RAID alone will not protect you against bit level corruption as it has no way to determine which disk has the correct data. Bit level errors are more common with large multi TB disks and flash memory based drives. Minimal setup is two drives /dev/{sda1,sdb1} boot (ext2 (md/raid1) 256MiB boot /dev/{sda2,sdb2} root (luks -> btrfs[raid1]) * root I haven't specifically tested the below steps as I don't currently have a machine available but its the basic procedure. # cryptsetup luksFormat -s 512 -c aes-xts-plain64 /dev/sda2 # cryptsetup luksFormat -s 512 -c aes-xts-plain64 /dev/sdb2 # cryptsetup open /dev/sda2 root # cryptsetup open /dev/sdb2 root1 # mkfs.btrfs -L BTRFS --d raid1 -m raid1 /dev/mapper/root /dev/mapper/root1 # mkdir /mnt/btrfs # mount /dev/mapper/root /mnt/btrfs # btrfs subvolume create /mnt/btrfs/root you then install root filesystem to /mnt/btrfs/root add the following to your kernel command line. "domdadm dobtrfs crypt_roots=UUID=[uuid for /dev/sda2] crypt_roots=UUID=[uuid for /dev/sdb2] real_root=LABEL=BTRFS real_rootflags=compress=lzo rootfstype=btrfs"
Created attachment 656064 [details, diff] minimal patch to add support for crypt_roots to genkernel 4.1.0
I am also here with the same issue. This is my first gentoo install, and I signed up to this bug tracker specifically to report this problem. If we want to use BTRFS RAID with LUKS for ROOT, then we have no choice - we need to be able to unlock multiple LUKS disks on boot! This is possible with systemd, but I came here to escape Debian in order to get away from that. I used to use mdadm, but btrfs raid is *much* better. I am absolutely not going back to either one. One way or another, if I'm staying with gentoo, I'm working around this problem.
@EdwardMiddleton - I haven't understood your patch yet, but please note the additional issue, that to cover the case when multiple disks have the same password, that password should be tried against each disk, to save typing it in multiple times. This is the way systemd handles it. There is a flag in /etc/crypttab to indicate disks that are to be opened on boot. In my case, I have 3 luks disks (root_1, root_2, & data). The first two need to be opened at boot.
(In reply to Diagon from comment #41) > @EdwardMiddleton - > > I haven't understood your patch yet, but please note the additional issue, > that to cover the case when multiple disks have the same password, that > password should be tried against each disk, to save typing it in multiple > times. This is a minimal patch that just allows you to setup multiple encrypted block devices.
Created attachment 657942 [details, diff] minimal patch to add support for crypt_roots to genkernel 4.1.2
I did get this to work on an install. It needs to be pointed out that: 1. It necessitates putting in the password separately for each disk 2. The opened disks have fixed names: /dev/mapper/root and root_1 So the kernel line is: dobtrfs root=/dev/mapper/root crypt_roots=/dev/sda crypt_roots=/dev/scc rootfstype=btrfs ro
patch now fails for version 4.2.1-r1, any chance to include this into genkernel?
Created attachment 696615 [details, diff] minimal patch to add support for crypt_roots to genkernel 4.2.1 minimal patch to add support for crypt_roots to genkernel 4.2.1
(In reply to Edward Middleton from comment #46) > Created attachment 696615 [details, diff] [details, diff] > minimal patch to add support for crypt_roots to genkernel 4.2.1 > > minimal patch to add support for crypt_roots to genkernel 4.2.1 Thanks so much Edward!
Created attachment 723700 [details, diff] minimal patch to add support for crypt_roots to genkernel 4.2.3
Thanks for the patches! I decided to document my Gentoo installation steps. As I always try to have as much as possible encrypted, I created the following repo: https://github.com/duxco/genkernel-patches I'll try to keep that up to date. Perhaps, it's helpful for the one or the other :)
I want to create a zfs mirror on top of two LUKS devices. I prefer zpool scrub to any scrub functionality that mdraid has.
I just realized that ZFS native encryption is almost as good as LUKS, so I'm going to use it.
fyi, support for "keyctl" got merged into genkernel: https://github.com/gentoo/genkernel/pull/10 Now, I just have to wait for the next genkernel release. If my tests succeed, I use that instead of integrating a keyfile into initramfs. This avoids accidental leakage of the keyfile and allows for having initramfs on unencrypted /boot which would otherwise have to realised via Grub's "GRUB_ENABLE_CRYPTODISK" setting. I personally use Secure Boot and GnuPG sign files required for booting.
genkernel-4.3.2 is stable! Here are the magic incantations to make keyctl work with David Sardari's patches[1] [1] https://github.com/duxco/genkernel-patches 1) Enable KEYCTL=yes in /etc/genkernel.conf 2) For each LUKS partition you want opened at boot, run: # cryptsetup token add --key-description potrzebie /dev/nvme0n1p6 (use raw block device, not the devmapper one) 3) Then, in /etc/default/grub, add 'keyctl_keydesc=potrzebie' to GRUB_CMDLINE_LINUX. Run `genkernel ramdisk`, regenerate grub.cfg and reboot. Passphrase prompt will be different, and you have to press Ctrl-D twice instead of Enter. Takes some getting used to, especially if you normally have CapsLock mapped to Control (which does not work in grub). But yes, it works, and it unlocks all LUKS partitions with only one passphrase. Many thanks, David Sardari, for your patches.
@Ed Santiago: Thanks for opening the PR :) https://github.com/duxsco/gentoo-genkernel-patches/pull/1 To be frank I forgot about Vladi's and my patches, because I switched over to systemd with dracut in the meantime, and that supports caching passphrases even with multiple devices of the same type, e.g. three root devices in BTRFS raid1c3. I expected for Vladi's and my patches to become obsolete with the introduction of keyctl: https://github.com/gentoo/genkernel/pull/10 But, I see now that the support for multiple devices of the same type wasn't introduced. As I don't use genkernel anymore, it's not good for me to take care of these patches anymore and I am unwilling to merge your PR, because I don't have a setup where I can test them on. I personally don't mind for you opening a PR at https://github.com/gentoo/genkernel to get multiple devices of the same type supported. You have at least a system where you can actually test these changes on ;) @Vladi Do you give your blessing for such a PR? I think you don't mind either, because you opened this bug report for the sole purpose of your patches being applied ;)
