Hi, Here's my scenario : One server, two nodes; One read-only NFS root for both nodes; I boot both nodes on this read-only NFS, and I want to overlay a rw unionfs. The attached patches to generic/linuxrc and generic/initrd.scripts that permit the following : # from a syslinux config file # kernel /boot/kernel-genkernel-x86-2.6.15.1 initrd=/boot/initramfs-genkernel-x86-2.6.15.1 ip=whatever \ nfsroot=192.168.0.1:/nfsroot real_root=/dev/nfs unionfs=192.168.0.1:/var/state/systems,node01 This way I can maintain one single fs image (/nfsroot), and have overlays that modify certain files only. This adds reboot-proof statefulness to the nodes. I've tested it on a mini 4 nodes cluster : it works ok. However, some init scripts, when unmounting filesystems, tend to give errors when rebooting. I will have to look for a solution for this and submit it in another bug report. If anybody has an idea on this, please tell me...
Created attachment 78740 [details, diff] linuxrc.diff (against genkernel 3.3.10)
Created attachment 78741 [details, diff] initrd.scripts.diff (against genkernel 3.3.10)
About the problematic unmounting, there is information in bug #99682 : - For halt.sh modifications - For nfsmount modifications These should solve the rebooting problem.
Created attachment 79385 [details, diff] linuxrc patch (against genkernel 3.3.10) This patch also adds the moving of the "/memory" mount to the new root, in "/.unions/memory". It makes it possible to change the state of the unionfs branch from the booted system, which seems necessary to properly unmount it or make it RO before shutdown. I thought that having a /.unions/ directory to contain all the "/" overlays would probably make it simpler to manage the unionfses, but I am still wondering if it would be more appropriate to put it in /mnt or /media (what would the FHS do?) This "/.unions/" directory is used as a mask by the init scripts (halt.sh and netmount -- see bug # 99682) to know what to _not_ unmount in a 'traditional' way. The scripts will use other methods for these particular mounts. By using a directory mask, we can have multiple overlays on / stored in /.unions/* and not have to always modify the init scripts to take it into account. (vs pointing explicitely to "/memory" for example) I am working on having multiple overlays over /, so the "/.unions" directory will greatly simplify my future experiments.
Forgot to add in this bug report (but did so in the referenced one...). This unionfs RW from NFS could also deprecate the 'add' tarballs used in nfs-LiveCD booting, since it permits to customize in place the LiveCD and have the modifications brought back when you reboot using the same unionfs overlay. Perhaps the LiveCD should contain "emerge" then... release engineers? :)
(In reply to comment #5) > Perhaps the LiveCD should contain "emerge" then... release engineers? :) No chance... ;] Anyway, the LiveCD does have emerge, it just won't work properly without a proper unionfs setup, which we don't bother with on the LiveCD. See, here's the deal. We added unionfs support because we thought it would be cool. It wasn't. In fact, it sucked pretty badly. It caused tons of errors in building and generally didn't work. Now, we've learned a lot since then on how unionfs is supposed to work, but we aren't planning on fixing genkernel 3 with regards to unionfs. It is simply too broken. In fact, it is very likely that you'll see us *removing* the remedial unionfs support that is currently in genkernel simply because it does not work. When genkernel 4 comes along, we will likely include proper unionfs support in it, but that is still a long way off. The basic message here is that unionfs is still too buggy and too early in its development cycle for us to want to have to support it, especially with genkernel in its current state. I'll discuss this with the other genkernel developers and we'll determine what works best for us after the 2006.0 release is out the door. It could be "fixing" the current genkernel, leaving it as is, or even stripping the broken unionfs support. I really don't know what we're going to do, but whatever we do, it'll be what is best for us at this time.
Thanks for the information. I also tried working with unionfs before and had several issues with it, so I gave up at that time. I was also in the "would be cool" mood back then... :) However, now, with unionfs-1.1.1, the issues I had are all gone, so, if I may simply add a "works for me"... Notice that I don't know about the LiveCD (which is quite important). The only thing I can state is that I have the above mentioned setup working ok with the patch submitted here + bug # 99682 patches. Perhaps I could maintain a 'union-ssi' USE-enabled patch for genkernel -- giving tons of warnings at emerge --, so you could strip out unionfs of the stock genkernel? I also tested OK the patched genkernel to build an SSI system, which is, IMHO, the most elegant way to build a diskless cluster or network of workstations, and I would be happy to see Gentoo be the first distro that supports it out of the box. If anybody would like to access my testing mini-cluster, I would be pleased to give you access; contact me personally. If you have any tests you would like me to run (to stress unionfs), also please tell me. Good luck for the 2006.0 release.
Well, I'll say that it also "works for me." In my case I'm not interested in clustering so much. I've got a hardware development environment where I need to provide a common netboot image for heterogeneous hardware. I had made some minor changes to linuxrc with the same net affect as these changes. It's really nice to have Gentoo and genkernel support this stuff out of the box. It makes my life _sooo_ much easier.
I've added these patches (slightly modified) to CVS... I'll resolve this bug when a version of genkernel using them is released.
This should be fixed in 3.3.11c, which is now in portage.
(In reply to comment #10) > This should be fixed in 3.3.11c, which is now in portage. > Could it be that you've missed one little change from the above patch of initrd.scripts? All changes have been applied but this one is missing, resulting in error-messages during boot (see below): --- initrd.scripts.orig 2006-04-14 11:22:18.000000000 +0000 +++ initrd.scripts 2006-04-14 11:22:47.000000000 +0000 @@ -567,7 +567,7 @@ } setup_unionfs() { - if [ "${USE_UNIONFS_NORMAL}" -eq '1' -a "${CDROOT}" -eq '1' ] + if [ "${USE_UNIONFS_NORMAL}" -eq '1' ] then # Directory used for rw changes in union mount filesystem UNION=/union messages during boot without the above change: >> Loading modules :: Scanning for usb-storage...scsi_mod, usb-storage loaded. :: Scanning for dm-mod...dm-mod loaded. :: Scanning for dm-mirror...dm-mirror loaded. >> Activating udev >> Determining root device... >> Mounting root... >> Attemping to mount NFS root on 10.113.2.1:/mnt/ThinClient/tftproot/wiggum mkdir: Cannot create directory `/newroot/tmp/.initrd': Read-only file system >> Booting (initramfs)...... messages during boot with the above change applied: >> Loading modules :: Scanning for usb-storage...scsi_mod, usb-storage loaded. :: Scanning for dm-mod...dm-mod loaded. :: Scanning for dm-mirror...dm-mirror loaded. >> Activating udev >> Loading unionfs module >> mounting 10.113.2.1:/mnt/ThinClient/union to /memory for unionfs support >> Determining root device... >> Mounting root... >> Attemping to mount NFS root on 10.113.2.1:/mnt/ThinClient/tftproot/wiggum >> Addition of /newroot to /union successful >> Booting (initramfs)......
You're right. I read that part of the patch backwards and was like "It's already there!" Anyway, this will be fixed in 3.3.11d whenever I get one out.
(In reply to comment #12) > You're right. I read that part of the patch backwards and was like "It's > already there!" > > Anyway, this will be fixed in 3.3.11d whenever I get one out. 3.3.11d now in Portage, thanks.
